waifu2x

Waifu2x is a well-known image super-resolution neural network for anime-style arts.

Link:

• (stable) https://github.com/AmusementClub/vs-mlrt/releases/download/model-20211203/waifu2x_v2.7z
• (preview) https://github.com/AmusementClub/vs-mlrt/releases/download/model-20211209/waifu2x_v3.7z; only required for vstrt.

Models

Includes all known publicly available waifu2x models:

• anime_style_art: requires pre-scaled input for the scaled2.0x variant
  • noise1 noise2 noise3 scale2.0x
• anime_style_art_rgb: requires pre-scaled input for the scale2.0x variant
  • noise0 noise1 noise2 noise3 scale2.0x
• photo: requires pre-scaled input for the scale2.0x variant
  • noise0 noise1 noise2 noise3 scale2.0x
• ukbench: requires pre-scaled input
  • scale2.0x
• upconv_7_anime_style_art_rgb
  • scale2.0x noise3_scale2.0x noise2_scale2.0x noise1_scale2.0x noise0_scale2.0x
• upconv_7_photo
  • scale2.0x noise0_scale2.0x noise1_scale2.0x noise2_scale2.0x noise3_scale2.0x
• cunet: tile size (block_w and block_h) must be multiples of 4.
  • noise0 noise1 noise2 noise3
  • scale2.0x
  • noise0_scale2.0x noise1_scale2.0x noise2_scale2.0x noise3_scale2.0x
• upresnet10
  • scale2.0x
  • noise0_scale2.0x noise1_scale2.0x noise2_scale2.0x noise3_scale2.0x

vsmlrt.py wrapper Usage

In order to simplify usage, we provided a Python wrapper module vsmlrt that provides full functionality of waifu2x caffe but with a more Pythonic interface:

from vsmlrt import Waifu2x, Waifu2xModel, Backend

src = core.std.BlankClip(format=vs.RGBS)

# backend could be:
#  - CPU Backend.OV_CPU(): recommended, generally faster on CPU.
#  - CPU Backend.ORT_CPU(num_streams=1, verbosity=2): vs-ort cpu backend.
#  - GPU Backend.ORT_CUDA(device_id=0, cudnn_benchmark=True, num_streams=1, verbosity=2)
#     - use device_id to select device
#     - set cudnn_benchmark=False to reduce script reload latency when debugging, but with slight throughput performance penalty.
flt = Waifu2x(src, noise=-1, scale=2, model=Waifu2xModel.upconv_7_anime_style_art_rgb, backend=Backend.ORT_CUDA())

Raw Model Usage

This section is mostly for reference purposes as the suggested way is to use the vsmlrt.py.

src = core.std.BlankClip(width=1920, height=1080, format=vs.RGBS)
flt = core.ov.Model(src, "upconv_7_anime_style_art_rgb_scale2.0x.onnx")

anime_style_art, anime_style_art_rgb, photo, ukbench models do not include builtin upscaling. Therefore, you need to upscale 2x using Catmull-Rom (bicubic(b=0, c=0.5)) before feeding the image to the models:

src = core.std.BlankClip(width=1920, height=1080, format=vs.RGBS)
flt = core.ov.Model(src.fmtc.resample(scale=2, kernel="bicubic", a1=0, a2=0.5), "anime_style_art_rgb_scale2.0x.onnx")

Notes

• cunet networks work best when the tile size (block_w/block_h) is in range 60 - 150 and multiples of 4.