New upscaling algorithm: waifu2x

[MaverickTse]

Very recently, there is a wave of interest among the Japanese community in developing a new algorithm for upscaling: waifu2x

The basic idea of this algorithm was developed in the Chinese University of Hong Kong, but only recently a Japanese implemented it in LUA and uploaded it to GitHub:

https://github.com/nagadomi/waifu2x
(... and it starts getting viral)

This original version use CUDA on Linux, and this seems to be problematic for many users...

Then shortly afterwards, people starts porting it to Windows, and with GUI. These have no CUDA acceleration however.

First Windows port (only upscaling, no training part):
https://github.com/WL-Amigo/waifu2x-converter-cpp

GUI All-in-one package:
http://inatsuka.com/extra/koroshell/
(Click on the floppy icon to download)

AVX optimization:
https://github.com/tanakamura/waifu2x-converter-cpp

AVX2 optimization:
https://github.com/logicmachine/waifu2x-opt

To use the GUI, launch waifu2x_win_koroshell.exe, then drag-and-drop your image file on to the main window.

Someone made an Avisynth port, but reportedly too slow to be of any use (0.02fps on Core i5):
http://waifu2x-avisynth.sunnyone.org/

I ported an OpenGL version to AviUtl:
https://github.com/MaverickTse/aviutl-waifu2x-gl

GUI CUDA Version (some what troublesome):
https://github.com/lltcggie/waifu2x-caffe/releases/tag/1.0.0
(download link at the bottom, ~130MB!)
This GUI is Japanese-only. I have made another multilingual GUI that wraps it. For details see post 41:
https://forum.videohelp.com/threads/372157-New-upscaling-algorithm-waifu2x?p=2423628&vi...=1#post2423628

Integration of AVX/FMA/CUDA/OCL...:
http://d.hatena.ne.jp/w_o/20150616#1434392833

AviUtl Plugin that use the above DLL:
https://github.com/YSRKEN/aviutl-waifu2x-w2xc
The release package already contains the DLL.
(DLL: http://d.hatena.ne.jp/w_o/20150619#1434643288)
↑At version 1.4.1, this should be the fastest AviUtl version if using with CUDA.
@640x360 -> 1280x720 takes ~3 sec on GTX970OC(GigaByte)
Note that earlier versions may have problem with CUDA acceleration.



VapourSynth Plugin(also use w2xc dll)
https://github.com/HomeOfVapourSynthEvolution/VapourSynth-Waifu2x-w2xc/


Here are some samples:

1. Fringe pattern. Left: Original 256x256 px. Right: Waifu2x double sized, no denoise. Quite accurate and no artifact IMO.


2. Japanese-style animation image (a capture from the PC game, Da Capo). You can try the doubling the image using conventional bicubic, sinc or spline, you should notice the edges in the waifu2x images are a lot crisper.

[racer-x]

It's very slow, but does a good job.

I don't understand Japanese, but it's a simple matter of drag and dropping the image into the program window. It saves new image as a PNG in the same folder as original.

[MaverickTse]

Some explanation on the GUI:

[MaverickTse]

For the CUDA-enabled GUI:


For my multilingual GUI wrapper:


Which is pretty self-explanatory

[MaverickTse]

To use the cuDNN acceleration, you need to sign-up the CUDA developer program on Nvidia's site and install the cuDNN package... quite troublesome.
It is fine to left out cuDNN IMO.

[themaster1]

I can't wait for a proper english translation of this gui and at least avi compression

[MaverickTse]

You can use the AviUtl plugin for the time being, if you don't mind a slow 0.1~0.2fps for upscaling from NTSC/PAL to double size.
Currently only the CUDA implementation may reach 1fps, but building that binary is as hard as building FFmpeg (and far less documentation).
There is a CLI version for the CUDA one, and of course in English.

[racer-x]

I'm not sure this would be a good option for video (too slow), but it is a great option for images. It would make a good free alternative to Genuine Fractals, for example.

[Cornucopia]

All those examples seem to be of limited palette/anime. Any real-world/natural images to compare?

Scott

[racer-x]

Here is a before / after on a jpg image I downloaded somewhere. The top one is the 2x upscaled version.

[Cornucopia]

Well, that's a very pleasant UPsize effect!

Slow as it is for video, there are a few projects where that could REALLY help/fix things. Good to know, thanks.

Scott

[newpball]

Neural networks for upscaling....

Okay, how is the network trained and how is it decided what looks go and what not for each training sample?

[Cornucopia]

Since very few of us here can read Japanese, that might be a little difficult to answer. Perhaps MaverickTse has some info...

Scott

[newpball]

Since very few of us here can read Japanese, that might be a little difficult to answer. Perhaps MaverickTse has some info...

Scott

Perhaps the noise suppression is neural...

Tried the waifu2x-cafe, works on jpg, crashes on png.

[racer-x]

Works here on PNGs. It gives some warning box that I can't understand, but click OK and it goes to work.

This program uses a seven step process, so it can take a long time depending on image size. The results are very good I think.

[El Heggunte]

And more importantly:
even though the GUI works well under Windows XP,
the command-line .EXE does not

[MaverickTse]

Sorry guys.
My main PC　got a mysterious short-circuit and the new rig will not be available until 13th June
(DDR4 RAM out-of-stock)

Limited support until then.

For how the algorithm actually works(in English), please visit the first GitHub link in the 1st post...
I cannot really answer that myself since I do not know what the heck is "convolution".
All i can tell is that everything works in YUV, and the heavy algorithm is only applied to Y. The UV planes are simple Bicubic resize.
Any magnification not 2X or 4X will involve extra shrinking step(area average or nearest neighbor)... so upscaling to 1.6X is actually slower than 2X

beware that on some images, waifu2x will create an excessively "oil-painting" feel, so that is why it is claimed to be designed for Anime-style images.

You can train the algorithm using your own image set, but you need a Nvidia rig on Linux, since only the Linux version has the trainig component.

Hopefully developers around the world will join the quest in making it faster...

[-Habanero-]

Neural networks are finally feasible? About goddamn time.
This looks very promising, for animation at least.

[foxmlg]

I upscaled the first episode of Death Note to 960p.. I cant provide any links here, but you can surely find it somewhere if you search for it.

[-Habanero-]

Upload a 30 second clip of it. I'd like to see it. How did you even convert a whole episode? It took me half a minute just to upscale ONE 480p image. A whole episode would take what, 10 days?

[foxmlg]

Upload a 30 second clip of it. I'd like to see it. How did you even convert a whole episode? It took me half a minute just to upscale ONE 480p image. A whole episode would take what, 10 days?

It took me 16h for a full 22 min episode.

2 sec per frame.

If anyone is interested in a download link, pm me, since its against the rules to post it here.

[pandy]

If anyone is interested in a download link, pm me, since its against the rules to post it here.

AFAIR 10 sec is allowed even under most strict Hollywood regulations - select few 10 sec parts.
I also try this one and i must say it works sometimes nice, sometimes not and yes it is slow but i don't expect speed from non optimized code - works more than good as a proof of concept.

https://www.youtube.com/watch?v=CJn_jC4FNDo

[MaverickTse]

Upload a 30 second clip of it. I'd like to see it. How did you even convert a whole episode? It took me half a minute just to upscale ONE 480p image. A whole episode would take what, 10 days?

It took me 16h for a full 22 min episode.

2 sec per frame.

If anyone is interested in a download link, pm me, since its against the rules to post it here.

2sec per frame !!!
which version did you use? and on what machine?

[foxmlg]

Upload a 30 second clip of it. I'd like to see it. How did you even convert a whole episode? It took me half a minute just to upscale ONE 480p image. A whole episode would take what, 10 days?

It took me 16h for a full 22 min episode.

2 sec per frame.

If anyone is interested in a download link, pm me, since its against the rules to post it here.

2sec per frame !!!
which version did you use? and on what machine?

This one: https://github.com/nagadomi/waifu2x

On Ubuntu 14.04 64Bit, with a GTX 770.

[-Habanero-]

The quality of your death note episode is not too good. It doesn't look any better than a substandard upscaler designed for cartoons.

I'm guessing you used a horribly inefficient version of waifu2x that allowed it to be that fast. Send me one original frame of your video and I'll see if the waifi i'm using will do any better.

[foxmlg]

The quality of your death note episode is not too good. It doesn't look any better than a substandard upscaler designed for cartoons.

I'm guessing you used a horribly inefficient version of waifu2x that allowed it to be that fast. Send me one original frame of your video and I'll see if the waifi i'm using will do any better.

Here is some original frames: http://imgur.com/a/R8xiX
And here is some upscaled frames: http://imgur.com/a/KHCTa

I used the one I said, it uses CUDA acceleration, thats why its fast. I also upscaled it 2x with 2x denoise. Maybe thats why it doesnt look as good as you want.
Also used the already trained waifu2x from the creator. Since I dont have enough ram to train it my self.
I heard that now they trained it better, it may yield better results now.

Also, if we were to train it extensively.. It would result in much better results..

[-Habanero-]

Those photos are 580p, show me the 960p ones.

[foxmlg]

Those photos are 580p, show me the 960p ones.

You must be blind, cus the first link is 480p, and the second link is 960p..

[-Habanero-]

The second link shows 1024x580 photos. https://forum.videohelp.com/images/imgfiles/kKIja2ah.png
Use a better image site.

[foxmlg]

The second link shows 1024x580 photos. https://forum.videohelp.com/images/imgfiles/kKIja2ah.png
Use a better image site.

Get a better computer, since the problem is with you.