Anime Upscale Script Improvement

[Osram123]

I'm trying to upscale a DVD quality anime into a HD anime with AviSynth. It indeed improved the quality, but I think it can be improved more.

First, take a look at the improvement.

DVD Quality (4:3, 720x480) --> Upscaled Quality (4:3, 1440x1080)

Example #1 http://screenshotcomparison.com/comparison/168394
Example #2 http://screenshotcomparison.com/comparison/168395
Example #3 http://screenshotcomparison.com/comparison/168396
My Script VS Waifu2x http://screenshotcomparison.com/comparison/168398

There's some problems I'm currently facing:
1. If I Sharpen more, the credits in Example #3 will look very awful.
2. If I don't Sharpen more, the frame will look dull, like DVD Quality.
3. I can't Denoise & Degrain any further with my script, any ideas?
4. It still look quite dull compared to the real Blu-ray of the anime, how can I make it brighter and more colorful?
5. I can't smooth out the jagged lines, you can spot it in Example #1.

DBZ.mp4.avs

LoadPlugin("C:\Program Files (x86)\MeGUI_2624_x86\tools\lsmash\LSMASHSource.dll")
LoadPlugin("C:\Program Files (x86)\MeGUI_2624_x86\tools\avisynth_plugin\nnedi3.dll")
LoadPlugin("C:\Program Files (x86)\MeGUI_2624_x86\tools\avisynth_plugin\masktools2.dll")
LoadPlugin("C:\Program Files (x86)\MeGUI_2624_x86\tools\avisynth_plugin\RemoveGrain.dll")
LoadPlugin("C:\Program Files (x86)\MeGUI_2624_x86\tools\avisynth_plugin\mvtools2.dll")
LoadPlugin("C:\Program Files (x86)\MeGUI_2624_x86\tools\avisynth_plugin\Repair.dll")
LoadPlugin("C:\Program Files (x86)\MeGUI_2624_x86\tools\avisynth_plugin\ColorMatrix.dll")
LoadPlugin("C:\Program Files (x86)\MeGUI_2624_x86\tools\avisynth_plugin\aWarpSharp2.dll")
LoadPlugin("C:\Program Files (x86)\MeGUI_2624_x86\tools\avisynth_plugin\flash3kyuu_deband.dll")
Import( "C:\Program Files (x86)\MeGUI_2624_x86\tools\avisynth_plugin\AnimeUpscale.avsi" )
Import( "C:\Program Files (x86)\MeGUI_2624_x86\tools\avisynth_plugin\FastLineDarkenMod.avsi" )
LSMASHVideoSource("C:\Users\microsoft\Desktop\DBZ.mp4").animeupscale()

Actually there's brackets after ".animeupscale", I don't know why it disappear in the code.

AnimeUpscale.avsi

function mDegrainSimple(clip, int "frames", int "blksize") {
originalvideo = clip

super = MSuper(originalvideo, pel=2, sharp=1)

backward_vec3 = (frames==3) ? MAnalyse(super, isb = true, delta = 3, blksize=blksize, overlap=2) : super
backward_vec2 = (frames>=2) ? MAnalyse(super, isb = true, delta = 2, blksize=blksize, overlap=2) : super
backward_vec1 = MAnalyse(super, isb = true, delta = 1, blksize=blksize, overlap=2)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, blksize=blksize, overlap=2)
forward_vec2 = (frames>=2) ? MAnalyse(super, isb = false, delta = 2, blksize=blksize, overlap=2) : super
forward_vec3 = (frames==3) ? MAnalyse(super, isb = false, delta = 2, blksize=blksize, overlap=2) : super
mvvideo = (frames==3) ? MDegrain3(originalvideo, super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,backward_vec3,forward_vec3,thSAD=400) : 

(frames==2) ? MDegrain2(originalvideo, super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400) : MDegrain1(originalvideo, super, 

backward_vec1,forward_vec1,thSAD=400)

return mvvideo
}

function LSFModLight( clip input, int "strength" ) {
ox = input.width
oy = input.height
strength    = default( strength,    100 )

xxs  = round(ox*1.25/8)*8
yys  = round(oy*1.25/8)*8

str  = float(strength)/100.0

dark_limit   = input.mt_inpand(U=1,V=1)
bright_limit = input.mt_expand(U=1,V=1)
minmaxavg    = mt_average(dark_limit,bright_limit,U=1,V=1)

method = minmaxavg
method = method.mt_lutxy(input,"x y < x 1 + x y > x 1 - x ? ?",U=1,V=1)

normsharp = mt_lutxy(input, method, yexpr="x x y - "+string(str)+" * +",U=1,V=1)

normal = mt_clamp(normsharp, bright_limit, dark_limit, strength/100, strength/100, U=1, V=1)
second = mt_clamp(normsharp, bright_limit, dark_limit, (strength/100)*2, (strength/100)*2, U=1, V=1)
zero   = mt_clamp(normsharp, bright_limit, dark_limit, 0, 0, U=1, V=1)

limit1 = normal
limit2 = limit1

sharpdiff = mt_makediff(input,limit2,U=1,V=1)
sharpdiff = mt_lutxy(sharpdiff,sharpdiff, "x 128 - abs y 128 - abs > y "+string(0)+" * x "+string(100-0)+" * + 100 / x ?",U=1,V=1)

diff  = mt_makediff(input,limit2,U=1,V=1)
diff2 = diff.temporalsoften(1,255,0,32,2)
diff3 = mt_lutxy(diff,diff2,"x 128 - y 128 - * 0 < x 128 - 100 / 20 * 128 + x 128 - abs y 128 - abs > x 20 * y 100 20 - * + 100 / x ? ?",U=1,V=1)

PP = mt_makediff(input,diff3,U=1,V=1)

out = PP.mergechroma(input)
src = input
in = input

shrpD  = mt_makediff(in,out,U=1,V=1)
shrpL  = shrpD.repair(mt_makediff(in,src,U=1,V=1),1,-1,-1).mt_lutxy(shrpD,"x 128 - abs y 128 - abs < x y ?",U=1,V=1)
output = out
return output
}

function AnimeUpscale(clip, string "ResizeMethod") {

ResizeMethod = Default(ResizeMethod, "spline36resize")

#Scales to PC Levels for Filtering
videoSD = clip.Levels(16, 1, 235, 0, 255, coring=false)

#RemoveGrain
videoSD = videoSD.removegrain(2).flash3kyuu_deband()

#Grain and Noise Removal
videoSD = videoSD.mDegrainSimple(frames=3,blksize=4).mDegrainSimple(frames=3,blksize=4)

#Sharpening SD
videoSD = VideoSD.lsfmodlight(strength=20).fastlinedarkenmod(Strength=30,thinning=10)

#Upscale the video to HD
videoHD = Eval("videoSD." + ResizeMethod + "(1440,1080)")

#Sharpening HD
videoHD = VideoHD.lsfmodlight(strength=100).fastlinedarkenmod(strength=100,thinning=15)

#Edge Enhancement
videoHD = videoHD.awarpsharp2(blur=2, depth=12)

#Debanding
videoHD = videoHD.flash3kyuu_deband()

#Scales back to TV Levels for Output
VideoHD = videoHD.Levels(0, 1, 255, 16, 235, coring=false)

#Convert ColorMatrix
VideoHD = VideoHD.ColorMatrix(mode="Rec.601->Rec.709")

return VideoHD
}

I like how Waifu2x upscale the anime, it's clean, the Denoise is perfect, but it can't let you sharpen and darken the lines more, and most important, it's so freaking slow!!! Like 30 sec per frame. I think my script over-sharpen the anime a little bit, I'm not sure because I've been comparing the quality all the day. I'm very appreciate any suggestions you made, I really want to make an Anime Upscaler better than Waifu2x

[LigH.de]

You forgot a few ideas which will probably have been discussed in more dozens of "Super Duper Ultra Anime Script" threads in more dozens of video forums, like:

a) First upscale larger than desired with Edge Directed Interpolation, e.g. using nnedi3_rpow2, before scaling down to the target dimensions.

b) NLMeans (in different implementations) can be a powerful noise killer, useful to iron out flat colored areas in cartoons.

Unfortunately, my remark will only increase the complexity of this thread, but not necessarily lead to an optimal result.