how to deal with posterization in anime animation

[zanzar]

i have anime dvds and i am trying to better their quality (i know some people hate this kind of thing here),and they will be posted and shared on a site.
i know you cant add detail (for that you need to hand draw each frame to add detail),and its NOT what i am trying to achieve.
since animation is so fundamentally different from video recordings,so also what applys to them is different.
most times animation (with emphasis on anime) is just different colors seperated by black lines (or different color scemes).
now ill seperate it to 2 categories.

1.close up drawings, in which you see most of the detail (lets say some charecter is talking and his size takes most of the frame)
2.far away drawings, lets say seeing a crowd of pople each drawn like he is far away.

since the source is DVD and only 480x720 in resolution, the detail from afar will be much less, well detailed.
90% of the animation is number 1 (generally i think animators know that so they avoid as much as they can number 2)


for example
1.a close drawing dvdrip vs my attempt to better it with filters LINK
2.here is what ill consider a far drawing LINK
(ill note that this is relatively bad comparison, and most times the result is better with different dvds i tested,i choose this because i have a blu-ray of it to compare to)

here is the bluray compared to the dvd LINK1 , LINK2

imo (and many others i asked) the attemt actually makes it more plesent to watch,the animation looks smoother and more clear,and there is much less noise or artifacts (its one thing to just compare shots and other to look at the whole result run).

how i do it in short.
rip the source into frames with a script that cleans it a bit and sharpens the lines.
use waifu2x-caffe on them,then use linedarkening scripts like "hysteria" and increse the levels if the colors are to washed out then resize it to 720p and recombine them back into a video file.

now lets get to the point.
if i increse the levels (even some times if i dont) it causes the posterization become more visble and it varies in its strength (in different sences)
i uploaded a smaple for comparison

1.original

2.processed with the process i told you and this script

ImageSource("D:\pic\pic%06d.jpg", fps=23.976)

converttoyv12()
ColorMatrix(mode="rec.601->rec.709")
levels(30, 0.85, 246, 5, 255, coring=false)
tweak(sat=1.3, coring=false)
Hysteria(strength=1.0, maxchg=30,lowthresh=8)
Sharpen(0.25)
Hysteria(strength=1.0, maxchg=35,lowthresh=6)
Blur(0.35)
Hysteria(strength=1.0, maxchg=30,lowthresh=4, highthresh=60)
Blur(0.35)
Dehalo_alpha(rx=3, ry=3, darkstr=0.6, brightstr=0.6)
nnedi3_rpow2(2, cshift="spline64resize", fwidth=1280, fheight=720)

notice the posterization going crazy aroud his shirt collar

3.my attempt in fixing the posterization.

converttoyv12()
ColorMatrix(mode="rec.601->rec.709")
smoothlevels(30, 0.9, 250, 5, 255)
Hysteria(strength=1.0, maxchg=30,lowthresh=8)
Sharpen(0.25)
Hysteria(strength=1.0, maxchg=35,lowthresh=6)
Blur(0.35)
Hysteria(strength=1.0, maxchg=30,lowthresh=4, highthresh=60)
Blur(0.35)
nnedi3_rpow2(2, cshift="spline64resize", fwidth=1280, fheight=720)
McTemporalDenoise(settings="medium")
GradFun3 ()
Dehalo_alpha(rx=3, ry=3, darkstr=0.5, brightstr=0.5)

also added 2 other samples- 1.with the first script 2.second with an attempt to fix it(didnt add the original)

i found out that "McTemporalDenoise" can take care of almost anything i throw at it (including this)the only problem it is slow. more so in 720p
if i try to increase the levels before the waifu2x, the source looses a lot of detail.

i am here to hear your suggestions on how to minimize the posterization (as you saw it can be minimized to berable extent).
also i am looking to by a new cpu.
does a multi threaded avisyth works well? (lets say 8 cores 16 threads) (i have 4 cores 4 threats...)virtualdub\Megui

[Bernix]

Deleted then. I thought you need bilateral filter to remove the posterisation.

[zanzar]

Hi,
I'm totally noob in avisynth, but for such think in picture manipulation is very good is bilateral method. Is this something you want achieve? Your post is very long and my english very poor. If so try Gradfun with bilateral method.
[Attachment 44562 - Click to enlarge]

well,i forgot to upload the sample Zip.
fixed it now.

i didnt understand your quastion and what you ment by posting part of the pic. what i generally try to get is more "clean" animation.
by eliminating noise and artifact and making the lines more firm and distinct.

the problem is that the process i do some times either causes or stresses out already present posterization (because of levels() increse)
and i am trying to figure out how to get rid of it.
you can download the samples and see
video - 1 its the original
video - 2 you can see posterization around the shirt collar
video - 3 i tried fixing it with the last script

and i want to hear out more about this thing, how to avoid it or to take care of it

edit:also you cant notice the posterization in the pics only in the video samples

[jagabo]

TTempSmooth() can lock the pixels down but may create smearing artifacts in panning shots if you set it too high and there will still be posterization. So follow it with GradFun3().