fixing standard noise (as well as MOSQUITO noise) issues with anime

[tomotakino]

I am trying to restore azumanga daioh by upscaling it with ESRGAN, but there is one annoying problem occurring, that being the weird noise present in the video stream. How do I fix it?
dont mind the file name, that was the first thing that came to my mind as "script.avi" had already been used.

also is there a way to fix the weird aliasing?

[tomotakino]

I've used a high bitrate for the render because I suck at mpeg2 muxing. Heres the mosquito noise blown up from the original mpeg2 file.



the upscaler sometimes treats em like details which is, after the process, insanely hard to clear up.

[jagabo]

Santiag() for the aliasing.

Upload the original MPEG2 video, not something you've processed and reencoded.

[Selur]

Santiag does seem to work fine for anti-aliasing.
No clue what 'weird noise' you mean. If you mean the compression artifacts in flat areas DPIR deblock or some masked deblocking or strong debanding could clean those up. Problem is, you will most likely lose some fine details in the background.
For example:
loosing small details: https://imgsli.com/MTM2MzYw
general effect: https://imgsli.com/MTM2MzYx
script: https://pastebin.com/Bqk2u5YF

Cu Selur
Ps.: jagabo was faster
Pss.: BasicVSR++ will clean up the flat areas even more, but it also loses more small details.

[tomotakino]

Santiag does seem to work fine for anti-aliasing.
No clue what 'weird noise' you mean. If you mean the compression artifacts in flat areas DPIR deblock or some masked deblocking or strong debanding could clean those up. Problem is, you will most likely lose some fine details in the background.
For example:
loosing small details: https://imgsli.com/MTM2MzYw
general effect: https://imgsli.com/MTM2MzYx
script: https://pastebin.com/Bqk2u5YF

Cu Selur
Ps.: jagabo was faster
Pss.: BasicVSR++ will clean up the flat areas even more, but it also loses more small details.

sent a bad sample, there were some other scenes with higher amounts of noise (like this one https://forum.videohelp.com/images/imgfiles/EHzSoSO.png) . nevertheless thanks for those scripts

pretty sure it was the problem with the source as I had been alternating between couple of different releases of azumanga daio on dvd

[tomotakino]

Upload the original MPEG2 video, not something you've processed and reencoded.

well the only thing that i've done was IVTCing the clip as well as encoding it in extremely high quality h264.

[jagabo]

Have you tried using Mpeg2Source's deringing filters? Hence the request for the original MPEG 2 stream.

[tomotakino]

here @jagabo

[jagabo]

Try building an index file with DgIndex (included with DgMpgDec) then using Mpeg2Source("cut.d2v", CPU2="ooooxx", Info=3) to open the MPEG2 source. That invokes its DCT deringing filters and will significantly reduce those artifacts. You must use the 32 bit version of DgMpgDec as the 64 bit versions don't implement the deblocking and deringing filters (at least the ones I've seen).

If that's not enough you can try adding a light denoising like KNLMeansCL(d=0, a=2, s=2, h=1.0, channels="Y"). That will blur some low contrast detail though (like the streaky blue sky at the start of the clip). KNLMeansCL is pretty good at retaining sharp edges while blurring away noise.

[poisondeathray]

Have you tried ESRGAN models that have built in denoising, or that were trained with slightly noisy anime sources?

Alternatively, RealESRGAN AnimeVideo-v3 will produce clean results, clean lines (maybe too clean textures for certain people's tastes, but you can mix models and composite/layer results)

[killerteengohan]

That looks more like compression artifacts to me. From the image though, sometimes this helps out a bit with the ones right along the lines in anime and cartoons.

hqdering(smoother=deen("c3d",1,5,9,2)) # Raise the 5 into a 7, or 9 if you need stronger cleaning (5 is usually plenty)

That should clean what you are asking about really well.


Try this out. I would do this before any upscaling if possible. This will be less effective if you do it after scaling.

source = last
dehalo_alpha(rx=1.1, ry=1.1, darkstr=0, lowsens=0, highsens=100, ss=1.0)
hqdering(smoother=deen("c3d",1,5,9,2))
MergeChroma(last, source)
xaa(mode="di2 nnedi3", mask=0, chroma=0, nns=4, cstr=0, mthr=24, mtype="TEMmod1")

Santiag can be pretty rough on details and lines, and can halo the video a bit at times, so I suggested xaa.

I gave kind of extreme AA settings that are kind of overkill since I did not have the original footage to examine for best settings. Ideally if you can get antialiasing good enough with these settings, you will preserve much more detail and soften things less.

xaa(mode="di nnedi3", mask=1, chroma=0, nns=4, cstr=0, mthr=24, mtype="TEMmod1")

If that's not enough change di into di2. If that's still not enough, change di2 back into di then set the mask to 0.


You can see how well it usually works at cleaning what you asked here in this before and after comparison.
(Use mouse click or keyboard arrows to swap comparison images)
https://slow.pics/c/Sc6ihaQ1

[tomotakino]

Will try.