Restoring an older VHS anime - "Rainbowing" and other various issues

[wubikens]

Hey,

I'm attempting to restore some footage for an older anime on VHS. I recently had some help solving a field blending issue, and now I'm trying to find the right denoisers, sharpeners, and smoothers to make the footage look a little better. I've been doing research and have found the following threads with some settings that I've tried:

https://forum.videohelp.com/threads/397357-90-s-Japanese-anime-VHS-clean-up-attempt#post2584187

https://forum.videohelp.com/threads/355366-anime-filtering-via-avisynth#post2235757
(minus deinterlacing stuff)

https://forum.videohelp.com/threads/362895-VHS-Anime-to-DVD-in-Virtualdub#post2306889
(minus deinterlacing stuff, looked pretty good but had a bit of a red hue on all white scenes)

https://forum.videohelp.com/threads/401326-Avisynth-filters-on-anime#post2615463
(this worked quite well for my video as well and made the colors pop a lot more)

I've also been following this guide:
https://www.animemusicvideos.org/guides/avtechbeta/avspostqual.html#rainbows

After trying various approaches, I tried coming up with a script with settings that I thought were the most helpful. Unfortunately, I'm still running into a few issues that I'm having trouble solving:

AVISource("source.avi")
ConvertToYV12()
ColorYUV(levels="TV->PC")
MCTemporalDenoise(settings="very high")
vmToon()

Pretty simple, but only because I have MCTemporalDenoise doing a lot of the heavy lifting. MCTD at "very high" is needed to reduce some "rainbowing" that I've been seeing in some scenes. I really don't like having this setting so high, I end up losing so much detail, but it's the only thing I've found that gets rid of that rainbowing. None of the other scripts I've tried, or any setting lower than "very high", seems to work. I've even tried using DFMDeRainbow with "mask=true" to help me determine which threshold value is needed (I ended up having to set it to "maskthresh=55"). I've also tried applying a sharpener (vmToon was recommended from that guide) to try and mitigate some of the blurriness from MCTD.

I'm feel like I'm going in circles and am making no progress, and thought about reaching out for help. I'll upload a couple clips, one daytime and one nighttime. I've found the rainbowing is most noticeable during the nighttime one, fwiw.

Thank you for any help!

[Selur]

looking at Day.avi
a. don't see rainbows
b. try adding the line darkening before applying MCTemporalDenoise
c. to me, the most annoying thing are the halos and the chroma shift
since it's an anime, and you are using Avisynth, I would suggest trying some of the avs-mlrt filters, since there are tons of filters that are trained to clean up animes.

Cu Selur

Ps: Quick&Dirty2 using 2x_LD-Anime-Compact for cleanup

[jagabo]

Regarding using MCTemporalDenoise to reduce chroma noise, you can apply it differently to the luma and chroma using MergeChroma():

light = MCTemporalDenoise(settings="low", chroma=false)
heavy =  MCTemporalDenoise(settings="very high")
MergeChroma(light, heavy) # luma from light, chroma from heavy

Or all on one line:

MergeChroma(MCTemporalDenoise(settings="low", chroma=false), MCTemporalDenoise(settings="very high"))

[wubikens]

looking at Day.avi
a. don't see rainbows
b. try adding the line darkening before applying MCTemporalDenoise
c. to me, the most annoying thing are the halos and the chroma shift
since it's an anime, and you are using Avisynth, I would suggest trying some of the avs-mlrt filters, since there are tons of filters that are trained to clean up animes.

Cu Selur

Ps: Quick&Dirty2 using 2x_LD-Anime-Compact for cleanup

Appreciate it! I didn't even realize AI ML was a thing for this. This is my first attempt at restoring something, and now that I feel like I have a somewhat decent understanding of the basics, I feel like this is a whole new thing to learn now.

I went ahead and tried using one of the models from this thread: https://forum.doom9.org/showthread.php?t=184768

The specific one I'm trying to use is 2x_LD-Anime_Skr_v1.0.onnx (it's in the same folder as the mlrt.dll). Here's my script right now:

AVISource("source.avi")
ConvertToPlanarRGB()
mlrt_ncnn(network_path="2x_LD-Anime_Skr_v1.0.onnx")

But it's giving an error when I try to open it in Vdub:

Avisynth open failure:
mlrt_ncnn: expects clip with type fp32

I've tried looking up info on the error, and I can't seem to find anything. From what I can see, fp32 is the data type of the model? Would you happen to know how I could go about solving this error?

Regarding using MCTemporalDenoise to reduce chroma noise, you can apply it differently to the luma and chroma using MergeChroma():

light = MCTemporalDenoise(settings="low", chroma=false)
heavy =  MCTemporalDenoise(settings="very high")
MergeChroma(light, heavy) # luma from light, chroma from heavy

Or all on one line:

MergeChroma(MCTemporalDenoise(settings="low", chroma=false), MCTemporalDenoise(settings="very high"))

Oh! I didn't realize I could do that, I'll go ahead and give it a try, thank you!

[jagabo]

Regarding fp32, see ConvertToFloat()
http://avisynth.nl/index.php/ConvertToFloat

ConvertToPlanarRGBA()
ConvertToFloat()

Or use z_ConvertFormat()
http://avisynth.nl/index.php/Avsresize#z_ConvertFormat

[wubikens]

Regarding fp32, see ConvertToFloat()
http://avisynth.nl/index.php/ConvertToFloat

ConvertToPlanarRGBA()
ConvertToFloat()

Or use z_ConvertFormat()
http://avisynth.nl/index.php/Avsresize#z_ConvertFormat

Awesome! Thank you so much, that helped solve the issue. I also spent all day yesterday messing around with different models, and "2x_LD-Anime_Skr_v1.0" produced incredible results. I was baffled at how good it looked.

One issue that I noticed though, is that in my source, there's something that looks similar to haloing, but it isn't haloing. I'm not exactly sure what the term to describe it is (is this the chroma shift that Selur was referring to?). But basically, it looks like some of the color bleeds outside the lines, and then LD-Anime ends up enhancing it so that it becomes more obvious and prominent. I'll uploaded a couple of pictures and a short 2 second clip.

[Selur]

There are halos all around, but that looks more like some blurring, blending, ghosting. That is not a chroma shift.
LGhost(clip, mode=[1,1,1], shift=[3,4,5], intensity=[60,60,60]) and DeHalo_alpha(): https://imgsli.com/MjYwMDc5 help somewhat and do not seem to mess too much up.

Cu Selur

[wubikens]

There are halos all around, but that looks more like some blurring, blending, ghosting. That is not a chroma shift.
LGhost(clip, mode=[1,1,1], shift=[3,4,5], intensity=[60,60,60]) and DeHalo_alpha(): https://imgsli.com/MjYwMDc5 help somewhat and do not seem to mess too much up.

Cu Selur

Oh sweet! Yeah that looks a lot better, thank you! If I may ask, I've been messing around with Dehalo_alpha and FineDehalo, and according to the docs, it seems FineDehalo is an upgrade to the former? Is there any difference really between the two?

Also, I was wondering if I could ask too, once the VsLGhost filter is applied, in place of that ghosting/blending/blurring issue, it seems to leave behind something more akin to a really strong halo in its place (it's more like a really light spot where the ghosting was at) which I can't seem to scrub away with the Dehalo filters. Is that something that can be removed? Or is it probably going to have to be a compromise, between the former problem and this one? I'm totally okay with experimenting with some other filters to try and remove it, I'm just not sure what filter would do the job, or if it's possible (it sucks being such a novice at this that I can't even identify what kind of problem is occurring lol). Thanks again!

Edit: Mostly asking about that light spot because LD-Anime ends up enhancing it, too.

[Selur]

Is there any difference really between the two ?

FineDeHalo(2) uses DeHalo_alpha with a few masks and optional contra-sharpening, it tries to remove halos without removing important details.
So depending on your settings, it can so more than DeHalo_alpha.

Is that something that can be removed? Or is it probably going to have to be a compromise, between the former problem and this one?

I would try some of the machine learning filters, but you might not be able to remove the effect completely.
The best way would probably to come up with some sort of mask to only apply LGHost just on some parts of the frame. (maybe some sort of inverted edge mask could do the trick; to that the effect of LGHost is not applied on regions near edges,...)

In general:
a. each filter has side effect. The challenge is to find the balance between the effect you want and the effects you don't want.
b. each filter destroys information. The challenge is to only destroy the information you don't want. (destroy noise vs keep details)
=> you lost when you started

it sucks being such a novice at this that I can't even identify what kind of problem is occurring

Sometimes not knowing what the problem is can be a blessing, especially if you know the problem, but no way to fix it.


Cu Selur

[wubikens]

Is there any difference really between the two ?

FineDeHalo(2) uses DeHalo_alpha with a few masks and optional contra-sharpening, it tries to remove halos without removing important details.
So depending on your settings, it can so more than DeHalo_alpha.

Ah okay, gotcha! Makes sense, thanks!

Is that something that can be removed? Or is it probably going to have to be a compromise, between the former problem and this one?

I would try some of the machine learning filters, but you might not be able to remove the effect completely.
The best way would probably to come up with some sort of mask to only apply LGHost just on some parts of the frame. (maybe some sort of inverted edge mask could do the trick; to that the effect of LGHost is not applied on regions near edges,...)

In general:
a. each filter has side effect. The challenge is to find the balance between the effect you want and the effects you don't want.
b. each filter destroys information. The challenge is to only destroy the information you don't want. (destroy noise vs keep details)
=> you lost when you started

Definitely, yeah. I like the LD-Anime model, but it does exacerbate a lot of underlying issues that aren't as visible in the source since the source can hide it with grain, noise, etc.

Another model I found that works just as good as LD-Anime, but preserves the "grainy"/original look a whole lot better is "AnimeClassics UltraLite." Because it preserves the "grainy"/original look better, the underlying issues are hidden a lot better than in LD-Anime. So I might go with that one instead.

it sucks being such a novice at this that I can't even identify what kind of problem is occurring

Sometimes not knowing what the problem is can be a blessing, especially if you know the problem, but no way to fix it.

Cu Selur

That's true! "Ignorance is bliss."

Thank you so much again for your help!

Edit: Also funny thing, without realizing it, I found a post you made from here back in 2022:

https://forum.videohelp.com/threads/405747-Restoring-the-hellhole-that-is-the-Evangeli...16#post2656948

And I was also doing Dehalo_alpha + Toon + LD-Anime lol

[jagabo]

For what it's worth, here's something a little different for removing that ghost to the right of the dark areas:

LWLibavVideoSource("birds.avi", cache=false, prefer_hw=2) 
ConvertToYV12()

Spline36Resize(400, height) # closer to the true resolution of VHS
Overlay(last, last, x=3, mode="subtract", opacity=0.2).ColorYUV(gain_y=60) # subtract ghost
dehalo_alpha(rx=2.5, ry=2.0, darkstr=1.2, brightstr=1.2) # then halo reduction

# sharpen and upscale back to the original frame size
aWarpSharp2(depth=5)
CAS(0.7)
nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=720, fheight=480)

before and after:
[Attachment 78740 - Click to enlarge]

[wubikens]

For what it's worth, here's something a little different for removing that ghost to the right of the dark areas:

LWLibavVideoSource("birds.avi", cache=false, prefer_hw=2) 
ConvertToYV12()

Spline36Resize(400, height) # closer to the true resolution of VHS
Overlay(last, last, x=3, mode="subtract", opacity=0.2).ColorYUV(gain_y=60) # subtract ghost
dehalo_alpha(rx=2.5, ry=2.0, darkstr=1.2, brightstr=1.2) # then halo reduction

# sharpen and upscale back to the original frame size
aWarpSharp2(depth=5)
CAS(0.7)
nnedi3_rpow2(2, cshift="Spline36Resize", fwidth=720, fheight=480)

before and after:
[Attachment 78740 - Click to enlarge]

You. Are. A. Life. Saver. This worked wonderfully, thank you so much! Between this and your help with the QTGMC Srestore, you've seriously helped me a ton. Thank you again!

[jagabo]

Glad to help out.