Restore/upscale Poorly Mastered Anime DVD -- Please help!!

[laserbolt5]

Hi! So I have been trying to restore/remaster and hopefully upscale this really terribly mastered anime DVD on and off for about 5 years now... No matter what I do, I haven't been able to give it a proper restoration. As I understand it, the main underlying problems are:

1. Bad compression artifacting/grain. It seems to my eyes that the original 2004 source had blocky compression artifacting and nasty banding to begin with, and whoever mastered the DVD tried to hide it with artificial grain. Every time I attempt to denoise the source, I get ugly cubes or blobs that shift around solid colored surfaces, although the painted backgrounds seem to denoise fairly well. My only option so far is leaving more grain, which in turn harms the upscale...

2. Terrible interlacing/inconsistent framerate. This source will go anywhere from seeming to be progressive, to interlaced, and oftentimes a single shot will be interlaced at a completely different (although constant) framerate than everything else. When I go into mediainfo with the DVD, the video reports a constant 24000/1001 framerate while the audio reports as 31.250 FPS, so clearly the framerate changes are hardcoded into the video stream (and the audio is somehow encoded differently?). Needless to say, no amount of IVTC cracks the code. Even my continued attempts with QTGMC have had limited success (it blurs and combs the lines more than it repairs them, leaving low framrate shots still ugly and generally worsening the compression artifacting overall.)

3. Minor dot crawl. Thankfully, Tcomb seems to clean this up quite well.

4. Soft lines. From what I can tell, the DVD itself might be an upscale, as it looks to me like the animation was likely smaller to begin with and upscaled to 480p. As a result I get my best results upscaling the source when I Tcomb and deintertance, then downscale debilinear(624, 416), and finally upscale. However, this is more speculation on my part than fact...

5. Faint but large haloing. Dehalo_alpha fights it well, at the cost of blurring the already somewhat blurred lines. Although for my purposes right now, this isn't as much of an issue, as I describe below.

For upscaling, I have used nnedi3 for the past few years but recently have found much better results with a custom ESRGAN model, even though it takes forever. Thankfully, ESRGAN completely removes the haloing--yay! But it has no idea what to do if I leave grain in

I've included examples of the original DVD extracted with DGIndex below. If possible, I would love to at the very least repair the interlacing properly, and hopefully fix the compression noise issue to get smooth, non-bloby surfaces on characters. I've reached my wits end a bit here... Any help would be SO appreciated!

https://drive.google.com/drive/folders/1Qk-3usRYP_q-SczOBWcRt1Obtk4rbDs1?usp=sharing

[poisondeathray]

2. Terrible interlacing/inconsistent framerate. This source will go anywhere from seeming to be progressive, to interlaced, and oftentimes a single shot will be interlaced at a completely different (although constant) framerate than everything else. When I go into mediainfo with the DVD, the video reports a constant 24000/1001 framerate while the audio reports as 31.250 FPS, so clearly the framerate changes are hardcoded into the video stream (and the audio is somehow encoded differently?). Needless to say, no amount of IVTC cracks the code. Even my continued attempts with QTGMC have had limited success (it blurs and combs the lines more than it repairs them, leaving low framrate shots still ugly and generally worsening the compression artifacting overall.)

audio does not really have a "framerate"

You probably have to use manual overrides.

For example, that section "Framerate wonky_interlacing.m2v" is really 12fps animation (there is only 12fps worth of unique frames information). You can make those sections VFR, or duplicate them to 24p or 23.976p

tfm(pp=0)
tdecimate(mode=7, rate=12)
#changefps(24000,1001)


Are there any main content with 23.976p real information, or 29.97, or 59.94 ? or other framerates ? The titles/credits might probably different , they often are

EDIT: yes, the "Noise banding.m2v" is 23.976p actual content. So you'd probably have to use overrides for sections

[barty777]

We're developing an AI software that deals with various problems that affect older footage.

This is the result after processing: https://drive.google.com/drive/folders/1DLNiw9GdkUGAU79W7JKlZLgHvrPUL2bn?usp=sharing

Let me know what you think.

[laserbolt5]

Thanks so much for the feedback!

tfm(pp=0)
tdecimate(mode=7, rate=12)
#changefps(24000,1001)

So I gave a crack at the code you suggested poisondeathray, and... it didn't work. But it was very helpful as I was chugging away this week because it led me to discovering new things about the source...

So from what I can tell, the original Japanese release was interlaced (I have a short promo clip from the JP movie release), and the team who made the American release did a poor IVTC on it. Also, it seems like the 12fps shots were really mangled by the bad IVTC, and I've been able to salvage them by running AnimeIVTC mode=3 using QTGMC as an external bobber, rather than internal, to modify settings and preserve detail:

function QTGMC_custom_presets(clip input)
{
input.QTGMC(Preset="placebo", tr2=1, tr0=0, tr1=0, SourceMatch=3, Lossless=2, matchpreset="placebo", matchpreset2="placebo", sharpness=0.0, ezkeepgrain=1.0)
}
AnimeIVTC(Mode=3, omode=2, ifade=true, cbob=5, extbob=QTGMC_custom_presets)

But this only works for the 12fps shots, as they seem to produce inverted bobbing when run without AnimeIVTC (and vice versa; the 24fps shots bob improperly when using AnimeIVTC), so unless there's something I'm missing it will likely still be manual overrides for those shots.

I have managed to discover the root of the denoising posterization problem though; the grain is WAY overcompressed unless I encode it in x264's lossless crf=0 prior to upscale (even crf=15 murders the fine grain).

We're developing an AI software that deals with various problems that affect older footage.

This is the result after processing: https://drive.google.com/drive/folders/1DLNiw9GdkUGAU79W7JKlZLgHvrPUL2bn?usp=sharing

Let me know what you think.

Thanks for sharing barty777, but I'm not sure that's quite what I'm aiming for... it seems at this stage to blur a fair bit more than it reveals, and the overcompression/haloing is magnified. What footage have you been training the AI on?

As for the progress on the upscale itself, I've been refining my ESRGAN model with more iterations, and with a final pass of Temporaldegrain2 and AddgrainC, I've been really happy with the outcome... even though it takes forever to upscale lol (~5s per frame). Below are couple upscale tests I did with the progressive clips I uploaded; I think it is really starting to come together!

Once I get this whole thing properly deinterlaced, hopefully I'll be in the clear! Thank you both for the help!

[poisondeathray]

So I gave a crack at the code you suggested poisondeathray, and... it didn't work. But it was very helpful as I was chugging away this week because it led me to discovering new things about the source...

So from what I can tell, the original Japanese release was interlaced (I have a short promo clip from the JP movie release), and the team who made the American release did a poor IVTC on it. Also, it seems like the 12fps shots were really mangled by the bad IVTC, and I've been able to salvage them by running AnimeIVTC mode=3 using QTGMC as an external bobber, rather than internal, to modify settings and preserve detail:

function QTGMC_custom_presets(clip input)
{
input.QTGMC(Preset="placebo", tr2=1, tr0=0, tr1=0, SourceMatch=3, Lossless=2, matchpreset="placebo", matchpreset2="placebo", sharpness=0.0, ezkeepgrain=1.0)
}
AnimeIVTC(Mode=3, omode=2, ifade=true, cbob=5, extbob=QTGMC_custom_presets)

But this only works for the 12fps shots, as they seem to produce inverted bobbing when run without AnimeIVTC (and vice versa; the 24fps shots bob improperly when using AnimeIVTC), so unless there's something I'm missing it will likely still be manual overrides for those shots.

What do you mean it doesn't work ? What source filter are you using ?

I suggested you use that on the 12fps sections, because it returns the original 12fps film frames (that's what you have for those sections)

AnimeIVTC + QTGMC will return jerky cadence (it's not 12fps with duplicates, step through the result) , and artifacted frames with aliasing, and warping artifacts from QTGMC in those interpolated frames . And those artifacts will get magnified if you're upscaling. Moreover, the original frame set is degraded and more blurry from QTGMC . You've been doing this off and on for 5 years, you might as well do it properly IMO

Personally I would make this VFR, so the framerate is perfect for each section. You have fewer frames to upscale. Fewer artifacts, less blurring

[poisondeathray]

"Noise+Banding-2xTetsujin28v4.0.7_1080p.mkv" has ghosting artifacts, and is 23.000 fps. For the ghosting, not sure if that was from post processing or pre processing, could be from QTGMC, but there is quite a bit of detail loss on some frames - some lines are blurred or ghosted away. The source was clean

It looks like the model you're using shifts the colors; I don't know if that was intentional. It's not a simple 601/709 shift . The original has a more green hue to the clips

[laserbolt5]

What do you mean it doesn't work ? What source filter are you using ?

I've been generally using D2Vsource within Staxrip 2.1.7.0 and its portable version of AVS+, using .d2v files from DGIndex, although I have also tried with MPEG2Source and (briefly) FFVideoSource. I've also tried running the code with MPEG2Source outside of the Staxrip/AVS+ environment, just using 32-bit Avisynth and x264, and the same result. I would like to think it's just a missing dependency, but... I have included the result below. An interlaced frame seems to jump in every so often, and the result is very repeatable.

I mention the poor IVTC job done on the original because often a single frame (especially after a cut) will be interlaced as well, when the rest of the footage is progressive. I can include a sample if it would help.

Truly, if there is a way I can avoid using QTGMC to get properly progressive frames, I would love to take it.

"Noise+Banding-2xTetsujin28v4.0.7_1080p.mkv" has ghosting artifacts, and is 23.000 fps. For the ghosting, not sure if that was from post processing or pre processing, could be from QTGMC, but there is quite a bit of detail loss on some frames - some lines are blurred or ghosted away. The source was clean

My bad on the 23fps, when I run the video clips through Cupscale with ESRGAN it speeds up the framerate from 24000/1001 to 23.98fps, so I always start the post-processing script with AssumeFPS -- I accidentally wrote AssumeFPS(24000/1001) rather than AssumeFPS(24000, 1001).

My current post-processing script is

TemporalDegrain2(degrainTR=1, postFFT=3, postSigma=3, meAlg=5) #Noise+Banding I sent in didn't include meAlg=5, I have been continuing to tweak the settings a bit to account for frame-to-frame blur
Gradfun3(thr=0.25)
AddGrainC(var=1.0, uvar=0.0, hcorr=0.0, vcorr=0.0, seed=6, constant=false, sse2=true)
nnedi3_rpow2(2, cshift="Spline64Resize", fwidth=1920, fheight=1080) #ESRGAN model only upscales (well) to 960p non-anamorphic, so this finalizes the frame to 1080p

Here's Noise+Banding after the upscale, no post-processing in AVS, just aspect ratio and fps correction from the MKV container. Like I mentioned, it acts a funny way with grain, likely due to the inpainting element of ESRGAN (although this shot showcases the grain highlighting artifacts the most intense out of any I've worked with). I had been trying to denoise prior to upscale, but too much detail was getting killed for ESRGAN to interpret it as well.

It looks like the model you're using shifts the colors; I don't know if that was intentional. It's not a simple 601/709 shift . The original has a more green hue to the clips

Thankfully, this part was a happy accident. I had already been shifting the colors manually via a .lut because the promo posters/materials always were lacking the strange greenish hue present on the DVD. The movie version had a 2019 blu-ray release with proper coloration, so I generated the ESRGAN model based on the difference between the JP DVD and the US Blu-ray; no further need for color correction! I think this is also why the model performs dehaloing so well.

Thank you again for your help!

[poisondeathray]

I've been generally using D2Vsource within Staxrip 2.1.7.0 and its portable version of AVS+, using .d2v files from DGIndex, although I have also tried with MPEG2Source and (briefly) FFVideoSource. I've also tried running the code with MPEG2Source outside of the Staxrip/AVS+ environment, just using 32-bit Avisynth and x264, and the same result. I would like to think it's just a missing dependency, but... I have included the result below. An interlaced frame seems to jump in every so often, and the result is very repeatable.

I mention the poor IVTC job done on the original because often a single frame (especially after a cut) will be interlaced as well, when the rest of the footage is progressive. I can include a sample if it would help.

Truly, if there is a way I can avoid using QTGMC to get properly progressive frames, I would love to take it.

I see the problem in your sample, but I cannot reproduce it on the original. There is 1 combed frame at the beginning but you can use override.

To be clear - I'm referring specifically about options to process those tricky 12fps sections only, ie. in that example when the car is crossing the bridge. Yes it sucks to have to use overrides and VFR, but AnimeIVTC/QTGMC artifacts are far worse - watch the bridge warping in the 2nd section after ~10sec when using AnimeIVTC+QTGMC - it's unusable. Look on the bright side - it also means fewer frames to process and temporal filters are far more effective on unique frames.

You would use different processing on other sections, such as the 23.976p sections

Here's Noise+Banding after the upscale, no post-processing in AVS, just aspect ratio and fps correction from the MKV container.

Much better in terms of the aforementioned ghosting issues and line problems - that strongly suggests the problem was from your post processing filtering chain. I would do something different with those filters. If you want me to post something highlighting the problems I was referring to on the 1st sample, let me know

It looks like the model you're using shifts the colors; I don't know if that was intentional. It's not a simple 601/709 shift . The original has a more green hue to the clips

Thankfully, this part was a happy accident. I had already been shifting the colors manually via a .lut because the promo posters/materials always were lacking the strange greenish hue present on the DVD. The movie version had a 2019 blu-ray release with proper coloration, so I generated the ESRGAN model based on the difference between the JP DVD and the US Blu-ray; no further need for color correction! I think this is also why the model performs dehaloing so well.

On those few samples, it looks better to me subjectively in terms of colors. eg. Skin tones look better more red than green.

[laserbolt5]

Wow, your sample looks flawless!! I'm going to create a new Staxrip folder/reinstall Avisynth+ and try again, I'll bet something got jumbled up over multiple updates.

I would do something different with those filters. If you want me to post something highlighting the problems I was referring to on the 1st sample, let me know

That would be great! You have a keen eye I really appreciate it.

Now time for me to study how to use overrides in TFM...

[poisondeathray]

I would do something different with those filters. If you want me to post something highlighting the problems I was referring to on the 1st sample, let me know

That would be great! You have a keen eye I really appreciate it.

99.99% the likely culprit is temporaldegrain2 for the ghosting and line issues. It's a common problem when using temporal denoising , especially certain types of cartoons and anime. Darker frames are commonly affected more

I just compare it to the original (just IVTC). I downscale the upscaled sample to compare to original and look for issues. I typically use avspmod to see the problems - You can use tabs or interleave. I like tabs because they are aligned if you have the same frames. For example you have 1 extra frame in the mkv version, you use trim to make them even and aligned. You can swap tabs quickly with number keys. You can seek/navigate and all tabs are aligned on that same frame. This way it's easy to compare to original and vs. various filtered versions. If you're "remastering" something, you want to be sure you're not making it worse than the original...


(These are apng's, they should "animate" in most browsers like chrome, firefox. )

Line preservation is typically important for anime/cartoon fans. That would be unacceptable for most people




It's common for darker scenes to have detail loss when using temporal filtering, notice loss in cloud details. There are various prefiltering methods commonly used to tackle dark scenes for things like anime




There are other issues too - the embers/glowing thingys are ghosted away in the filtered version in some frames. If you want I can post those comparisons too

[poisondeathray]

Now time for me to study how to use overrides in TFM...

How many sections have that type of "bad interlace" ? By that mean that 12fps in 29.97 but with misaligned fields such as that car/bridge scene? One sample (Noise banding.m2v) was 23.976p content , but the "Characters Noise" is actually ~ 8fps content

For the bad interlace cadence, if there are just a few, it's much easier to use a few trim() or replaceframes() to process segments differently

For the Characters Noise, if that is representative of the majority of the content , I would still use VFR, because you're going to be processing ~2/3 fewer frames in ESRGAN. It's mostly triplicate frames (so 8fps), but few sections that are duplicates (12fps). It's waaay faster to process fewer frames in ESRGAN. Not only that, temporal filtering is more effective on unique frames. Duplicates makes everything less effective for temporal filtering