Issues with QTGMC producing visual artifacts

[Downgraded286]

So, my girlfriend's mom gave us a bunch of Spongebob DVDs, and I'm trying to rip them and put them on my Plex server... but, as always, issues arise. Mediainfo identifies the source as Progressive, and most of the time, it looks fine, but sometimes there are obvious combing artifacts left over, such as shown in the first attachment. QTGMC does clean up that scene nicely, but it is introducing other issues. The second attachment shows the untouched scene, which looks fine, and the third attachment shows some ghosting with the arms. Also, if you pay attention to the very top part of the video in the third attachment, you can see something going on there (shimmering/flickering) that isn't present in the original version (second attachment).

Just to be sure, I ran an encode without QTGMC, and the ghost arms nor that shimmering are present. The only other filter I'm using is spline36resize, and I've also verified that isn't affecting it either. It's QTGMC. I even went all the way down to the Placebo preset and it still happens. I've also tried qtgmc(preset="placebo", inputtype=1) and the shimmering/flickering artifact at the top of the video isn't present, but it doesn't touch the comby pineapple at all. inputtype 2 and 3 produce very jerky video.

[davexnet]

WIthout a sample of the original mpeg-2 (dvd) source and your complete script
your post has very little context

[Selur]

In general:
I think one should have used IVTC instead of deinterlacing.
I do not think QTGMC cleans this up nicely and is not the right tool for the job at all.
If you really think QTGMC should be used, use a faster preset, set inputtype=2 and source match at least to 1, or manually adjust all its parameters.

Taking the second video as source, I think the most suited tools are ai based ones.
If your setup support Vapoursynth and VSGAN, there are a bunch of Spongebob trained models over at https://upscale.wiki/wiki/Model_Database which should be able to clean this up.
A bunch of AI tools should be able to clean this up, here's what 'DPIR(clip=clip, strength=15.000, task="denoise", provider=1, device_id=0, dual=True)' does:

BasicVSR++:

(also see attached video also attached a version where I used 32 instead of 18 as quantizer to see whether I could keep the file size down)
Different anime supporting resizers could be used:
RealCugan:

RealESRGAN:

Waifu2x:

Anime4KCPP:

(Topaz AI should also be able to clean this up nicely.)

More traditional denoisers will have a hard time with this, since these artifacts are not really temporal.
So for normal denoisers you would have to increase their strength and probably mask them to not smooth everything away.

=> I agree that starting with the source content and doing a proper job in handling it would be the best approach, and then clean up is probably way easier.

Cu Selur

Ps.: I also attached a version where I treated pinapple as interlaced, applied QTGMC and BasicVSR++.

[Downgraded286]

WIthout a sample of the original mpeg-2 (dvd) source and your complete script
your post has very little context

The first two are original samples. They've just been cut and compressed with Virtualdub so the filesize isn't horrendous.

snip

Well, that's an undertaking... looks like VapourSynth would be a lot to learn. Guess I'll try messing around with QTGMC more. I don't think IVTC is the answer here, though... there's no clear pattern of progressive and interlaced frames, there's just random combing artifacts sometimes and when it happens, it's every frame.

[davexnet]

I thought the source was DVD? I was expecting DVD spec mpeg-2.
Hybrid, a well respected tool authored by Selur, can be downloaded from the software
section of this site
I believe his tool incorporates Vaporsynth and possibly some of the AI components such as VSGAN
(Perhaps he can confirm)

[Selur]

I believe his tool incorporates Vaporsynth and possibly some of the AI components such as VSGAN
(Perhaps he can confirm)

To use VSGAN (and some other ai tools), aside from an NVIDIA GPU, an extra addon is needed.

I thought the source was DVD? I was expecting DVD spec mpeg-2.

Same here, just thought I try to see what could be done with the samples and for cartoons ai based stuff really isn't bad. (you lose some details, but with some clever masking that effect can be reduced)

They've just been cut and compressed with Virtualdub so the filesize isn't horrendous.

Problem is, you used a format which doesn't support interlaced encoding, so you made things worse by encoding interlaced content as progressive. :/
(side note: First clip is 47 frames, pineapple clip is 108, even when using lossless compression the file size wouldn't be 'horrendous'. DGIndex could be used to cut a small portion from a vob/mpeg file. )

Cu Selur

[jagabo]

The pinapple shot is 30p captured out of phase. You can restore it to 30p with:

LWLibavVideoSource("pineapple.avi") 
AssumeTFF()
SeparateFields().Trim(1,0).Weave()

Some other 30p shots may not be out of phase so it's safer to use:

LWLibavVideoSource("pineapple.avi") 
AssumeTFF()
TFM()

The arms shot can also be TFM'd to 30p:

LWLibavVideoSource("arms.avi") 
AssumeTFF()
TFM()

That leaves some "random" duplicates but that may be intentional. Unless there are other shots that are problematic I'd just use TFM() and leave the video at 30p. Decimating to any other frame rate will leave the panning shots obviously jerky. And QTGMC is completely wrong for these clips.

v1 = LWLibavVideoSource("arms.avi", cache=false, prefer_hw=2) 
v2 = LWLibavVideoSource("pineapple.avi", cache=false, prefer_hw=2) 
v1 + v2
AssumeTFF()
TFM()

[davexnet]

Hi Selur re:Hybid/VSGAN requirements:
Thanks for clearing that up

[Downgraded286]

Jagabo, TFM seems to be the correct answer. It seems to remove all the combing artifacts, even the pineapple, without introducing any artifacts. How did you arrive at this answer? Any time I've dealt with telecined content in the past, there was that clear pattern with interlaced frames and progressive frames. The entire pineapple clip has combing artifacts. It seemed to me like it was originally interlaced content that was badly processed. The source files from the DVD show up as 23.976fps progressive, but given the combing artifacts in every frame (where present) I assumed it was badly deinterlaced video. Is this also the reason why QTGMC was causing garbage?