Best way to fix these interlacing artifacts from progressive source?

[sebastiaaan]

Hi there,

so I'm currently trying to restore some older tv shows from 2000ish. I have noticed that nearly all of them were deinterlaced very poorly and I really ask myself who was in charge of this, they just ruined precious content.
I'll provide a sample .mkv and an image for reference. So the whole video has weird lines when people are talking or moving, I'm 100% sure these come from bad deinterlacers used (what on earth did they use, even yadif does a better job??), but sometimes there are some weird artifacts like little dots (as shown in the screenshot, on the guy's face).
Does anyone know what those dots are? I'd say that this is a side effect of the horizontal combing leftovers and after the video got compressed it results in those dots the encoder probably didn't know how to handle properly.

I first ran QTGMC through it and it does a pretty good job eliminating the combing/horizontal lines. However sometimes you can still notice those dots as QTGMC seems to not eliminate them completely. I also noticed that QTGMC denoises pretty heavily, removing some of the digital noise (and I'm one of the very few who actually likes noise as long as it's not distracting and suits the content) which makes the output look way more compressed regardless of the CRF or tuning in x264 used.

I ran it a few times each time adjusting the settings a little bit.

QTGMC(preset="Placebo", InputType=2, sourceMatch=3, sharpness=0.3, tr2=2, ediThreads=8)
probably the best all around solution, combing is gone but occasionally those dots are still visible and digital noise is reduced (which is a con). tr2=1 and sharpness 0.4 helps a little bit retaining the noise but makes those dots more perceptible.

QTGMC(preset="Placebo", InputType=3, sourceMatch=3, sharpness=0.3, tr2=2, ediThreads=8)
Does a better job eliminating those dots but makes the overall image look way softer and degrades the image quality even more.


Is there anything else I could try? I mean I get it, I'm working with a lossy source and obviously I'm gonna lose some details. But I feel like there might be some "experts" in the forum who have more knowledge than I do and I'm looking forward to see what you guys think and how I should handle this source.

Thanks in advance.

[Sharc]

... I'm 100% sure these come from bad deinterlacers used (what on earth did they use, even yadif does a better job??),

The source was probably telecined film and should have been IVTC'ed (Inverse Telecined i.e. field matched and decimated) rather than deinterlaced.

Try:

tdecimate(cycle=5,cycleR=1) #removes the duplicates
qtgmc(inputtype=1) #cleaning up
vinverse() #removing residual combing artefacts

[sebastiaaan]

... I'm 100% sure these come from bad deinterlacers used (what on earth did they use, even yadif does a better job??),

The source was probably telecined film and should have been IVTC'ed (Inverse Telecined i.e. field matched and decimated) rather than deinterlaced.

Try:

tdecimate(cycle=5,cycleR=1) #removes the duplicates
qtgmc(inputtype=1) #cleaning up
vinverse() #removing residual combing artefacts

Thanks! I didn't notice the duplicate frames at first, but your script took care of them. I'm not quite sure how this show (Drake & Josh, TV show from 2004) was shot originally but definitely not on film. It's weird because the same episode as provided in the sample is also available as an encode coming from a DVD where the frame rate is also 29.970 but with no duplicate frames (although lower quality). I do not have the original DVD's so I'm not sure but doesn't that mean that most likely the content was 29.970i aka resulting in 59.94p when bob deinterlaced?
I don't know what the studio did when authoring the content because I'm certain the original framerate was not 23.976. Those same dot artifacts also show up on other shows from the 90's (same studio) which were evidently shot on tape, without the obvious combing but with still some leftovers.

Unfortunately those dots are still visible no matter what I do, at this point I think it's impossible to properly filter them out. Also vinverse doesn't seem to do anything, see the screenshot below (it's a different frame than the one above). Only at the bottom of the picture the very small white and blue part of objects are different, rest is the same. Also weirdly enough, just this minor difference is 24KB.


[Attachment 59753 - Click to enlarge]
vinverse on

[Attachment 59755 - Click to enlarge]
vinverse off

[Sharc]

I'm not quite sure how this show (Drake & Josh, TV show from 2004) was shot originally but definitely not on film. It's weird because the same episode as provided in the sample is also available as an encode coming from a DVD where the frame rate is also 29.970 but with no duplicate frames (although lower quality). I do not have the original DVD's so I'm not sure but doesn't that mean that most likely the content was 29.970i aka resulting in 59.94p when bob deinterlaced?
I don't know what the studio did when authoring the content because I'm certain the original framerate was not 23.976.

The duplicates indicate that the footage was telecined from 24p (or 23.976p) to 29.97i 720x480 at some stage for NTSC DVD compliance (which will be seen on a progressive monitor as a repetitive pattern of 3 clean pictures followed by 2 combed pictures). It was then probably deinterlaced instead of IVTCed, and resized to square pixels 720x540.
You can regenerate the Telecined DVD compliant 29.97i from your .mkv with the hard-telecining script:

TDecimate() # gives 23.976p
spline36resize(720,480) #NTSC resolution
separatefields()
SelectEvery (8, 0,1, 2,3,2, 5,4, 7,6,7)
Weave()

I have no idea what weird steps your footage went through though.

[pm-s]

what on earth did they use

It looks like it has been deinterlaced with one of those "smooth deinterlacers" years ago, then resized multiple times. Most of the artefacts can be filtered out with Sharc's script, but they will never be completely gone

[sebastiaaan]

I'm not quite sure how this show (Drake & Josh, TV show from 2004) was shot originally but definitely not on film. It's weird because the same episode as provided in the sample is also available as an encode coming from a DVD where the frame rate is also 29.970 but with no duplicate frames (although lower quality). I do not have the original DVD's so I'm not sure but doesn't that mean that most likely the content was 29.970i aka resulting in 59.94p when bob deinterlaced?
I don't know what the studio did when authoring the content because I'm certain the original framerate was not 23.976.

The duplicates indicate that the footage was telecined from 24p (or 23.976p) to 29.97i 720x480 at some stage for NTSC DVD compliance (which will be seen on a progressive monitor as a repetitive pattern of 3 clean pictures followed by 2 combed pictures). It was then probably deinterlaced instead of IVTCed, and resized to square pixels 720x540.
You can regenerate the Telecined DVD compliant 29.97i from your .mkv with the hard-telecining script:

TDecimate() # gives 23.976p
spline36resize(720,480) #NTSC resolution
separatefields()
SelectEvery (8, 0,1, 2,3,2, 5,4, 7,6,7)
Weave()

I have no idea what weird steps your footage went through though.

Yeah whoever was responsible for this mess shouldn't be in that position, just messing up old content. It's also weird that those artifacts differ from season to season.
Season 1 for example has the same artifacts but there are no duplicate frames, even at 29.970.
Season 3 on the other hand is 23.976p with no combing except every few seconds but no duplicates. I'll try and make a sample for this as I have no clue how I can tell QTGMC that it should only process those combed frames and leaving the rest untouched.

I'm just a bit confused about your second script you posted though. What would I gain by doing this? Your first script with TDecimate already took care of the dupes. So not sure why I should regenerate the interlaced version or am I missing something?

[Sharc]

I'm just a bit confused about your second script you posted though. What would I gain by doing this? Your first script with TDecimate already took care of the dupes. So not sure why I should regenerate the interlaced version or am I missing something?

It's just an example (or exercise) how the footage with the duplicated frames MAY have been put on the DVD, namely as (hard-)telecined 24fps. Or in case you would need a 29.97i version which causes less judder than the dumb frame duplication. No other purpose than for demo, no solution to the problem. But as you don't have the DVD you can't even compare. Sorry for the confusion.

[killerteengohan]

This will usually get rid of them, and it will be less destructive than vinverse when it comes to detail retention. It's one of the best filters I know of to clean those leftovers out.

santiag(strv=0, nns=4, nsize=5)

If detail retention is important to you, this will retain detail a bit better, but it will be slightly less effective at removal of the artifacts. It's still pretty good at removing those artifacts though.

source = last
santiag(strv=0, nns=4, nsize=5)
MergeLuma(last, source, weight=0.3)
MergeChroma(last, source)

If you are going to use QTGMC, this will ensure great quality from it. This is MUCH better than the default settings and a lot less destructive to lines and details. Do not bother with Placebo preset, it's a HUGE waste of time.

QTGMC(preset="slow", matchpreset="slow", matchpreset2="slow", sourcematch=3, InputType=1, tr1=2, tr2=1, NoiseTR=2, sharpness=0.1)

# Raise sharpness to 0.2 if you want a sharper outcome. I wouldn't go higher unless you want halo's in your video which is a negative thing.
# If you somehow find this a bit more destructive than you would like, take out the sourcematch=3 and see if you prefer that.

You most likely wont need QTGMC if you use Santiag. Santiag is very good at cleaning those combing artifacts. If you can get away with using just Santiag, you will save your video some destruction. If you decide you want to use both, make sure to use Santiag after QTGMC in the script, and not before it.

[manono]

...and I really ask myself who was in charge of this, they just ruined precious content.

In charge? You're implying this was some sort of a studio effort, which it isn't. Just some incompetent fool at a computer.

I'm not quite sure how this show (Drake & Josh, TV show from 2004) was shot originally but definitely not on film.

I wouldn't be so sure, if I were you. 23.976fps means film.

I do not have the original DVD's so I'm not sure but doesn't that mean that most likely the content was 29.970i...

No, not necessarily. As already mentioned. 29.97i is required for NTSC DVD. The source for that could easily have been progressive 23.976fps, hard or soft telecined to achieve 29.97i output.

I don't know what the studio did when authoring the content...

A studio didn't create what you have.

I'm certain the original framerate was not 23.976.

Once again, you're wrong. The source framerate of your sample was 23.976fps.

[pm-s]

I'm not quite sure how this show (Drake & Josh, TV show from 2004) was shot originally but definitely not on film.

You know 24p video-cameras existed back then?

[jagabo]

IMDB indicates 1080i for seasons 2 to 4. I suspect it was shot on film in season 1 and broadcast as SD. Then they switched to HD 1080i video for seasons 2 through 4.

https://www.imdb.com/title/tt0363328/technical?ref_=tt_spec_sm

Most TV shows shows were shot on film in those days. The main exceptions were news, soaps, talk shows, and sports.

[pm-s]

Most TV shows shows were shot on film in those days. The main exceptions were news, soaps, talk shows, and sports.

Again, it wasn't shot on film. It was shot on 24p digital camera. It was recorded in 2004 not 1994

[manono]

Again, it wasn't shot on film. It was shot on 24p digital camera. It was recorded in 2004 not 1994

Now you're nitpicking. Your original contention was that it was interlaced 29.97fps, meaning shot using video cameras.

[Skiller]

Besides, it could have very well been shot on film in 2004.
I could name a couple of more examples, but take "Scrubs" for example. It was shot on 16mm film all the way from 2001 to the end of the final season in 2010!