Detect duplicate frames and write to text file

[nesburf]

I have a bunch of 60i video that needs to be IVTC'd to recover the progressive frames.

My problem is, depending on what IVTC method I use, I occasionally get duplicated frames for some reason. I can use one method on one clip and it's completely fine. And using that on another will cause duplicated frames.

I've been using

uncomb()
decimate(5)

telecide(1)
decimate(5)

tfm()
tdecimate()

I know they were all originally shot at 23.976p, but I'm guessing the way it was broadcast is throwing that IVTC filters for a loop.

What I'm looking for is a script that I can use to find out which works best with each video. Ideally it would output a text file and show me, if there are any duplicated frames. Then I can decide based on that which IVTC method will work best for that specific video.

Is there an avisynth script that will do just that?

Thanks

[poisondeathray]

The potential issue is "duplicates" are detected based on a threshold value. With broadcast noise and temporal compression a true duplicate might "look" different due to compression differences. If you set the wrong value you will either miss some or detect false positives. You can use WriteFileIF() with YDifferenceFromPrevious() . Use show=true to tweak the threshold value that is appropriate for your source

Or another approach would be to look at the actual differences detected between frames e.g. look at the dedup log file. You can examine it in spreadsheet program - that way you don't miss any

[jagabo]

Sometimes unusual pulldown patterns are used so the usual "remove 1 of every 5 frames" decimation doesn't work. You can often get around this with TDecimate(Cycle=10, CycleR=2) rather than the default TDecimate(Cycle=5, CycleR=1). Progressive PAL is sometimes converted to NTSC with with a 3:2:3:2:2 pulldown and can be converted to 24.975 fps with TDecimate(Cycle=12, CycleR=2), or 25 fps with TDecimate(mode=2, rate=25). Be aware that pulldown can be used to convert any frame rate from 19.98 fps to 29.97 fps progressive to 29.97i. So you need to decimate to the right frame rate to get no duplicates and no drops.

[nesburf]

Thanks guys

I will give the WriteFileIF() and YDifferenceFromPrevious() a try. And also changing the TDecimate parameters.

jagabo - I know the source is definitely a 23.976 source...as it was a North American production. I'm thinking it may be an issue with the broadcast noise and compression that is causing the IVTC to fail in certain spots. The show was originally shot on 16mm film so there is definitely some grain in the picture. That combined with mpeg2 compression is likely causing my issue.

[jagabo]

Can you post a short sample? Especially a clear medium speed panning shot.

[nesburf]

Sure, at work now, when I get home I'll upload a 10 second sample.

[nesburf]

So here is an example of a shot where telecide().decimate(5) causes jerkiness.

However, uncomb().decimate(5) properly matches fields and removes the duplicate frames.

I have others that are the exact opposite. Problem is, I don't know if its the same throughout...because it's only at certain spots

[LMotlow]

I don't seem to have a problem with that sample.

AssumeTFF().TFM().TDecimate()

Attached is mpeg2 23.97fps stereo SD 16:9.

[poisondeathray]

nesburf - how are you loading the videos? What is the source filter used ?

There doesn't seem to be any problems with any method

EDIT: sorry, telecide().decimate() does have a duplicate , at least at default settings. Probably the noise pattern is giving it problems. A "duplicate" can be confused as a non duplicate because of different noise patterns. In this specific case, it's a field matching issue with Telecide

Did you say TFM.TDecimate fails on other sections at default settings?

[nesburf]

here is my script

SetMemoryMax(512)
LoadPlugin("C:\Program Files (x86)\AviSynth 2.5\plugins\DGDecodeNV.dll")
SetMTMode(6,8)
dgsource("test1.dgi")
uncomb() or telecide(1)
decimate(5)
SetMTMode(2,8)
crop(4,6,-4,-4)
LanczosResize(640,360)

LMotlow - That works for me as well, however I have other episodes where that will throw back duplicates and another field matcher/decimate filter will work,

That's why I was looking for a way to detect if there was any duplicate frames and output it to a text file. I could try different IVTC methods, and then pick which one would work best for each video. This way I wouldn't have to watch each episode to find out if there are duplicates.

[jagabo]

I don't know if you noticed this but your source has a field location problem -- the top field is below the bottom field rather than above it. You need to SwapFields() before ivtc.

The 14+ logo doesn't have this problem so it looks like it was broadcast this way!

[LMotlow]

Like poisondeathray and jagabo said, you run into all kinds of things with frame rate work. With an HD PVR I recorded a whole season 1 of an Australian import broadcast as NTSC on PBS. For every season 1 episode I noted what looked like 3 "interlaced" frames in a row, very consistent for the whole 50 minutes. None of the IVTC routines worked. I got 23.976 fps progressive and no glitches with this code:

QTGMC(whatever preset)
sRestore(frate=23.976)

Along comes Season 2. Nothing works anymore. I load an episode into VirtuaDub, plug in a bob filter, and I see interlaced, telecined, and duplicate frames together. Started a short section with TFM().TDecimate() and end up with 23.976 but 11 dropped frames out of 286. So looks like the crew changed the way that series is shot. For one thing, it don't look like film any more, it looks digital. The only code that gave me 23.976 with no problems (but PAL film 24fps had glitches) :

QTGMC(whatever preset)
sRestore(frate=25)
AssumeFPS("ntsc_film",sync_audio=true)

Looks OK to me for the whole 50 minutes. Also downsized that for DVD backup and encoded at 23.976 NTSC with 3:2 pulldown, and they play like a charm. Go figure. That's a lot of trouble for a backup, though, so I ordered the NTSC BluRay set.

Then I have another BBC import series from the UK recorded off PBS with the HD PVR, and plain old TFM().TDecimate() works every time.


There are umpteen threads here and at doom9 about routines to detect one kind of frame problem or another. None are perfect, many gives unpredictable results. And sometimes you run into nightmares like blended frames/telecine and other whacky stuff that can't be totally fixed.

[nesburf]

the cap is actually an HDMI capture from a cable box. So I'm wondering if the encoder is actually screwing up the field order.

[jagabo]

One of the reasons for the differences between telecide() and uncomb() is an orphaned field at the first field of the fade in. Uncomb() passes that through without any post processing. telecide() matches the next pair of fields leaving no combing. Both also fail to match one frame later in the sequence, leaving comb artifacts. Lowering vthresh to 40 in telecide() takes care of that.

[jagabo]

the cap is actually an HDMI capture from a cable box. So I'm wondering if the encoder is actually screwing up the field order.

But the 14+ logo doesn't have the problem. So if the capture device swapped the fields, the logo was overlaid with the fields swapped and the capture device fixed it. You should check other caps from the same device.

I'd be curious to see if swapping the fields takes care of your problem with telecide().decimate(5) on the other clips.

[nesburf]

I will give that a shot and see.

Also going to try recording some other channels and see if they exhibit the same issue