Solutions for a film with duplicate frames without predictable cycle?

[TB54]

Hello everyone,

I have the blu-ray of a silent film in 25fps, with about 3 duplicated frames per second – it seems, basically, that a 22fps master has been badly converted to 25fps.

These duplicate images are sometimes separated by 7 frames, sometimes by 8 frames. The worry is that this succession of gaps between two duplicate frames follows no logical sequence. It's random (7-8-8-7-7-8-7-8-7-8-8-7-...). So I'm reduced to deleting them manually in an editing software, searching for the duplicate frames every 7-8 frames, which will take me weeks.

Is there a scripted solution?
• I've tried ffmpeg with mpdecimate, but the result is too fluctuating (duplicate images retained, non-duplicate images deleted, scene with more deleted frames than others), and it's even worse with intertitles and fades to black.
• I can't use cyclic deletion, as the space between two duplicated images can be randomly 7 or 8 images long.
• I've tried a compromise, asking chatgpt for python scripts to delete 30 images every 10 seconds, choosing the 30 frames the most identical to their neighbors, on condition that at least 7 images separate them, but with no convincing result.

In short, apart from doing it by hand, I'm out of ideas...
But I'm a newbie in ffmpeg (and don't know coding at all), so I maybe miss something obvious. If someone has already encountered this problem (regular duplicate images but no strict cycle), and found a solution, I'd be very interested!


If you'd like to see what the source actually looks like, or test it, here's a swisstransfer link with two elements:
• An excerpt taken directly from the bluray. However, the bluray is interlaced (for some reason I don't understand, because when I check in virtualdub the two frames of each image always appear identical).
• The same extract de-interlaced with neddi2 (to facilitate image comparison operations)

[Sharc]

In Avisynth try

TDecimate(cycle=25,cycleR=3)

It returns 22.0 fps

[jagabo]

Yes, Cycle=25, CycleR=3 (remove 3 of every 25 frames) appears to remove all the duplicates and no originals from the shorter clip. If it you find the pattern is sometimes too irregular for that you can try Cycle=50, CycleR=6. Or even higher multiples.

[Sharc]

For smoother motion 22fps->50fps interpolated using RIFE

ffms2("Piz-Palu-excerpt-BDremux.mkv")
spline36resize(1280,720) #framesize reduced for this test only
Tdecimate(cycle=25,cycleR=3) #returns 22fps

# temporal interpolation
RGB=z_ConvertFormat(pixel_type="RGBPS", colorspace_op="709:709:709:l=>rgb:709:709:f") 
RIFE=RGB.RIFE(gpu_thread=1, model=16, fps_num=50000, fps_den=1000, sc=true, sc_threshold=0.3) 
YV12=RIFE.z_ConvertFormat(pixel_type="YUV420P8", colorspace_op="rgb:709:709:f=>709:709:709:l") 

return YV12

[Alwyn]

@Sharc, what's your file filter? I'm using

LWLibavVideoSource("D:\VH\TB54\Piz-Palu-excerpt-BDremux.mkv")

TDecimate(cycle=25,cycleR=3)

And it's giving me 44fps (from VDub "File Info"), and the render plays at double-speed. I see MediaInfo is reporting the MKV as 50fps.

[Sharc]

The source filter ffms2 returns the framreate 25fps.
LWLibavVideoSource returns 50fps (the field rate)
MediaInfo shows the 'Original Frame Rate' as 25fps and the 'Frame Rate' as 50fps which is the interlaced field rate.

[jagabo]

LWLibavVideoSource() gets the frame rate wrong (50 fps). You can force it to assume 25 fps with:

LWLibavVideoSource("Piz-Palu-excerpt-BDremux.mkv", fpsnum=25000, fpsden=1000)

Or you can AssumeFPS() after it:

LWLibavVideoSource("Piz-Palu-excerpt-BDremux.mkv") 
AssumeFPS(25)

Or use ffVideoSource():

ffVideoSource("Piz-Palu-excerpt-BDremux.mkv")

[Alwyn]

Thanks both.

I did try ffms2 and it created a hot mess (aka doesn't fix anything); AVISynth+ 64bit, ffms2.dll, 16074KB.

ffVideoSource is doing the same.

[jagabo]

I did try ffms2 and it created a hot mess (aka doesn't fix anything); AVISynth+ 64bit, ffms2.dll, 16074KB.

ffVideoSource is doing the same.

You're right. ffVideoSource gets the right frame rate but there are many duplicates (in addition to the actual duplicates in the underlying video), missing frames, and the timing is way off (especially on random seeks).

[TB54]

Thank you all!

TDecimate(cycle=25,cycleR=3)

... works perfect on several excerpts I've tried!
I will apply it on the entire movie now.

For the bluray source, I don't understand it myself... It's supposed to be a 25fps interlaced video, but it seems to be understood as a 50fps video by several softwares. That's why i used neddi2 to get rid of this first issue (even though it strangely redoubled the film as a result, I had to cut the result halfway, and it may not be the best deinterlacing solution if the two fields are similar, and can therefore be merged).

Thanks again, I'm stuck with this since days, I should have asked here first!

[Sharc]

I did try ffms2 and it created a hot mess (aka doesn't fix anything); AVISynth+ 64bit, ffms2.dll, 16074KB.

ffVideoSource is doing the same.

You're right. ffVideoSource gets the right frame rate but there are many duplicates (in addition to the actual duplicates in the underlying video), missing frames, and the timing is way off (especially on random seeks).

Strange. I suspect the problem is that there are different versions (forks?) of ffms2 floating around. Here I get exactly the same 25fps sequence as with the framerate corrected LWLibavVideoSource() for this source, even for random seeking with AvsPmod. Both creating an index file.
(My ffms2.dll (64) is 25'696'256 Bytes of 2024-05-28)

[Alwyn]

Updated ffms2 to @Sharc's version (latest published) and it now works a treat.