Fix duplicate frames on old digicam video

[Homestar]

I have an old home video clip in wmv format from about 2007.
It was recorded on a random old point-n-shoot camera, and according to the metadata was converted to wmv using an ancient copy of Windows MovieMaker 2.1

The file is 320x240, supposedly 15fps.
The problem is that it plays back very jerky, even for 15fps.

If I step through the video frame by frame, I can see that there are a lot of duplicated frames, not in any clear pattern.

For example, the frame sequence could be like this:
1-2-3-4-4-6-7-8-9-10-11-11-11-14-15-16-17-18-19-19-21... etc

What I'd like to do is replace the duplicate frames with new motion-interpolated frames, while keeping the original frame rate intact, and not mangle the other "good" frames.

The mpdecimate filter on ffmpeg does a great job of detecting the duplicated frames.
As a test only, I used the filter "-vf mpdecimate,setpts=N/FRAME_RATE/TB" to output a file with the duplicates dropped completely.
This works but changes the run time, as it just skips over the duplicate frames.

Is there a way to use the minterpolate filter to only interpolate specific frames that mpdecimate identifies?

I'm familiar with ffmpeg and Adobe Premiere, but am open to try other tools to accomplish this.

First time poster here, and any help or advice is appreciated!

[johnmeyer]

True duplicates can be easily replaced with motion-estimated frames. I've posted my "FillDrops()" and "FillDropsI()" (for interlaced video) over at Doom9.org many times. It works really well.

However, most video that contains duplicate frames -- especially a video that was created in the way you have described your video -- usually includes those duplicates because a frame was dropped a few frames earlier in the video. If nothing is done, this will screw up the sound sync. So, to keep the sound in sync, the capture card adds a duplicate frame. Thus, you may have a more difficult problem than the one you describe, and you'll therefore need to do the much, much tougher task of figuring out where there is a dropped frame. Once you detect that point, you insert a motion estimated frame at that point, rather than at the place where the duplicate happens. Thus, what you probably want to do is:

1. Detect the duplicate and delete it (easy).
2. Detect the dropped frame (difficult, because the software has to look for a larger visual jump than what happened on adjacent frames).
3. Insert a motion estimated frame at the location where you think you've detected a dropped frame (easy).
4. Make sure that you only add frames if there is a duplicate frame near a dropped frame (hard).

The last one is needed so you don't change the length of the video.

I tried to create a script that does this, and posted the result in this thread I started in doom9.org seven years ago:

Automatically fix dups followed (eventually) by drops

This also gives you links to the filldrops() function.

[manono]

AviSynth's Filldrops does what you want.

I linked to that thread because johnmeyer's version might be slightly better than the original. The original thread is here.

Oh, too late.

[johnmeyer]

Yes, my filldrops script is based on the really old MugFunky script. He used the original MVTools, and I updated that to MVTools2; adjusted several settings; and adapted it to also work with interlaced video.

[Homestar]

I'm new to AVISynth and scripts to apologies if I've messed this up.

I installed AviSynth 2.6.0. 32-Bit.
I have latest ffmpeg 32-Bit. (Had 64 originally, but trying to 32-bit to match "bitness" of avisynth.
I installed the mvtools2.dll plugin.

I saved the script as magic.avs, updating it for my source wmv and correct path to the mvtools2.dll plugin.

Running "ffmpeg -i magic.avs" results in this error:

[avisynth @ 074f46c0] Script error: there is no function named "mt_merge"
(magic.avs, line 47)
magic.avs: Unknown error occurred

I tried googling this but didn't find a simple answer and was a bit overwhelmed.

Any ideas?

[manono]

Please give the complete AviSynth script and any error message you get when opening the script in Virtual Dub. Which might be the same as the ffmpeg error message.

[jagabo]

[avisynth @ 074f46c0] Script error: there is no function named "mt_merge"

You need masktools2. http://avisynth.nl/index.php/MaskTools2

But, filldrops/filldrops2 don't work right for multiple duplicates in a row. You'll need to call them twice for frames repeated 3 times, or three times for frames repeated 4 times, etc. Even then, the motion won't be quite right.

[johnmeyer]

If you have LOTS of multiple duplicates in a row, the video probably cannot be rescued.

[manono]

But, filldrops/filldrops2 don't work right for multiple duplicates in a row.

I didn't even notice that - that there might be multiple dupes in a row. But there was only one place in his example where there were 2 dupes in a row, so it still might work reasonably well.

What's the answer then? Pull out all the duplicate frames with multidecimate or dedup followed by interpolating to whatever framerate he wants? Which probably won't work very well either.

But then, a 320x240 15fps WMV source with a bunch of duplicate frames must look like crap to begin with. Especially if, as johnmeyer supposes, each dupe frame means a missing unique frame elsewhere.

[johnmeyer]

But then, a 320x240 15fps WMV source with a bunch of duplicate frames must look like crap to begin with. Especially if, as johnmeyer supposes, each dupe frame means a missing unique frame elsewhere.

Most of the videos I've seen that have this problem are from streaming sources, or from an older capture card. Both of these often have problems keeping up with the incoming video and, when they can't keep up, they drop a frame. This creates a temporal gap, and with something like a smooth camera pan, these gaps are painfully obvious. Then, to keep the audio from going out of sync, the streaming software or capture app will duplicate a frame. The dupe is created at a later moment in time, but the number of frames between the drop and the duplicate are seldom the same. Thus, unfortunately, you cannot simply look for duplicate frames, something that is relatively easy using one of the YDifference functions, count forward a set number of frames, and interpolate a new frame there.

The script I came up with, that I linked to above, used some clever (well, they seemed that way to me) techniques invented by a guy named Didée over at doom9.org. I adapted his ideas and came up with a function that does what I describe. However, because the drops and dups are often so irregular, my approach required looking at a whole bunch of frames at once. I think it was something like 20-30 frames in a cycle. As a result, the script is pretty slow.

The good thing about the script is that when it makes a mistake and mis-identifies a dropped frame, the motion estimation is good enough that you don't usually notice the spurious addition.

[poisondeathray]

For example, the frame sequence could be like this:
1-2-3-4-4-6-7-8-9-10-11-11-11-14-15-16-17-18-19-19-21... etc

The frame numbering he gave would suggest the duplicates in his case are "place holders" but in the correct sequence; ie. not temporally displaced. I think this type of pattern is far more common for point and shoot cameras which have been processed, and streaming web videos

Another clue is WMM . Those frames are probably duplicates added by WMM from a lower base framerate to make up the new 15fps framerate . In avisynth-ese it would be analgous to changefps(15) from say a 12fps video or some lower fps video

The distinction is important, because drops from a true 15 fps sequence would be in the correct time exactly. If it's like the way he indicated, if you were to interpolate those (ie. the missing 5,12,13,20) , it would be smooth. But a changefps-like manipulation in WMM or video editor would be a lower fps to higher fps conversion by duplicating frames in approx the correct location. Frames are not "evenly spaced" in time. If you were to interpolate those, it would still be choppy . In the 2nd case it would be better to delete duplicates using dedup/multidecimate or some other duplicate removal then reinterpolate / retime completely using something like mvtools2/smoothfps2/interframe/framerateconverter... etc... but with likely ugly artifacting results, but at least the timing would be "evenly spaced"

[jagabo]

If you want to go through the video frame by frame and locate the longer strings of duplicate and missing frames you can use ReplaceFramesMC() to replace multiple missing frames with motion interpolated frames (overwriting duplicates that were inserted in their place), and InsertFramesMC() to insert one or more motion interpolated frames.

https://forum.videohelp.com/threads/352741-Frame-interpolation/page2#post2226119

And if you're really ambitious, after generating a clean(ish) 15 fps video, you can double the frame rate to 30 fps with motion interpolation with Interframe2, DoubleFPS2, or some other such filters. Or even 60 fps.

[tonik2000]

Hi.

I have some jerky video vs lot of dup frame and some drop one: sorce (3,5mb)
I tried to fix it whis johnmeyer "magic" script but got much more mess: result (3,5mb)

Here is script. Where I was mistaken?

LoadPlugin("C:\soft\[url=https://www.videohelp.com/software/MeGUI]MeGui[/url]\tools\avisynth_plugin\mvtools2.dll")
#Threshold for detecting jumps. Increase to catch more jumps. Should always be less than 1.0
JumpThresh = 0.8
showdot = false # true for troubleshooting; otherwise, false

#SetMTMode(5,4)
LoadPlugin("C:\soft\MeGui\tools\avisynth_plugin\masktools2.dll")
LoadPlugin("C:\soft\MeGui\tools\avisynth_plugin\TIVTC.dll")
LoadPlugin("C:\soft\MeGui\tools\lsmash\LSMASHSource.dll")

global source=LWLibavVideoSource("E:\2004.04.14 (1).mkv").ConvertToYV12

#SetMTMode(2)

global BlackFrame = BlankClip( source, Color=$000000 )
global WhiteFrame = BlankClip( source, Color=$FFFFFF )

super = showdot ? source.subtitle("***").MSuper(pel=2) : source.MSuper(pel=2) 
bvec  = MAnalyse(super, overlap=4, isb = true, search=4, dct=5) 
fvec  = MAnalyse(super, overlap=4, isb = false, search=4, dct=5) 
double = source.MFlowFps(super, bvec, fvec, num=60, den=1, blend=false)

#Remove comment from ShowMetrics, and change "return final" to "return test" to look at metrics in order to determine proper JumpThresh
#test=ShowMetrics(source)

#Generate a white or black frame, depending on frame difference
BWMask=GenerateMask(source)

#Generate the 2x framerate mask needed to choose the motion-estimated frames
themask = interleave(BlackFrame,trim(BWMask,1,0))

#Merge double framerate from original with motion-esimated frames, but only where there are jumps
#(i.e., original frames are used except at jump points)

interleave(source,source).mt_merge(double,themask,luma=true,U=3,V=3)

#Decimate
RequestLinear
final=tdecimate(display=false,mode=1,cycleR=10,cycle=20)  # Decimate half of all frames (set to twice the length of "normal" dup/drop cycle)

#---------------------
#Alternate two-pass approach to decimation 

#Pass 1
#RequestLinear(debug=false)
#final=tdecimate(display=false,mode=4,output="e:\metrics.txt")

#Pass 2 (remember to un-comment "requestlinear")
#RequestLinear
#final=tdecimate(display=false,mode=2,rate=30,input="e:\metrics.txt",maxndl=20)
#---------------------

return final
#return stackvertical(source,final)

#----------------
#This function displays the YDiff value that will be used for detecting big jumps
#Each YDiff must eliminate Ydiff=0 (duplicate) from moving average

function ShowMetrics (clip c) 
{
  fixed=source.ScriptClip("Subtitle(String(
    \ (( (YDifferenceFromPrevious(selectevery(source, 1, 2)) < 0.2 ? 
    \       YDifferenceFromPrevious(selectevery(source, 1, 3))  :
    \       YDifferenceFromPrevious(selectevery(source, 1, 2)) )
    \  +
    \    (YDifferenceFromPrevious(selectevery(source, 1, 1)) < 0.2 ? 
    \     YDifferenceFromPrevious(selectevery(source, 1, 2))  :
    \     YDifferenceFromPrevious(selectevery(source, 1, 1))  )
    \  +
    \    (YDifferenceFromPrevious(selectevery(source, 1, -1)) < 0.2 ? 
    \     YDifferenceFromPrevious(selectevery(source, 1, -2))  :
    \     YDifferenceFromPrevious(selectevery(source, 1, -1))  )
    \  +
    \    (YDifferenceFromPrevious(selectevery(source, 1, -2)) < 0.2 ? 
    \     YDifferenceFromPrevious(selectevery(source, 1, -3))  :
    \     YDifferenceFromPrevious(selectevery(source, 1, -2))  )
    \     )/4) / 
    \    (YDifferenceFromPrevious(source) + 0.01)
    \ ))")
  return fixed
}



#----------------
#This function returns a white clip whenever a big jump is detected; otherwise a black clip is returned
#Each YDiff must eliminate Ydiff=0 (duplicate) from moving average

function GenerateMask (clip c)
{
  MyMask=c.ScriptClip("""
    \ (( (YDifferenceFromPrevious(selectevery(source, 1, 2)) < 0.2 ? 
    \       YDifferenceFromPrevious(selectevery(source, 1, 3))  :
    \       YDifferenceFromPrevious(selectevery(source, 1, 2)) )
    \  +
    \    (YDifferenceFromPrevious(selectevery(source, 1, 1)) < 0.2 ? 
    \     YDifferenceFromPrevious(selectevery(source, 1, 2))  :
    \     YDifferenceFromPrevious(selectevery(source, 1, 1))  )
    \  +
    \    (YDifferenceFromPrevious(selectevery(source, 1, -1)) < 0.2 ? 
    \     YDifferenceFromPrevious(selectevery(source, 1, -2))  :
    \     YDifferenceFromPrevious(selectevery(source, 1, -1))  )
    \  +
    \    (YDifferenceFromPrevious(selectevery(source, 1, -2)) < 0.2 ? 
    \     YDifferenceFromPrevious(selectevery(source, 1, -3))  :
    \     YDifferenceFromPrevious(selectevery(source, 1, -2))  )
    \     )/4) / 
    \    (YDifferenceFromPrevious(source) + 0.01) <= JumpThresh 
    \ ? WhiteFrame : BlackFrame """)
  return MyMask
} 

[tonik2000]

Del