Removing Slight Stuttering Effect in Videos

[Spiralagnus]

I have a batch of videos that all appear to have a slight stutter to them, almost as if there was a slight discrepancy between the frame rate of the video and the frame rate at which the video was playing. I tried a few different tricks with ffmpeg to fix the problem, including re-encoding the video, changing the frame rate, and playing around with minterpolate, but to no avail.

Finally, I stepped through a piece of one video one frame at a time, and what I discovered is that after every 2 or 3 frames, the next frame in the sequence is a duplicate of the previous one. A quick bit of counting revealed that in one second of a (constant rate) 24 fps video, I have 17 frames of "change" inter-weaved with 7 frames of "still". ex. ABBCDDEFGGHIIJKLL, etc. (I also have another set of stuttering videos with 16 frames of "change" for every 8 "still".)

This explains the stutter, but I'm still not sure how to fix it (if it even can be fixed). These are videos pulled from Blu-Ray discs, so I'm not working with old content. I'd welcome any suggestions the community may have. It would be great if there was a way to remove the "still" frames and have a program like ffmpeg interpolate in new ones.

Also, as a point of curiosity, how does a video end up in this sort of state in the first place? This is the first time I've ever run across it.

Here's the information for one of the videos, if this helps:

Format                                   : Matroska
Format version                           : Version 4

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : Main@L4.1
Format settings                          : CABAC / 4 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 4 frames
Muxing mode                              : Container profile=Baseline@L3
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 1 min 0 s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Default                                  : Yes
Forced                                   : No

And here's a sample from one of the 16/8 batch.

[poisondeathray]

You can use filldrops or (filldrops rife variations for higher quality) in avisynth or vapoursynth if you want duplicates replaced with interpolated "inbetween" frames

But you have to consider if that is the correct thing to do. Examine if the duplicates were indeed missing frames that were dropped, and if (a) that is the correct "location in time" for them , or (b) are they displaced

how does a video end up in this sort of state in the first place?

What is the background information ? What is the source from, what genre is it, what pathway did it take to get to where you have it ? Is there any audio ?

There are other content framerates than 23.976 - e.g. silent film, 8mm, super8 . ie. the 17 or 16 fps could be the correct framerate that, and the correct procedure would be to remove the duplicates, not to interpolate with motion interpolated frames. You could do that with TDecimate in avisynth or vapoursynth . The ffmpeg decimate is limited to 1 in X decimation, and not good enough

If you don' t know, then post a sample. A good video sample is worth 10,000 words

[poisondeathray]

These are videos pulled from Blu-Ray discs

It could also be meant to be that way. BD does not support anything lower than 23.976. So if the content frame rate was lower that 23.976, the studio would have to duplicate frames, hopefully in a distributed pattern to minimize the judder to put it on BD

eg. If "ABBCDDEFGGHIIJKLL" was an accurate description for a repeating cycle, that would suggest the real content frame rate was lower than 23.976, and you should just decimate the duplicates.

You would use TDecimate(cycleR=5, cycle=17) because this is "M in N decimation", or 5 duplicates are discarded in each group 17 frames. That should give you "ABCDEFGHIJKL" and about 16.92 unique frames /s .

[Spiralagnus]

If you don' t know, then post a sample. A good video sample is worth 10,000 words

Good idea. I've edited the OP with a link to a sample at the bottom.

[poisondeathray]

"sample.mp4" looks like jumps in motion (dropped frames), and duplicates.

The notation would be

AABDDEGGHJJ..

So you're missing C, F, I...etc.. It looks like it should have been 24/1 fps , but likely an error somewhere in making the BD


This next issue is unlikely to drastically change the findings, but was the sample made ? It's variable framerate, and the minimum framerate suggests there is a drop in making the sample

Frame rate mode                          : Variable
Frame rate                               : 24.000 FPS
Minimum frame rate                       : 23.994 FPS
Maximum frame rate                       : 24.000 FPS

There is more than one way to do this; if the cadence is the same, you can decimate and interpolate the missing frame in a pattern. If not, there are adaptive scripts that try to detect duplicates and fill in drops in the spots with the largest difference. If the sample is representative, the way I would do it is the former

This is an avisynth script version

LWLibavVideoSource("sample.mp4")
TDecimate(cycler=1, cycle=3)
z_convertformat(pixel_type="RGBPS")
RIFE(model=44, sc=true)
z_convertformat(pixel_type="YV12")
selectevery(4, 0,2,3)

One potential issue is when you get the dropped frame at the end of a scene. This means the interpolation algorithm will blend the 2nd last frame of the current scene , and the 1st frame of the next scene to make the "inbetween" frame (synthesize the last frame of the scene) and it will look ugly. I would disable scene change interpolation for that reason (that's what the sc=true means in the RIFE call), and it will place a duplicate, it's probably good enough

There are algorithms that can predict from 1 direction - ie. current frame from 2 previous frames (or reversed - current frame, from 2 next frames) , instead of "inbetween" two known "goalposts" or reference points. So that would be ideal for a bad scenechange where you have a botched 1st or last frame in a scene. An example is DMVFN, it's machine learning from some of the same authors as RIFE. I posted some examples at Doom9 forum. It has not been ported to vapoursynth or avisynth, but you can run python scripts to generate a frame instead of the duplicate at the end of a scene change if there are any if it bothers you

The output of the script is attached