Interpolate occasional dropped frame in VFR MKV.

[ajingo]

I have a video that is 23.976fps but MKV with occasional (~1 every few seconds) frame drops.
I believe these are encoded in the MKV as 2 frame gap but have no way of verifying, only that the avg framerate drops to ~22.9.

Ideally I want to interpolate just the missing frames (the video is full of many panning shots), which should be easily detectable from the mkv timing.
Most instructions I found online seem to be quite drastic measures for more broken footage - Interpolating the whole lot and dropeven, or detecting the duplicates and then replacing with interpolated verison (While I could go this route, I would be replacing a known dropped frame with a duplicate that then requires heuristics to detect.

I tried ffmpeg with -vf "minterpolate=fps=24/1001" ~but it was running at 0.2x speed, so it was likely doing more than interpolating a frame every few seconds.~ This just interpolated the first frame ad infinitum.

I tried Hybrid, with Frame Interpolation set to SVP @ 23.976 but this interpolated everything leacing artifacts on every frame.

I tend to use flowframes for interpolation, but trying to adapt that to this circimstance is probably more effort than it is worth, and with only the occasional frame missing lower quality interpolation is fine.

Any suggestions on a way forward would be great.
Thanks in advance.

The frametimes from MPC--BE

[Attachment 58970 - Click to enlarge]

MediaInfo

General
Format                                   : Matroska
Format version                           : Version 4
File size                                : 2.39 GiB
Duration                                 : 1 h 2 min
Overall bit rate                         : 5 487 kb/s
Encoded date                             : UTC 2020-09-03 03:59:30
Writing application                      : mkvmerge v47.0.0 ('Black Flag') 32-bit
Writing library                          : libebml v1.3.10 + libmatroska v1.5.2

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4
Format settings                          : CABAC / 5 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 5 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 1 h 2 min
Bit rate                                 : 5 231 kb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Variable
Frame rate                               : 23.822 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.106
Stream size                              : 2.27 GiB (95%)
Writing library                          : x264 core 160 r10 22fcbe1
Encoding settings                        : cabac=1 / ref=5 / deblock=1:0:0 / analyse=0x3:0x113 / me=hex / subme=8 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=34 / lookahead_threads=5 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / stitchable=1 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=infinite / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=50 / rc=crf / mbtree=1 / crf=20.0 / qcomp=0.60 / qpmin=5 / qpmax=69 / qpstep=4 / vbv_maxrate=5500 / vbv_bufsize=15000 / crf_max=0.0 / nal_hrd=none / filler=0 / ip_ratio=1.40 / aq=1:1.00
Default                                  : Yes
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : A_AAC-2
Duration                                 : 1 h 2 min
Bit rate                                 : 253 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Stream size                              : 113 MiB (5%)
Default                                  : Yes
Forced                                   : No

[jagabo]

detecting the duplicates and then replacing with interpolated verison (While I could go this route, I would be replacing a known dropped frame with a duplicate that then requires heuristics to detect.

filldrops() in AviSynth does this and works very well -- but it only works for a single drop/dup. With VFR sources you have to use a constant frame rate source filter that inserts duplicates when a single frame has a duration of more than one frame.

LWlibavVideoSource("filename.ext", fpsnum=24000, fpsden=1001)
filldrops()

Search these forums for filldrops.

-vf "minterpolate=fps=24/1001"... just interpolated the first frame ad infinitum

I'm not familiar with that filter, and don't know if it does what you want, but shouldn't it be -vf "minterpolate=fps=24000.0/1001.0" ? I think it will use integers if you don't use floats -- and 24/1001 is zero as an integer -- exactly the behavior you saw.


Also, you can use ffprobe to see timestamps for each frame.

"g:\program files\ffmpeg\bin\ffprobe.exe" -v 32 -stats -y -hide_banner -i "input.mp4" -select_streams v -print_format csv -of csv -show_entries "frame=media_type,coded_picture_number,pkt_pts_time,pict_type" >> "output.csv"

That will give you a CSV that you can open with Notepad or other simple text editor. Or import into a spreadsheet.

[ajingo]

Thanks for the pointers

filldrops() in AviSynth does this and works very well -- but it only works for a single drop/dup. With VFR sources you have to use a constant frame rate source filter that inserts duplicates when a single frame has a duration of more than one frame.

I just got to the avisynth page on VFR and got it working with frame duplication. I was however trying with Duplicity which is way more complicated than filldrops(), and I still haven't gotten it running

Unfortunately filldrops() has way too many false positives, it filled in the drop, along with a dozen other frames.
This was my main reasoning to try without heuristics, but I'll investigate other similar threads. Wrong filldrops, see edit

There was one about replacing missing frames with black placeholders and then using that to replace them. However, I don't know a good way to get LWlibavVideoSource to put black frames in instead of duplicates.

I'm not familiar with that filter, and don't know if it does what you want, but shouldn't it be -vf "minterpolate=fps=24000.0/1001.0" ? I think it will use integers if you don't use floats -- and 24/1001 is zero as an integer -- exactly the behavior you saw.

That was my mistake there, it does run now, I don't know how the quality is, because it runs at 1 fps (Ryzen 3700x), so it is untenable whatever.

Annoyingly

ffmpeg -i '.\original.mkv' -vsync 1 -r 24000/1001 -t 2:00 -acodec copy test1.mkv
frame= 1270 fps=116 q=-1.0 Lsize=   17115kB time=00:00:53.63 bitrate=2614.2kbits/s dup=8 drop=0

Duplicates frames in the right place, so if only I could replace the duplication there with a interpolated frame.

If there is there some way of logging these duplicated frame numbers, maybe I could then feed that into a avsynth script to replace the specific frames.
I'll have to use a lossless intermediate, but that's not the end of the world.

Edit: I have a mastroska timestamp file that was produced by one of the tools (I forget which one), and by measuring the gap between the timestamps I can easily spot the duplicates with 83ms gaps.
I now have a list of the frame timestamps that are duplicates, I will try to feed that into avisynth.

Just realised you were involved in that post

So, I have a list of frame numbers I need to insert new frames at (around 600), ReplaceFrameSVPFlow looks promising, but what is the best way to do that without copying the function 600 times?

Final Edit: I used the wrong filldrops (filldrops3) rather than your filldrops ported to mvtools2 (as I was using 64bit), and yours has many less false positives. I'm still curious about my last question, but I am content with filldrops.

Thanks again

[jagabo]

Filldrops has a threshold parameter.

fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", "0.1")

The "0.1" at the end is the threshold. Lower it if you are getting too many false positives. Raise it of not enough dups are detected. The other versions have such a threshold too.