How to scan a video file for lost frames/stuttering?

[Shadowlord]

Hi,
I'm searching for a way to detect lost frames/stuttering in videos I've recorded with a capture card (mkv, avc). Maybe some tool that would scan the video and create a list showing when these errors occur in the video.

I've found out that MPC-BE can display a graph (press Ctrl+j), which shows spikes when the video file stutters. I can also activate a statistics windows (press Ctrl+4). Here I can see that the values for "Sync Offset" and "Jitter" change drastically when there's stuttering.
Sadly, MPC-BE does not have an option to log these values.

So, is there any other way to detect and log such errors in a video?


These are the graph/statistics in MPC-BE when there's an error in the video.


These are the graph/statistics in MPC-BE when there's no error in the video.

[jagabo]

MPCBE is showing you playback statistics, not properties of the video.

AviSynth can log duplicate frames with something like:

WriteFileIf("Dups.txt", "YDifferenceFromPrevious<0.5", "current_frame", flush=true)

I've seen scripts that attempt to log skipped frames by looking for differences between frames that are unusually large -- but less than most scene change thresholds.

[Sharc]

VirtualDub(2) menu: Go->next drop frame.
I don't know what the discovery of dropped frame is based on though. Duplicates as a substitute of skipped frames?

[jagabo]

I believe VirtualDub's frame drop detection only detects D frames in AVI files. Some capture programs, when they detect that a frame hasn't show up in time, insert a D frame in the cap. That is a special frame type that indicates a dropped frame (it requires very few bits) and the decoder usually just substitutes a copy of the last frame. It's not an actual copy of the last frame (which requires lots of bits).

[Sharc]

I believe VirtualDub's frame drop detection only detects D frames in AVI files. Some capture programs, when they detect that a frame hasn't show up in time, insert a D frame in the cap. That is a special frame type that indicates a dropped frame (it requires very few bits) and the decoder usually just substitutes a copy of the last frame. It's not an actual copy of the last frame (which requires lots of bits).

I see, thanks. I have always been wondering how this works.

[Shadowlord]

I know that MPC-BE only shows playback statistics, but using these statistics stuttering can be detected very easily while playing/watching the video.

Logging the affected frames through AviSynth sounds good to me, but unfortunately, I have very little experience with their script language. The mentioned code does not work.
Using

DirectShowSource("00.mkv")
WriteFileIf("Dups.txt", "YDifferenceFromPrevious<0.5", "current_frame", flush=true)

it just creates a file which looks like this

0
1
2
3
4
5
6
7
8
9
.
.
.

VirtualDub2 does not detect any dropped frames in my videos. When I use the "next drop frame" function, I receive the message "No next dropped frame found".

However, I know that the video file is faulty, because no matter which media player I use, the stuttering always occurs at exactly the same frames.

[poisondeathray]

It's probably VFR drops (low fps sections) if its in sync . It should be 50 fps CFR if captured properly, but it's 12.86 avg in the "bad" file

Avisynth won't detect that with that script, only the actual encoded frames. (Things like actually encoded duplicates it will detect - but you have sections that are missing, and "held together" in sync by timecodes, telling the player to display the frame longer - effectively lower fps sections)

You can extract the timecodes (e.g. mkvextract for mkv , or mp4fpsmod for mp4) , then use a v2 to v1 timecode converter (e.g. tcconv) to convert them into human readable (v1) format

[johnmeyer]

I took a shot at doing exactly this, in this thread over in doom9,org:

Automatically fix dups followed (eventually) by drops

Finding duplicates, especially if they are exact, is pretty easy using YDifference, as already mentioned. Finding drops is much more difficult because the difference metrics between frames after only one single frame is removed gets to be pretty tough, depending on the motion. To illustrate the point, if you put the camera on a tripod and take a video indoors of an empty room, if you remove a frame, you cannot possibly detect it.

However, the approach I developed does sort of work, despite that, because when you erroneously replace a frame where there is no motion, and that frame didn't really need to be replaced, the replacement algorithm is so perfect that the new frame looks just fine.

So, try it out and see what you think. Use the last script that I posted, not the first one, before I improved it.

[jagabo]

I know that MPC-BE only shows playback statistics, but using these statistics stuttering can be detected very easily while playing/watching the video.

I don't think you understand. The statistics you are getting from MPCBE have to do with timing -- when a frame is supposed to be displayed vs. when it actually displayed. For example, if a frame takes too long to decompress it will be displayed at a later time than it was supposed to -- that timing error that is reported. It cannot detect actual duplicate or dropped frames in a video file.

Logging the affected frames through AviSynth sounds good to me, but unfortunately, I have very little experience with their script language. The mentioned code does not work.
Using

DirectShowSource("00.mkv")
WriteFileIf("Dups.txt", "YDifferenceFromPrevious<0.5", "current_frame", flush=true)

it just creates a file which looks like this

0
1
2
3
4
5
6
7
8
9
.
.
.

Lower the threshold from 0.5 to 0.1 or whatever value eliminates false positives but still shows true duplicates. Also be aware that a video may contain many essentially duplicate frames. For example in a fade to black all the black frames may look like duplicates. Or in a still shot the frames may look like duplicates -- depending on how much film grain was retained during compression (often the first thing to go). And conversely, compression artifacts may make true duplicate frames look different.

DirectShowSource() may work for this but it should generally be used as a last resort (it's not always frame accurate). The best thing for you to do would be to download the LSMASH source filter for AviSynth and use LSmashVideoSource() for MP4 files and LWlibavVideoSource() for others.

[Shadowlord]

Thanks, but after taking a closer look at the solutions, I am not sure that they are leading in the right direction.
As described at the beginning, I would refer to the errors as "lost" or "dropped" frames (I don't know what the correct term is). They are not "duplicate" frames.

To make things even more complicated: The video is 50 fps. The original source is 25 fps, so every frame is a duplicate frame. Of course, this doesn't cause stuttering when playing.

Therefore, I believe jagabo's code and Johnmeyer's script won't work for this problem, because I'm not trying to find duplicate frames.

I don't think you understand. The statistics you are getting from MPCBE have to do with timing -- when a frame is supposed to be displayed vs. when it actually displayed. For example, if a frame takes too long to decompress it will be displayed at a later time than it was supposed to -- that timing error that is reported. It cannot detect actual duplicate or dropped frames in a video file.

Yes, I know that MPC-BEs statistics are based on timing. "The graph shows how much the audio/video timing differs from the ideal situation" - but nevertheless the graph is always showing big spikes when there's an error in the video that I want to detect. I don't know what causes these timing errors.


I've uploaded a small sample, maybe this makes things clearer.

[johnmeyer]

If all you need to do is find and replace duplicate frames (you probably don't want to remove them because that will ruin audio sync) then Filldrops is the function you want. I posted a version for interlaced video here:

FillDropsI()

Here's the same function, but for progressive video:

function filldrops (clip c)
{
  super=MSuper(c,pel=2)
  vfe=manalyse(super,truemotion=true,isb=false,delta=1)
  vbe=manalyse(super,truemotion=true,isb=true,delta=1)
  filldrops = mflowinter(c,super,vbe,vfe,time=50)
  fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", "0.1")
  return fixed
}

You can change how similar the two adjacent frame must be in order for them to be considered duplicates by changing the number after the "lessthan" word in the next to the last line. It is currently set to 0.1. If you change the to 0.0, only absolute duplicates will be detected and replaced with a motion-estimated frame.

This function is nice because it fixes the problem and doesn't require you to output a list of frames and then do something else. If you truly want a list of frame numbers, you'll have to change the conditionalfilter line.

[jagabo]

I've uploaded a small sample, maybe this makes things clearer.

That video is variable frame rate and has bad time codes. In the places where you see the frame lock up and jump there are discontinuities in the time code. Here's an example starting at frame 101

 type,  time,  duration, type, frame_number
-------------------------------------------
video,  2080,  0.020000,    P,   101
video,  2100,  0.020000,    P,   102
video,  2120,  0.020000,    P,   103
video,  2141,  0.020000,    P,   104
video,  2160,  0.020000,    P,   105
video,  2180,  0.020000,    P,   106
video,  2200,  0.020000,    P,   107
video,  2220,  0.020000,    P,   108
video,  2440,  0.020000,    P,   109
video,  2461,  0.020000,    I,   110
video,  2480,  0.020000,    P,   111
video,  2500,  0.020000,    P,   112
video,  2520,  0.020000,    P,   113
video,  2540,  0.020000,    P,   114
video,  2560,  0.020000,    P,   115
video,  2580,  0.200000,    P,   116
video,  2720,  0.020000,    P,   117
video,  2742,  0.020000,    P,   118
video,  2760,  0.020000,    P,   119
video,  2780,  0.020000,    P,   120

Frame_number is the displayed frame number. You can see the table includes frames 101 to 120. Time in that table is the time at which the frame should be displayed (in milliseconds from the start of the video). You can see it stepping at intervals of 20 ms from frame 101 to frame 108, 2080 ms to 2220 ms (except for a minor glitch at frame 104 which is 1 ms off). But thenthere's a big jump to 2440 at frame 109, a 220 millisecond jump. So when the player gets to frame 108 it displays that frame for 220 ms instead of 20 ms. If you step through the actual video frames you can see there is a big jump in motion between frames 108 and 109. Several frames of the original video were dropped between what's now frame 108 and 109. Later you see another jump at frame 117, 140 ms.

<edit>
So it may not be right to say there are errors in the time code. If there are 10 frames (5 unique, 5 duplicates) missing between frame 108 and 109 the time codes are right.

In fact, filling the "gaps" with motion interpolated frames leads to nice smooth motion. Sample with original video on the left and the fixed version on the right attached.

But what you really need to do is get the caps right (no dropped frames) to start with.
</edit>

[Sharc]

So, is there any other way to detect and log such errors in a video?

You could for example analyze the file with ffprobe (included in the ffmpeg package):

ffprobe.exe -show_frames -print_format xml "sample.mkv" >output.xml

[jagabo]

Alternate ffprobe command line with just relevent data:

ffprobe -v quiet -pretty -print_format csv -show_entries "frame=coded_picture_number,pkt_pts,pkt_duration,pict_type," -select_streams v:0 input.mkv >output.csv

[Shadowlord]

@johnmeyer:
I've tried your Filldrops-script, but I don't see any improvement when applied to my sample.mkv. Maybe I'm doing something wrong?

LWLibavVideoSource("sample.mkv")
filldrops()


function filldrops (clip c)
{
  super=MSuper(c,pel=2)
  vfe=manalyse(super,truemotion=true,isb=false,delta=1)
  vbe=manalyse(super,truemotion=true,isb=true,delta=1)
  filldrops = mflowinter(c,super,vbe,vfe,time=50)
  fixed = ConditionalFilter(c, filldrops, c, "YDifferenceFromPrevious()", "lessthan", "0.1")
  return fixed
}

Saved as fix.avs
then run ffmpeg -I fix.avs new.mkv



@jagabo:
Looking at the time code of my sample video does indeed help to see if there are missing/dropped frames. But how can I detect uneven intervals in longer videos? Is there a way to list only uneven intervals?

What did you use to fill these gaps with motion interpolated frames in the sample.mkv.avs.mkv? Was this achieved with an Avisynth filter/plugin?

[jagabo]

Filldrops-script, but I don't see any improvement

It won't work well when multiple consecutive frames are missing. It will only insert one frame per jump.

Looking at the time code of my sample video does indeed help to see if there are missing/dropped frames. But how can I detect uneven intervals in longer videos? Is there a way to list only uneven intervals?

The easiest way would be to write a quick program to subtract successive time codes (from ffprobe) and output the frame number and number of missing frames when the time exceeds a threshold (25 ms?). If you use a script like the one below it would make sense for the program to output the actual AviSynth script.

What did you use to fill these gaps with motion interpolated frames in the sample.mkv.avs.mkv? Was this achieved with an Avisynth filter/plugin?

I used AviSynth and one of the motion interpolation filters, InsertFramesMC(). That's a modification of RX(). Both can be found in these forums. Be warned, this type of motion interpolation works well with panning shots but can introduce ugly artifacts with large and/or complex motions.

LWLibavVideoSource("sample.mkv") 
AssumeFPS(50)

InsertFramesMC(117, 6)
InsertFramesMC(109, 10)
InsertFramesMC(101, 3)

There was at least on small jump before frame 101, and lots of jumps after 117.

[johnmeyer]

I finally looked at the video. It has the wrong speed (50 fps) and is therefore duplicating each frame. That is easy to fix.

Once that is fixed, there are then both jumps and dups, exactly the thing the script I linked to above was designed to fix. The Filldrops() scripts will be useless with this video. In the script I linked to (over at doom9.org), you'll need to use a pretty big window (lots of frames) because the dups and drops are separated by such a large number of frames.

The cause of this is probably a lousy capture card. If you can re-capture, try to tune the software and hardware to avoid the drops. What usually creates this is a capture where a frame is dropped so, to keep the sound in sync, a later frame is duplicated, in order to keep the frame count correct. This can also happen if you use a screen capture utility (e.g., Camtasia) to capture an online video. Some streaming sites use variable frame rate which can also produce this effect which is why I think Jagabo went off in that direction.

[Sharc]

Frame drops and subsequent duplicates to maintain sync during capturing with PC are often caused by temporary system overload (CPU, HDD access and write speed etc.).
Try to disable tasks and processes running in the background like virusscanner, downloads and preparation activities for Windows updates, telemetry, updates of apps etc.

[jagabo]

There are time code gaps of as many as 33 frames in the sample video:

frame_number  missing_frames

   101              3
   109             10
   117              6
   286              2
   299             10
   331             14
   332              1
   338              1
   374             12
   412             27
   531              2
   557             16
   566              2
   894             33
   901              6
   907              6
   917             16
   937             30

Here's an example of InsertFramesMC to fill the gaps. Again, original (converted to CRF) on left, filled on right. You'll see some of the artifacts motion interpolation creates later in the clip. Other motion interpolators might work a little better.

[Shadowlord]

Thank you all!
I can't write a program to check the time codes, but I was able to do something like this in Excel, which I will use to check my videos.

I will also take a closer look at the scripts and filters for filling the gaps, but of course it is more important to capture without any errors in the first place. I believe the errors were caused by a poor HDD access/write speed. I've switched to a faster disk now.

[jagabo]

I wrote a quick dirty program that spit out commands ready to be pasted into an AviSynth script:

InsertFramesMC(   101,  3)
InsertFramesMC(   112, 10)
InsertFramesMC(   130,  6)
InsertFramesMC(   305,  2)
InsertFramesMC(   320, 10)
InsertFramesMC(   362, 14)
InsertFramesMC(   377,  1)
InsertFramesMC(   384,  1)
InsertFramesMC(   421, 12)
InsertFramesMC(   471, 27)
InsertFramesMC(   617,  2)
InsertFramesMC(   645, 16)
InsertFramesMC(   670,  2)
InsertFramesMC(  1000, 33)
InsertFramesMC(  1040,  6)
InsertFramesMC(  1052,  6)
InsertFramesMC(  1068, 16)
InsertFramesMC(  1104, 30)

The sample video in post #19 used that script.

[Shadowlord]

I wrote a quick dirty program that spit out commands ready to be pasted into an AviSynth script

Interesting! May I download this program somewhere?

[jagabo]

It's a quick dirty program, not very robust, with little error handling, and it only handles 50 fps videos. Just a proof of concept really. Don't blame me if it crashes on unusual input. The input must be in the form from the ffprobe command I gave earlier:

ffprobe -v quiet -pretty -print_format csv -show_entries "frame=coded_picture_number,pkt_pts,pkt_duration,pict_type," -select_streams v:0 input.mkv >output.csv

The CSV file should look like:

frame,0,20,I,0
frame,20,20,P,1
frame,40,20,P,2
frame,60,20,P,3
frame,80,20,P,4
frame,100,20,P,5
frame,120,20,P,6
frame,140,20,P,7
frame,160,20,P,8
frame,180,20,P,9
frame,200,20,I,10
frame,220,20,P,11
frame,240,20,P,12

By default the program sends output to the console. To run the program with output to a file:

tcc output.csv >inserts.txt

The output will look like:

InsertFramesMC(   101,  3) #    101   3
InsertFramesMC(   112, 10) #    109  10
InsertFramesMC(   130,  6) #    117   6
InsertFramesMC(   305,  2) #    286   2
InsertFramesMC(   320, 10) #    299  10
InsertFramesMC(   362, 14) #    331  14
InsertFramesMC(   377,  1) #    332   1
InsertFramesMC(   384,  1) #    338   1
InsertFramesMC(   421, 12) #    374  12
InsertFramesMC(   471, 27) #    412  27
InsertFramesMC(   617,  2) #    531   2
InsertFramesMC(   645, 16) #    557  16
InsertFramesMC(   670,  2) #    566   2
InsertFramesMC(  1000, 33) #    894  33
InsertFramesMC(  1040,  6) #    901   6
InsertFramesMC(  1052,  6) #    907   6
InsertFramesMC(  1068, 16) #    917  16
InsertFramesMC(  1104, 30) #    937  30

You must use an AviSynth source filter that outputs VFR frames (ie, does not automatically insert dups for longer duration frames). I use LWlibavVideoSource(). And you must AssumeFPS(50). A full AviSynth script will look like:

LWLibavVideoSource("sample.mkv") 
AssumeFPS(50)

InsertFramesMC(   101,  3) #    101   3
InsertFramesMC(   112, 10) #    109  10
InsertFramesMC(   130,  6) #    117   6
InsertFramesMC(   305,  2) #    286   2
InsertFramesMC(   320, 10) #    299  10
InsertFramesMC(   362, 14) #    331  14
InsertFramesMC(   377,  1) #    332   1
InsertFramesMC(   384,  1) #    338   1
InsertFramesMC(   421, 12) #    374  12
InsertFramesMC(   471, 27) #    412  27
InsertFramesMC(   617,  2) #    531   2
InsertFramesMC(   645, 16) #    557  16
InsertFramesMC(   670,  2) #    566   2
InsertFramesMC(  1000, 33) #    894  33
InsertFramesMC(  1040,  6) #    901   6
InsertFramesMC(  1052,  6) #    907   6
InsertFramesMC(  1068, 16) #    917  16
InsertFramesMC(  1104, 30) #    937  30

The numbers after the hash marks (comments, for reference) are the original frame numbers and the number of frames to be inserted. Note that after each call to InsertFramesMC() all later frame numbers are increased by the number of inserted frames. The first call to InsertFramesMC() adds three frames -- so the second call to InsertFramesMC() is specified at frame 112, not the original frame number of 109. So you can't remove a line (or lines) without modifying all the subesequent lines.

A 64 bit exe and ANSI C source code is attached.

[Shadowlord]

It's a quick dirty program, not very robust, with little error handling, and it only handles 50 fps videos. Just a proof of concept really. Don't blame me if it crashes on unusual input.

Thanks for the program and the instructions on using it!