Analog Capture: inserted and dropped frames

[lollo]

When dealing with analog signals, the captured video stream can suffer of two different problems: inserted frames and dropped frames.

A dropped frames can happen at any step of the signal path, i,e, in the player (VCR), in the external devices (DVD-R in pasthrough mode, external TBC, analog procamp, video mixers, ...), in the capture software (caused by an instable signal or by a PC momentary weakness).

An inserted frame is only created by the capture software in an attempt to keep the audio and video stream in synch. It occurs when a frame does not arrive at the expected time, and is then replaced by the previous frame or it may be created because the internal procedures of the capture software decides to replicate a frame to keep audio and video in synch.

The two problems are then different, and may be indipendent (not be related to each other) or correlated (a dropped frame at some point in time may generate an inserted frame later in time).

AmarecTV provides a clear reporting of what happens (at capture software level) while capturing an analog signal, containing all the details of the captured/dropped/inserted video and audio frames with associated timing.

An inserted frame is not a real frame duplicated, but a few bytes in the video stream instructing the player to repeat the previous frame while playing the video. A part the report file generated from AmarecTV, it can then be easily detected by a video software (i.e. VirtualDub) because this clear pattern. After loading a video file in VirtualDub the inserted frames can be easily found using its menu; unfortunately the naming in VirtualDub is wrong and what is called next/previous dropped frame is in fact a next/previous inserted frame.

A dropped frame is not obviously mapped as such, and then is impossible for a video software to detect it.

A trained human eye can identify an inserted frame (pause in the motion) and a dropped frame (jump in the motion) if the video itself contains a clear advancement of its content across the frames, which is not so common.

In the following an example of an inserted frame and of a dropped frame, comparing two different captures of the same video content, one of them featuring the problem and the other not. In the case of inserted frame, fot this example there is not a correspondant dropped frame.

Inserted frame

capture with problem: ufo_s3b_amtv.avi

AmarecTV report (reduced to concerned portion):

...
AT=00:43:17.788s(64939f), Bsy=  0ms, Dif=-4032, Smp=9600
VT=00:43:17.829s(64943f), Cap=65314f(  0D), Enc= 3,464ms, Siz= 302KB( 37%)KEY, Drp=0, (+)0, (-)0, Buf= 1, o
VT=00:43:17.870s(64944f), Cap=65315f(  0D), Enc= 4,185ms, Siz= 304KB( 37%)KEY, Drp=0, (+)0, (-)0, Buf= 1, o
VT=00:43:17.908s(64945f), Cap=65316f(  0D), Enc= 2,683ms, Siz= 305KB( 37%)KEY, Drp=0, (+)0, (-)0, Buf= 1, o
VT=00:43:17.950s(64946f), Cap=65317f(  0D), Enc= 6,426ms, Siz= 305KB( 37%)KEY, Drp=0, (+)0, (-)0, Buf= 1, o
AT=00:43:17.950s(64944f), Bsy=  0ms, Dif=-5952, Smp=9600
VT=00:43:17.991s(64947f), Cap=65318f(  0D), Enc= 6,311ms, Siz= 305KB( 37%)KEY, Drp=0, (+)0, (-)0, Buf= 1, o
NT=00:43:17.991s(64948f), Total=1
VT=00:43:18.030s(64949f), Cap=65319f(  0D), Enc= 6,509ms, Siz= 303KB( 37%)KEY, Drp=0, (+)1, (-)0, Buf= 1, o
VT=00:43:18.070s(64950f), Cap=65320f(  0D), Enc= 4,646ms, Siz= 302KB( 37%)KEY, Drp=0, (+)1, (-)0, Buf= 1, o
VT=00:43:18.108s(64951f), Cap=65321f(  0D), Enc= 3,583ms, Siz= 303KB( 37%)KEY, Drp=0, (+)1, (-)0, Buf= 1, o
VT=00:43:18.149s(64952f), Cap=65322f(  0D), Enc= 3,674ms, Siz= 304KB( 37%)KEY, Drp=0, (+)1, (-)0, Buf= 1, o
VT=00:43:18.188s(64953f), Cap=65323f(  0D), Enc= 3,734ms, Siz= 303KB( 37%)KEY, Drp=0, (+)1, (-)0, Buf= 1, o
IT=00:43:18.188s, HDD=7,6MB/s( 95%), fps=25,0f/s
AT=00:43:18.188s(64949f), Bsy=  0ms, Dif=-2112, Smp=9600
VT=00:43:18.228s(64954f), Cap=65324f(  0D), Enc= 3,644ms, Siz= 305KB( 37%)KEY, Drp=0, (+)1, (-)0, Buf= 1, o
VT=00:43:18.268s(64955f), Cap=65325f(  0D), Enc= 3,646ms, Siz= 306KB( 37%)KEY, Drp=0, (+)1, (-)0, Buf= 1, o
VT=00:43:18.308s(64956f), Cap=65326f(  0D), Enc= 3,665ms, Siz= 306KB( 37%)KEY, Drp=0, (+)1, (-)0, Buf= 1, o
VT=00:43:18.350s(64957f), Cap=65327f(  0D), Enc= 4,319ms, Siz= 306KB( 37%)KEY, Drp=0, (+)1, (-)0, Buf= 1, o
...
VT=00:46:11.426s(69284f), Cap=69654f(  0D), Enc= 1,823ms, Siz= 230KB( 28%)KEY, Drp=0, (+)1, (-)0, Buf= 1, o
AT=00:46:11.426s(69284f), Bsy=  0ms, Dif=-9792, Smp=   0
--------------------------------- Encode  Result ---------------------------------
Compression        : FourCC=HFYU, Description=Huffyuv v2.1.1
Setting            : 720*576, 25,00fps
Original Video Size: 54804MByte
Compress video Size: 19943MByte(36%, 1/2,748), 7368KB/s(58944kbps)
Frame count=69285f, Drp=  0f, Avg Enc=4,395ms, Avg Cmp= 36%

capture without problem: ufo_s3b_amtv_2.avi

AmarecTV report (reduced to concerned portion):

...
AT=00:43:17.788s(64939f), Bsy=  0ms, Dif=-4032, Smp=9600
VT=00:43:17.829s(64943f), Cap=65314f(  0D), Enc= 3,464ms, Siz= 302KB( 37%)KEY, Drp=0, (+)0, (-)0, Buf= 1, o
VT=00:43:17.870s(64944f), Cap=65315f(  0D), Enc= 4,185ms, Siz= 304KB( 37%)KEY, Drp=0, (+)0, (-)0, Buf= 1, o
VT=00:43:17.908s(64945f), Cap=65316f(  0D), Enc= 2,683ms, Siz= 305KB( 37%)KEY, Drp=0, (+)0, (-)0, Buf= 1, o
VT=00:43:17.950s(64946f), Cap=65317f(  0D), Enc= 6,426ms, Siz= 305KB( 37%)KEY, Drp=0, (+)0, (-)0, Buf= 1, o
AT=00:43:17.950s(64944f), Bsy=  0ms, Dif=-5952, Smp=9600
VT=00:43:17.991s(64947f), Cap=65318f(  0D), Enc= 6,311ms, Siz= 305KB( 37%)KEY, Drp=0, (+)0, (-)0, Buf= 1, o
NT=00:43:17.991s(64948f), Total=1
VT=00:43:18.030s(64949f), Cap=65319f(  0D), Enc= 6,509ms, Siz= 303KB( 37%)KEY, Drp=0, (+)1, (-)0, Buf= 1, o
VT=00:43:18.070s(64950f), Cap=65320f(  0D), Enc= 4,646ms, Siz= 302KB( 37%)KEY, Drp=0, (+)1, (-)0, Buf= 1, o
VT=00:43:18.108s(64951f), Cap=65321f(  0D), Enc= 3,583ms, Siz= 303KB( 37%)KEY, Drp=0, (+)1, (-)0, Buf= 1, o
VT=00:43:18.149s(64952f), Cap=65322f(  0D), Enc= 3,674ms, Siz= 304KB( 37%)KEY, Drp=0, (+)1, (-)0, Buf= 1, o
VT=00:43:18.188s(64953f), Cap=65323f(  0D), Enc= 3,734ms, Siz= 303KB( 37%)KEY, Drp=0, (+)1, (-)0, Buf= 1, o
IT=00:43:18.188s, HDD=7,6MB/s( 95%), fps=25,0f/s
AT=00:43:18.188s(64949f), Bsy=  0ms, Dif=-2112, Smp=9600
VT=00:43:18.228s(64954f), Cap=65324f(  0D), Enc= 3,644ms, Siz= 305KB( 37%)KEY, Drp=0, (+)1, (-)0, Buf= 1, o
VT=00:43:18.268s(64955f), Cap=65325f(  0D), Enc= 3,646ms, Siz= 306KB( 37%)KEY, Drp=0, (+)1, (-)0, Buf= 1, o
VT=00:43:18.308s(64956f), Cap=65326f(  0D), Enc= 3,665ms, Siz= 306KB( 37%)KEY, Drp=0, (+)1, (-)0, Buf= 1, o
VT=00:43:18.350s(64957f), Cap=65327f(  0D), Enc= 4,319ms, Siz= 306KB( 37%)KEY, Drp=0, (+)1, (-)0, Buf= 1, o
...
VT=00:46:11.426s(69284f), Cap=69654f(  0D), Enc= 1,823ms, Siz= 230KB( 28%)KEY, Drp=0, (+)1, (-)0, Buf= 1, o
AT=00:46:11.426s(69284f), Bsy=  0ms, Dif=-9792, Smp=   0
--------------------------------- Encode  Result ---------------------------------
Compression        : FourCC=HFYU, Description=Huffyuv v2.1.1
Setting            : 720*576, 25,00fps
Original Video Size: 54804MByte
Compress video Size: 19943MByte(36%, 1/2,748), 7368KB/s(58944kbps)
Frame count=69285f, Drp=  0f, Avg Enc=4,395ms, Avg Cmp= 36%

You can use the following script to play the videos side by side and check yourself the existing frames:

/*
capture 1, inserted frame at 64948, cutted from 64739 to 64964 = 225; inserted frame now at 209

capture 2, no inserted frame, cutted from 64375 to 64599 = 224
*/

video_1="ufo_s3b_amtv.avi"

video_2="ufo_s3b_amtv_2.avi"

v1=AviSource(video_1)
v2=AviSource(video_2)

stackhorizontal(\
subtitle(v1.showFrameNumber(scroll=true),video_1,size=20,align=2),\
subtitle(v2.showFrameNumber(scroll=true),video_2,size=20,align=2)\
)

Dropped frame

capture with problem: ufo_sIII1d_amtv_v2.avi

AmarecTV report (reduced to concerned portion):

...
VT=00:44:45.036s(67126f), Cap=146988f(  0D), Enc= 1,900ms, Siz= 261KB( 32%)KEY, Drp=0, (+)0, (-)0, Buf= 1, o
VT=00:44:45.077s(67127f), Cap=146989f(  0D), Enc= 1,918ms, Siz= 268KB( 33%)KEY, Drp=0, (+)0, (-)0, Buf= 1, o
VT=00:44:45.117s(67128f), Cap=146990f(  0D), Enc= 1,936ms, Siz= 262KB( 32%)KEY, Drp=0, (+)0, (-)0, Buf= 1, o
VT=00:44:45.156s(67129f), Cap=146991f(  0D), Enc= 1,897ms, Siz= 269KB( 33%)KEY, Drp=0, (+)0, (-)0, Buf= 1, o
VT=00:44:45.198s(67130f), Cap=146992f(  0D), Enc= 2,554ms, Siz= 262KB( 32%)KEY, Drp=0, (+)0, (-)0, Buf= 1, o
VT=00:44:45.238s(67131f), Cap=146993f(  0D), Enc= 2,588ms, Siz= 268KB( 33%)KEY, Drp=0, (+)0, (-)0, Buf= 1, o
VT=00:44:45.278s(67132f), Cap=146994f(  0D), Enc= 2,542ms, Siz= 262KB( 32%)KEY, Drp=0, (+)0, (-)0, Buf= 1, o
AT=00:44:45.278s(67124f), Bsy=  0ms, Dif= 5040, Smp=9600
NT=00:44:45.278s(Drop), Total=1
VT=00:44:45.357s(67133f), Cap=146996f(  0D), Enc= 2,109ms, Siz= 262KB( 32%)KEY, Drp=0, (+)0, (-)1, Buf= 1, o
VT=00:44:45.397s(67134f), Cap=146997f(  0D), Enc= 2,309ms, Siz= 268KB( 33%)KEY, Drp=0, (+)0, (-)1, Buf= 1, o
AT=00:44:45.397s(67129f), Bsy=  0ms, Dif= -240, Smp=9600
VT=00:44:45.437s(67135f), Cap=146998f(  0D), Enc= 2,599ms, Siz= 262KB( 32%)KEY, Drp=0, (+)0, (-)1, Buf= 1, o
VT=00:44:45.477s(67136f), Cap=146999f(  0D), Enc= 1,899ms, Siz= 266KB( 32%)KEY, Drp=0, (+)0, (-)1, Buf= 1, o
VT=00:44:45.517s(67137f), Cap=147000f(  0D), Enc= 2,766ms, Siz= 261KB( 32%)KEY, Drp=0, (+)0, (-)1, Buf= 1, o
VT=00:44:45.558s(67138f), Cap=147001f(  0D), Enc= 2,004ms, Siz= 266KB( 32%)KEY, Drp=0, (+)0, (-)1, Buf= 1, o
VT=00:44:45.597s(67139f), Cap=147002f(  0D), Enc= 2,246ms, Siz= 263KB( 32%)KEY, Drp=0, (+)0, (-)1, Buf= 1, o
AT=00:44:45.597s(67134f), Bsy=  0ms, Dif= -240, Smp=9600
...
VT=00:49:47.117s(74677f), Cap=154540f(  0D), Enc= 2,377ms, Siz= 240KB( 29%)KEY, Drp=0, (+)0, (-)1, Buf= 1, o
AT=00:49:47.117s(74674f), Bsy=  0ms, Dif=-4080, Smp=5760
--------------------------------- Encode  Result ---------------------------------
Compression        : FourCC=HFYU, Description=Huffyuv v2.1.1
Setting            : 720*576, 25,00fps
Original Video Size: 59071MByte
Compress video Size: 19555MByte(33%, 1/3,021), 6703KB/s(53626kbps)
Frame count=74678f, Drp=  0f, Avg Enc=2,207ms, Avg Cmp= 33%

capture without problem: ufo_sIII1d_amtv_2_v2.avi

AmarecTV report (reduced to concerned portion):

VT=01:00:40.671s(91018f), Cap=92107f(  0D), Enc= 2,259ms, Siz= 246KB( 30%)KEY, Drp=0, (+)0, (-)0, Buf= 1, o
AT=01:00:40.671s(91014f), Bsy=  0ms, Dif=-2112, Smp=7680
--------------------------------- Encode  Result ---------------------------------
Compression        : FourCC=HFYU, Description=Huffyuv v2.1.1
Setting            : 720*576, 25,00fps
Original Video Size: 71997MByte
Compress video Size: 23322MByte(32%, 1/3,087), 6559KB/s(52475kbps)
Frame count=91019f, Drp=  0f, Avg Enc=2,310ms, Avg Cmp= 32%

You can use the following script to play the videos side by side and check yourself the existing frames:

/*
capture 1, dropped frame at 67133, cutted from 67067 to 67170 = 103; dropped frame now at 66

capture 2, no dropped frame, cutted from 84545 to 84649 = 104
*/

video_1="ufo_sIII1d_amtv_v2.avi"

video_2="ufo_sIII1d_amtv_2_v2.avi"

v1=AviSource(video_1)
v2=AviSource(video_2)

stackhorizontal(\
subtitle(v1.showFrameNumber(scroll=true),video_1,size=20,align=2),\
subtitle(v2.showFrameNumber(scroll=true),video_2,size=20,align=2)\
)

The provided captures are from a TV show, so what I call "video without problem" are captures that have been validated in term of content versus their "digital" broadcasted version and their DVD version, carefully comparing frame by frame the videos.

Here the comparison videos and the video sources and scripts so that anyone can repeat my experiment on his side.

DVB-S dump

video source: ufo02vi_1_2.mpg

comparing script:

# FFmpegSource
loadPlugin("C:\Users\giuse\Documents\VideoSoft\MPEG\AviSynth\extFilters\ffms2_87bae19\x86\ffms2.dll")

video_1="ufo_sIII1d_amtv_2_v2.avi"
video_2="ufo02vi_1_2.mpg"

v1=AviSource(video_1)
v2=FFmpegSource2(video_2).trim(9,0).ConvertToYUY2().Spline36Resize(720,576)

stackhorizontal(\
subtitle(v1.showFrameNumber(scroll=true),video_1,size=20,align=2),\
subtitle(v2.showFrameNumber(scroll=true),video_2,size=20,align=2)\
)

comparison image:


comparison video: compare_dvb-s.mp4

DVD rip

video source: ufo02.mp4

comparing script:

video_1="ufo_sIII1d_amtv_2_v2.avi"
video_2="ufo02.mp4"

v1=AviSource(video_1)
v2=FFmpegSource2(video_2).trim(7,0).ConvertToYUY2().Spline36Resize(720,576)

stackhorizontal(\
subtitle(v1.showFrameNumber(scroll=true),video_1,size=20,align=2),\
subtitle(v2.showFrameNumber(scroll=true),video_2,size=20,align=2)\
)

comparison image:


comparison video: compare_dvd.mp4

I am also attching the entire directory with all sources, files and scripts used for this post if anyone is interested: https://drive.google.com/file/d/1oqCgb5bwUgbas0I4A-nacfhiAArXh9nT/view?usp=sharing

Let me know your feedback. Next topic could be "what to do in post-processing for inserted and dropped frames?"

[dellsam34]

From personal experience from moving from 100% computer based analog to digital conversion and ingest, to only computer digital ingest, to 100% computer-less workflow, most of the problems related to frame issues were in the first category, In the second category where computer is only used to ingest digital signal I rarely get the ingest software to abandon the capture (it's set to stop capture if one frame is dropped), but when it does it is mostly because I was doing something else on the same computer like playing a video or watching something on social media or YT, In the third category I just pop the tape in the VCR and the capture can go for as long as the tape is with no problems of frames at all, The workflows described above are:

1- S-VHS VCR -> USB Capture Device -> PC with SSD (Vdub, AmarecTV)
2- S-VHS VCR -> Analog to SDI Device -> PC with SSD (BM MediaExpress)
3- S-VHS VCR -> Analog to SDI Device -> BM HyperDeck Shuttle 2, SDI recorder with SSD

The 3rd workflow is my final permanent setup, The HyperDeck has a HDMI output for monitoring so I got a new DELL 4k monitor that has the capability of 2 HDMI inputs with PIP or split screen as shown below and a headphone output in addition to built in speakers so I can use the PC for working on the files or doing other tasks on the left and monitor the capture on the right.

[916Area52]

@ PDR...

Very informative, looking forward to your follow up "Next topic could be "what to do in post-processing for inserted and dropped frames?""

[poisondeathray]

@ PDR...

Very informative, looking forward to your follow up "Next topic could be "what to do in post-processing for inserted and dropped frames?""

lollo posted that

[916Area52]

@ PDR...

Very informative, looking forward to your follow up "Next topic could be "what to do in post-processing for inserted and dropped frames?""

lollo posted that

My bad, I hate when that happens

@ lollo...

Very informative, looking forward to your follow up "Next topic could be "what to do in post-processing for inserted and dropped frames?""

[poisondeathray]

What to do about it post capture - depends on what is actually happening in the captured video and the relationship or offset of dupes and drops

An inserted frame is only created by the capture software in an attempt to keep the audio and video stream in synch. It occurs when a frame does not arrive at the expected time, and is then replaced by the previous frame or it may be created because the internal procedures of the capture software decides to replicate a frame to keep audio and video in synch.

What happens to the original expected frame ? Does that not imply it's dropped and replaced by the inserted duplicate frame ?

If you keep inserting video frames, unless there were some complimentary drops somewhere, the total duration of the video is going to increase for each insert . Also, unless the captured audio inserted gaps, or streches to match - it's going to be progressively worsening out of sync - unless there was also a video frame drops

=> Do you ever get independent inserted duplicates without drops on this setup? If so, what happens to the total duration , total framecount, and audio duration ?

An inserted frame is not a real frame duplicated, but a few bytes in the video stream instructing the player to repeat the previous frame while playing the video.

Not sure if I'm misunderstanding what you wrote, but I might phrase that a bit differently - because the end result is a real inserted duplicated frame in the captured video - and that has functional significance. The inserted duplicates shift all the frames and timing of all the subsequent frames. You can see that when comparing the first two videos.

For huffyuv - it's more than a few bytes - the video duplicates are really encoded as full picture worth of bytes as an I frame format - 10 duplicate frames in a 10 frame video increases the video size by about 10 for huffyuv. For lagarith it's about the same size as 1 frame plus about hundred bytes if null frames are enabled. Null frames are essentially an instruction to encode only 1 frame per duplicate and just repeat them for display

Here is an example of a NTSC 720x480 4:2:2 frame. Both display 1 frame and 10 frames respectively in a video player - but not all video players support lagarith null frames, official lagarith supports null frames - Fmpeg/libav variants of lagarith often have problems with lagarith null frames

huffyuv 1 frame 159,740 bytes
huffyuv 10 frames 1,523,492 bytes
lagarith 1 frame 9,272 bytes
lagarith 10 frames 9,488 bytes

In the case of inserted frame, fot this example there is not a correspondant dropped frame.

In the full capture, was there a corresponding drop for each inserted duplicate ?

If not, what is the total frame count and total duration ? Since you have a matching reference version, you can align beginning, end points and compare

If the beginning and end match, and the frame count is the same, but one version has duplicate frames - there has to be drops as well for a CFR video

[dellsam34]

Not necessarily and depends on what causes the drop or insertions, inserted without drop, or dropped without insertion can happen, this is why audio would go out of sync in these situations.

[poisondeathray]

Not necessarily and depends on what causes the drop or insertions, inserted without drop, or dropped without insertion can happen, this is why audio would go out of sync in these situations.

Yes, out of sync was mentioned. It wouldn't be a constant offset, it would be progressively worsening sync because AV durations do not match

If you keep inserting video frames, unless there were some complimentary drops somewhere, the total duration of the video is going to increase for each insert . Also, unless the captured audio inserted gaps, or streches to match - it's going to be progressively worsening out of sync - unless there was also a video frame drops

And assuming that the source was ok to begin with - it's just math . Total duration is a function of FPS, frame count. If duration of audio and video streams don't match, it's going to be progressively out of sync

If you have more frames, the total video duration is going to change for a given constant frame rate, is it not ? What happens to the audio ? Same for drops without inserts. The complimentary drops and inserts is usually the mechanism that keeps the capture in sync so that video stream and audio stream have the same total duration and is mostly in sync (might be 1 frame out of sync in small sections, then get back into sync once the duplicate is inserted)

The reason this is important is you have to examine the relationship of the drops and inserted duplicates in order to do something about it. Each capture setup is different and potentially has it's own issues. There are already scripts for some cases, but not for others. If you don't examine the relationship, good luck fixing that

[Alwyn]

@Lollo, thanks very much for this.

[dellsam34]

I don't think dropped frames can be fixed in post, Inserted without drop maybe, by snipping them off and re-align audio, unless some sort of magic frame interpolation by averaging the motion of the adjacent fields to the dropped frame, but what if more than one frame is dropped?

[Sharc]

I don't think dropped frames can be fixed in post, ....

One can synthesize and insert the dropped frame(s) by temporal interpolating adjacent frames, using temporal interpolators based on mvtools or RIFE for example. The difficulty is if and how reliably such process can be "automated" though. As far as I remember scripts have been proposed in the past by users johnmeyer and StainlessS in this forum or over at doom9.
If there is more than one dropped frame in a row one should consider a re-capture I think.

[lollo]

From personal experience from moving from 100% computer based analog to digital conversion and ingest, to only computer digital ingest, to 100% computer-less workflow, most of the problems related to frame issues were in the first category, In the second category where computer is only used to ingest digital signal I rarely get the ingest software to abandon the capture

Yes, but keep in mind that the loss/insertion of frames related to PC weakness can be reduced to 0 with an appropriate set-up of the hardware and software.
The only loss/insertion of frames left is then due to the instability of the incoming signal.
Moreover, if you remove the PC you cannot use a software like AmarecTV telling you exactely when and where a frame is dropped/inserted (your BM apps do not have that feature), So you are at the mercy of hoping that the incoming digital stream is perfect in term of frames content, which, btw, should be the common case (but you have no monitoring, and I do not like that). On the other hand, you can set the software to stop the capture if a frame is dropped, as you specified later, this being an extreme solition (and you are missing the option to insert frames if needed).

it is mostly because I was doing something else on the same computer like playing a video or watching something on social media or YT

I do not even move the mouse while capturing

An inserted frame is only created by the capture software in an attempt to keep the audio and video stream in synch. It occurs when a frame does not arrive at the expected time, and is then replaced by the previous frame or it may be created because the internal procedures of the capture software decides to replicate a frame to keep audio and video in synch.

What happens to the original expected frame ? Does that not imply it's dropped and replaced by the inserted duplicate frame ?

Not necessarly. To my understanding of the capture software (but I am not the developper of AmarecTV and VirtualDub) If a frame does not arrive at all in the relative time window (2x frame rate) is dropped and it may be replaced or not by an inserted frame; if a frame arrive late but still in time, the previous is duplicated and the audio shifted.
As I said, you may have inserted frames without a corresponding dropped frame (look to the samples in my first post).

To be honest, the principle of frame insertion performed by AmarecTV and VirtualDub is not completely clear in all details to me, I just base my conclusions on test experiences.
I developped my own capture software (based on GraphEdit) and I limited it to just drop frames when they do not arrive in time relying on the Microsoft filter AviMux, which is very strict in this regard:
https://learn.microsoft.com/it-it/windows/win32/directshow/capturing-video-to-an-avi-f...ectedfrom=MSDN
https://learn.microsoft.com/it-it/windows/win32/directshow/avi-mux-filter



I refused / was not able easily to introduce a routine for insertion of frames inside GraphEdit filtering, and just used my capture software to double check AmarecTV, VirtualDub and VirtualVCR behaviour.

If you keep inserting video frames, unless there were some complimentary drops somewhere, the total duration of the video is going to increase for each insert . Also, unless the captured audio inserted gaps, or streches to match - it's going to be progressively worsening out of sync - unless there was also a video frame drops

Yes. In capture of inserted frames, if their number is limited the audio synch is kept, while if you have many the audio tends to anticipate the video, and then async.
Again, the capture software tries to keep a/v synch inserting frames (and probably adjusting audio timestamp), but first, it does not always succeed and second it cannot compensate for largely instable signals.

=> Do you ever get independent inserted duplicates without drops on this setup? If so, what happens to the total duration , total framecount, and audio duration ?

That's what my dedicated sample in post 1 show. In case of inserted frame (without a corresponding dropped frame) the frame count is simply increased by 1.

An inserted frame is not a real frame duplicated, but a few bytes in the video stream instructing the player to repeat the previous frame while playing the video.

Not sure if I'm misunderstanding what you wrote, but I might phrase that a bit differently - because the end result is a real inserted duplicated frame in the captured video - and that has functional significance. The inserted duplicates shift all the frames and timing of all the subsequent frames. You can see that when comparing the first two videos.

There is and additional duplicated frame in the video under all points of view (duration, display, etc.), but I understand that the frame is not phisically present, just an instruction to repeat it. That's why VirtualDub is able to easily find it, and why this "information" of inserted frame disappears if you encode or manipulate the captured stream.

For huffyuv - it's more than a few bytes - the video duplicates are really encoded as full picture worth of bytes as an I frame format - 10 duplicate frames in a 10 frame video increases the video size by about 10 for huffyuv. For lagarith it's about the same size as 1 frame plus about hundred bytes if null frames are enabled. Null frames are essentially an instruction to encode only 1 frame per duplicate and just repeat them for display

I think the capture software acts differently, and does not provide the full duplicate frame to HuffYUV encoding. In any case, the internal format and management of the inserted frame is not that important for our goals, the important is that finally we have an additional frame.

In the case of inserted frame, fot this example there is not a correspondant dropped frame.

In the full capture, was there a corresponding drop for each inserted duplicate ?

No. Once more there are inserted frames without a corresponding dropped frame

If not, what is the total frame count and total duration ? Since you have a matching reference version, you can align beginning, end points and compare

If the beginning and end match, and the frame count is the same, but one version has duplicate frames - there has to be drops as well for a CFR video

They do not match, as showed in the samples in post 1

Not necessarily and depends on what causes the drop or insertions, inserted without drop, or dropped without insertion can happen, this is why audio would go out of sync in these situations.

Yes.

If you keep inserting video frames, unless there were some complimentary drops somewhere, the total duration of the video is going to increase for each insert . Also, unless the captured audio inserted gaps, or streches to match - it's going to be progressively worsening out of sync - unless there was also a video frame drops

Yes. Except that sometime the capture software is able to adjust the audio stream accordingly to the inserted frame.

If you have more frames, the total video duration is going to change for a given constant frame rate, is it not ? What happens to the audio ? Same for drops without inserts.

For dropped frames there is no compensation possible and the audio will go out of synch.

The complimentary drops and inserts is usually the mechanism that keeps the capture in sync so that video stream and audio stream have the same total duration and is mostly in sync (might be 1 frame out of sync in small sections, then get back into sync once the duplicate is inserted)

Yes, but once more it is not the only one. I found it only with really bad incoming signals, where the software tries to drop and insert many frames to keep stability. For 99% clean signal this mechanism of insertion of a single frame without a corrispondent dropped frame is quite common.

looking forward to your follow up "Next topic could be "what to do in post-processing for inserted and dropped frames?""

I don't think dropped frames can be fixed in post, Inserted without drop maybe, by snipping them off and re-align audio, unless some sort of magic frame interpolation by averaging the motion of the adjacent fields to the dropped frame, but what if more than one frame is dropped?

In case of inserted frame, if they are just 1/2 per hour there is no need to anything in post-processing. If they are many and the audio is not in synch, they can be removed, but the a/v synch must be checked, because is not insured.
My recommendation is to recapture if there are too many inserted frames.

Sharc already answered for dropped frames. To complete, in case of a bad frame like this:



I use an interpolation, see here: https://www.digitalfaq.com/forum/video-restore/12929-how-replace-frames.html#post86442 (but RIFE is better than MVTools2).
The same can be used for filling dropped frames.

The data contained in the AmarecTV report file can be used to automate the process

[poisondeathray]

As I said, you may have inserted frames without a corresponding dropped frame (look to the samples in my first post).

I looked at the cut sample - I was asking about about the full capture. I didn't download the zip, just the 1st two samples.

The reason I'm asking is you need to enter information of the number of dupes/drops , cycle parameters for some of the scripts that deal with displaced dupes/drops to work. They need to know how "far" to look and the approximate number of dupes to delete and/or inserts for interpolated frames

Except that sometime the capture software is able to adjust the audio stream accordingly to the inserted frame.

For 99% clean signal this mechanism of insertion of a single frame without a corrispondent dropped frame is quite common.

In those cases where you have sync, but only inserted frames (or more inserted frames than drops) - you have more total frames, longer audio duration than you should have for a "perfect" capture . A 2 hour movie might now be 2 hours and 1/2 second , something like that ?

For Alwyn an the other thread who I believe have similar setup, mentioned "even if I find some dupes, there's not much I can do about them, as removing them would upset the audio sync." - If you assume only inserted frames (or more inserted frames than drops) , extended duration - is that a common pattern for this setup ?

[lollo]

I looked at the cut sample - I was asking about about the full capture

The whole captures (1.5 hours) of the samples I posted only contain 1 inserted frame in one case and 1 dropped frame in the other case. Nothing else.

The reason I'm asking is you need to enter information of the number of dupes/drops , cycle parameters for some of the scripts that deal with displaced dupes/drops to work. They need to know how "far" to look and the approximate number of dupes to delete and/or inserts for interpolated frames

I take that info (location in time / frame number) about dropped and inserted frame from the AmarecTV report; no need to search for anything or to specify cycle parameters as I do not use any existing script (developed for film transfer essentially). Just a trim out of the inserted frames or an interpolation of a dropped frames is required, quite easy to handle.

In those cases where you have sync, but only inserted frames (or more inserted frames than drops) - you have more total frames, longer audio duration than you should have for a "perfect" capture . A 2 hour movie might now be 2 hours and 1/2 second , something like that ?

Yes, in principle. But the capture software may also adjust the audio stream accordingly sometimes (with an unknown mechanism to me). I do not have more than a couple of inserted frames for clean captures, so I cannot really conclude on that. For captures where the signal is corrupted, the number of inserted and dropped frames is quite high, but in this case is difficult for the capture software to keep audio and video in synch, even with the dropped / inserted frames approach. Sometime it succeeds, but is not certain. So difficult to generalize and extract a clear pattern.

What is maybe worth to mention is that I have a specific tapes where across 5 different captures I always get an (1) inserted frame at the same time / frame number, meaning that for sure there is a (smal) defect in the tape at a given location and that the capture/insertion procedure is predictable, then quite solid.

[poisondeathray]

The whole captures (1.5 hours) of the samples I posted only contain 1 inserted frame in one case and 1 dropped frame in the other case. Nothing else

Thx for clarification

I take that info (location in time / frame number) about dropped and inserted frame from the AmarecTV report; no need to search for anything or to specify cycle parameters as I do not use any existing script (developed for film transfer essentially). Just a trim out of the inserted frames or an interpolation of a dropped frames is required, quite easy to handle.

You mentioned the report deals with the capture software level. What about other problems not found in the report ? (It's nice to have a list of already known issues)

In those cases where you have sync, but only inserted frames (or more inserted frames than drops) - you have more total frames, longer audio duration than you should have for a "perfect" capture . A 2 hour movie might now be 2 hours and 1/2 second , something like that ?

Yes, in principle. But the capture software may also adjust the audio stream accordingly sometimes (with an unknown mechanism to me). I do not have more than a couple of inserted frames for clean captures, so I cannot really conclude on that. For captures where the signal is corrupted, the number of inserted and dropped frames is quite high, but in this case is difficult for the capture software to keep audio and video in synch, even with the dropped / inserted frames approach. Sometime it succeeds, but is not certain. So difficult to generalize and extract a clear pattern.

I was under the impression, at least for alywn case, that there were more than a few inserts (or a least reported ones)

For the situation where you have more inserted frames than drops, and it's in sync, what does the audio waveform typically look like near the inserts? (The uploaded sample has silent audio) . ie. What is the device actually doing with the audio in terms of the waveform during the insert - and what are the ramifications of trimming video and audio for 1 frame where an insert occurs ? Can you hear a glitch, etc...

Some capture setups have a generalized pattern / behaviour and somewhat consistent in the way they handle certain problems. Drop then dupe +/- displacement the most common. But john meyer posted about a client setup where the device would consistently insert a duplicate before a drop. That's exceedingly rare, almost as if the device was precognizant or could predict what's coming. Scripts which would assume drop, dupe would fail that scenario.

[lollo]

Thx for clarification

You're welcome.

You mentioned the report deals with the capture software level. What about other problems not found in the report ? (It's nice to have a list of already known issues)

The other sources for dropped frames are in the hardware before the signal enters into the capture card (and then under capture software control). There is no way to locate them in the final captured video except using a slow motion seen with expert eyes, and depending on the video content.
They can be generated by the player or any other element in the signal path, and then not recognized by the capture card/software, because baked in the input signal.
The wished list of known issues is then limited to these basic considerations. In the past there has been an attempt to create specific test pattern to record on tape and to capture among doom9 forum users to identify all possible combinations, with no concrete results.

For the situation where you have more inserted frames than drops, and it's in sync, what does the audio waveform typically look like near the inserts? (The uploaded sample has silent audio) . ie. What is the device actually doing with the audio in terms of the waveform during the insert - and what are the ramifications of trimming video and audio for 1 frame where an insert occurs ? Can you hear a glitch, etc...

The audio packet is non existing at the time of the inserted frame (which is understandable, given the reason for inserting a frame). I will try to confirm this across multiple cases, because most of my saved capture are clean in term of inserted frames (I usually discharge and recapture in case of errors, so I have to recapture in purpose some well know difficult tape).
In term of consequences of trimming out an inserted frame I see no bad consequences on the audio pattern (again to be confirmed on a larger set of test cases).

Some capture setups have a generalized pattern / behaviour and somewhat consistent in the way they handle certain problems. Drop then dupe +/- displacement the most common. But john meyer posted about a client setup where the device would consistently insert a duplicate before a drop. That's exceedingly rare, almost as if the device was precognizant or could predict what's coming. Scripts which would assume drop, dupe would fail that scenario.

johnmeyes is more oriented to film capture than vhs/s-vhs capture, which may explain some more consistent behaviour found. He may chime in to give his point of view on this subject.