Shorter Audio Stream = Delay?

[Djard]

I have a move wherein the audio is skewed, delayed throughout by approximately 150 ms. The speech trails the video. VirtualDub states the following:

Video: 1:32:30.010
Audio: 1:32:29.640

So if the audio is shorter by 370 ms, I figure it should play ahead of the video instead of behind. So am I correct in thinking that a shorter audio is stretched in playback to match video, whereby delaying the audio?

[ndjamena]

The player does whatever the container tells it to.

Which container is the movie in?

[Djard]

The container is AVI, and the asynchrony is about the same, whether I use WMP or MPC-HC.

I'm also wondering what is the most common cause of AV asynchrony: Is it because the start of the movie was trimmed without regard for the I-frame, or is it caused by crappy software? I've seen an audio stream lengthened by an appreciable amount when converted from MP3 to AC3. I suppose there are many reasons.

I'm thinking now my theory is incorrect: if the player automatically made the audio stream longer to match the video length then audio skew would not occur. Instead, the problem would be audio-drift, where the asynchrony would gradually worsen over time.

Any thoughts?

[El Heggunte]

Please tell us what MediaInfo says about that AVI file.

If the audio is AAC and was encoded by NeroAacEnc, or/and the AVI was created by a sub-amateur,
problems like that are expected.

[manono]

So am I correct in thinking that a shorter audio is stretched in playback to match video, whereby delaying the audio?

No. If you found a negative delay at the beginning why didn't you set that delay when muxing (or remuxing) and test it out? Or remove the delay entirely using DelayCut, depending on the kind of audio?

If the audio ends before the video your whole theory goes out the window. That is, it's very possible for you to get the audio in synch by setting a delay of -150ms and than have the audio end 520ms before the video. I'm not claiming that's what's going on, but it's a possibility. Anything can happen with downloaded video files made by idiots.

[jagabo]

The audio and video do not need to start or end at the same time. Nor do they need to be equal length. For example, it's very common on DVDs to have audio that starts earlier or later than the video. When those are converted to other formats it's common for the advance/delay to remain rather than trimming the audio or video at the start.

[Djard]

I wasn't having any difficulty in synchronizing the audio to the video; I was trying to understand how asynchrony works. But as you folks make clear, there are a myriad of possibilities. Seems to me that building web pages from scratch and writing code for Windows is sooooo much simpler than editing video.

Hope you don't mind me picking your brains and asking another question. Would AV asynchrony occur if someone is split or concatenated two or three video segments without regarding I-frames?

Recently I discovered that dragging the progress button all the way to the right in Avidemux didn't mean the selection for trimming was at the end of the video. For some reason I started clicking on the frame-forward icon and notice the time still advance. So now I practice the latter until the time clock stops advancing. So far, every movie I've examined ends on an B-frame or P-frame, whether it has or hasn't an audio sync problem.

Am I correct in assuming that when cutting out a video clip one must always edit from a keyframe (I-frame) but can end on any frame type, without causing an AV sync problem?

[jagabo]

Recently I discovered that dragging the progress button all the way to the right in Avidemux didn't mean the selection for trimming was at the end of the video. For some reason I started clicking on the frame-forward icon and notice the time still advance.

This is probably a "fence post" issue. Say you need to cover a length of 20 feet with a fence. You're going to place posts every 10 feet. How many posts do you need? 20 feet / 10 feet per segment = 2 posts, right? No, you need three fence posts: one at the start, one in the middle, and one at the end. If you leave out the end post you have some fence hanging off the last post. That's most likely what's happening here. Many programs make this mistake.

Am I correct in assuming that when cutting out a video clip one must always edit from a keyframe (I-frame) but can end on any frame type, without causing an AV sync problem?

Assuming your talking only of dumb cut editors: Generally any segment you keep must start on an I frame. It can end on an I or P frame without problems (in the best case scenario). Whether it can end on a B frame may depend on whether that B frame is backward predicted (ie, whether it relies on future frames to be reconstructed).

But the issue is more complicated than that. The order in which frames are stored is not necessarily the same as the playback order. To reduce seek times frames are stored in decoding order, not presentation order. For example, a sequence like IBBP (presentation order) is stored as IPBB. That's because the P frame has to be decoded (probably) before the B frames can be decoded.

There's also the issue of open GOPs. This complicates matter because the last few B frames of the GOP may reference the I frame of the next GOP. Since that I frame has to be decoded before the B frames can be decoded the B frames are stored after the I frame of the next GOP. Ie, instead of ...PBBI... they are store as ...PIBB...

Then there's the issue of how the sound is multiplexed with the video. Audio doesn't necessarily come as convenient "frame sized" packets, ie, one frame of video, on frame worth of audio, one frame of video, one frame worth of audio... Audio may come as fixed fixed 1 second (or any length) packets. It could even come as one big chunk before or after all the video frames (not often because this is very difficult for a player to deal with).

Finally, you have variable bitrates and variable frame rates to further complicate matters.

Different players and editors may handle those issues differently. So "simple" cutting may be much more complex than you imagined.

[Djard]

Thank you for a very clear explanation. Is there a book on the subject of video editing that you might recommend?