Audio slowly goes out of sync

[ratball3d]

Hello. I have quite a bit of experience working with video and I'm usually able to find solutions to problems online myself, but I cannot solve the problem I'm having now for the life of me. This problem is so persistent that I've gone through the trouble of making an account on this site just to seek help, something I've never done before.

I've recently undertaken a project in which I record footage using a PCIe Hauppauge capture card and dub it over using audio recorded by an SD Sony Handycam. The captured footage is not edited or spliced in any way; the capture card audio and the camcorder audio is simply mixed together and re-multiplexed with the original footage, with no change in the total duration of the audio. No matter what combination of frame rates, sample rates, video codecs, or audio codecs I use, the sound always starts off synchronized at the beginning and gradually slips out of sync as the video progresses.

Now for the details. The Hauppauge capture card records 8500 kbps MPEG2 video at 29.970 FPS (interlaced, bottom field first) with 128 kbps CBR MP2 audio sampled at 48 kHz, multiplexed in an MPEG2 transport stream. The original video plays back perfectly in both Media Player Classic and VLC. I then feed the transport stream to HDTVtoMPEG2 (downloaded here: https://www.videohelp.com/tools/HDTVtoMPEG2) and play back the resulting MPEG2 program stream with no problems.

The camcorder records 9100 kbps MPEG2 video at 29.970 FPS (interlaced, top field first) with 256 kbps CBR AC3 audio sampled at 48 kHz, multiplexed in a DVD-compliant MPEG2 program stream. Again, the original video plays back with no problems. I only use the camcorder footage for the audio; the video is discarded.

I've tried several combinations of video software to dub the Hauppauge footage, but the end result is always the same. First, I demux the Hauppauge audio and camcorder audio into their respective elementary streams. Using either Sony Vegas 10a or Audacity 2.1 (I've tried both) with both project and encoder settings set to 48 kHz, I line up the camcorder audio and the Hauppauge audio and export the project as either AC3 or MP2 (I've tried both). When I remux the new audio track with the original video using avidemux (after applying the appropriate audio offset), the newly created video appears to be normal for the first two minutes. Then, at about three minutes, the audio appears to play ahead of the video by about 500 milliseconds. Seeking further into the video (I've tried pressing stop or pause first, it doesn't make a difference), the audio is even further ahead, until it is off sync by about seven seconds at the end of the hour-long video. When I play back the video and two separate audio tracks from within Vegas, the sound is perfectly in sync all the way through. I've even tried completely re-encoding the entire project as an h.264 MP4 with Vegas and an h.264 MP4 (AVC) and xvid MP4 (ASP) with avidemux, and the audio sync problem persists across all resulting video files.

I double-checked the frame rate to ensure that it wasn't changed, and sure enough it was still 29.970 FPS. I double-checked the audio to ensure that the total duration hadn't changed, and the new audio was exactly the same length as the old audio. The sample rate was also unchanged at 48 kHz, but I've even tried using 41.1 kHz and got the same result. I've tried rigging it by stretching the audio, speeding up the video, and every combination inbetween, but still the audio utterly refuses to play back in sync with the video. The severity of the synchronization problem varies slightly between solutions, but the overall result does not change.

I don't know what else to do. I've spent over 40 hours painstakingly testing countless combinations of video and audio codecs, containers, and settings, and I'm at the point of abandoning the project altogether. I've even formatted my computer, reinstalled Windows, and toggled the playback sample rate and bit depth under playback devices. I give up.

[Ennio]

Reading all this I don't think it's in the audio, but the video. Did you try running the m2v stream through restream or dvd-patcher?

[geneaum]

Sounds to me like a 'clock drift' issue. I've had something similar with my work main conf room electronics - the audio and video paths are so complex involving different devices that the recorded av has always had sync issues, including drift.

I've never been able to completely solve this, but am revamping the whole system and am keeping simplicity in mind wrt the configuration.

Have you run tests starting with capture at the end point (computer) only? Then add components to see when the drift starts.


Good luck on this one. Steve

[sambat]

I assume that if you put the original demuxed audio back with the original demuxed video, there's no problem?

[ratball3d]

It's very strange. I can play back the raw video with no issues, but if I modify it in the slightest way using a video editor, the audio drift occurs. I've even tried demuxing the audio and immediately remuxing it back again, and suddenly the audio slips out of sync upon playback.

However, I've found the solution.

When the audio slips out of sync, I noticed that it always plays ahead of the video, which suggests that the sample rate has something to do with the problem. I previously tried messing with the sample rate using the video software, but it was resampling the audio without changing its duration.

This time, I fed the demuxed audio track to Audacity and gradually reduced the source sample rate to increase the total duration of the audio, and exported the track at 48000 Hz. After trying a few different values, I found that the magic number is 47874 Hz. Now the audio plays in sync to the end.

It's still an utter chore to edit though. Vegas has no sync issues using the original video, but it gets horribly off sync with the new audio. Then when I try to add the camcorder dub to the mix, those two tracks no longer line up either. I fixed one problem but introduced another.

Meh, it's a pretty weak victory.

[turk690]

In a professional setup like a broadcast studio, there is house sync which feeds all audio and video devices. All of what is recorded on separate media are in perfect sync without missing a beat. We like to think that electronics are absolute devices that keep perfect time; however, the quartz crystals and the circuitry that support them eventually drift in varying degrees. This is why in more domestic situations that you and I on this forum are likely to encounter, where there could be maybe 3 camcorders and an audio recorder that stand alone and have no connection to one another in any way, drift between audio & video does and IS expected to happen. You can try to mux, demux, remux, convert to other formats, vary bitrates, pull out your teeth, and they will still all merrily drift. In my experience, having shot a number of events, over a two-hour period, the camcorder video is shorter by anywhere from 5 to 10 frames or more compared with captured audio. I normally use a Zoom R16 to capture audio, and Canon XA-10 & HFG10, and Sony CX12, SR12, and HC9 for video. The Canons drift more; the Sony SR12 has been known to be in perfect sync for more than an hour, in two hours not drifting more than 3 frames max. The differences in audio and video length are obvious and easy to see. I normally first mix down and process the R16 audio and lay it as first audio track in a Premiere Pro project. Then in the video tracks I drag the AVCHD *.mts files. After syncing the 1st video clip to the audio, then enabling snapping on the timeline, at the end of about two hours one can see that video is shorter than the audio and has gone out of sync. Very fortunately, both Sony and Canon camcorders capture AVCHD in 2GB chunks, which, depending on bitrate, are from 11 to 16minutes in duration apiece. I simply disable snapping, then click the middle part of each video clip and move and sync it to the audio file at that point in the timeline . After this is done, there will be a gap of 1 to 3 frames between video clips on a track, but since this is a multicam setup, gaps happen on different parts and never coincide between any two video tracks. I have observed that it's only noticeable & unpleasant if audio and video are significantly more than 3 or 4 frames apart. With the method I have mentioned, around the middle of any video clip is in perfect sync with the R16 audio, and any part of the video clip is never more than 1 or 2 frames out-of-sync. We all like to go brute force or fancy and stretch and all that to sync audio with video, but with the method I do, it's not necessary to go there. All that is needed is an expanded timeline, moving the video clips back & forth and that's it. If the video is not in chunks (such as that captured as DV AVI), I just cut it in 15minute chunks and move each chunk to sync with the audio. The resulting gaps of 1 or 2 frames between chunks after syncing the lot is not a problem with a multi-cam setup. With a single cam, external (non-camcorder captured) audio can also be cut, the ends of each cut rolled back a second each (30 frames or so), video is similarly dealt with, then both are linked, then crossfaded with the next audio & video-linked clip.

[ratball3d]

I've been experimenting with various settings and I believe I've refined the process down to a science. I'm posting this in hopes of helping others who are being frustrated with similar issues.

To begin, the two programs I used were Avidemux and Audacity with the LameMP3 and FFMPEG plug-ins.

First, I open the source video in Avidemux. On the menu bar, under Audio, I select Save and save the audio track with the extension found under File->Information (in this case, *.mp2). I then set the audio delay to the value found under Audio->Main Track.

After that, I seek about 5 minutes into the video and play. Listening carefully, I determine if the audio is playing ahead or behind. This will determine whether the audio sample rate needs to be increased or decreased. Next, I seek to almost the end of the video and look for an event that makes a distinct sound. Stepping the video along frame by frame, I note the exact time (hh:mms.ms) of the action.

In Audacity, I set the project sample rate to 48000 Hz and the bit depth to 16 bits. After opening the demuxed audio track, I seek to the time noted earlier, and then advance the cursor by the amount of time equal to the audio delay detected by Avidemux. I add (if the audio is behind) or subtract (if the audio is ahead) 100 Hz from the track sample rate to get it to within 10-15 seconds of where it should be. I play the video from the cursor position and listen for the sound that is supposed to occur. Once I hear the sound, I note its shape on the waveform and then fine tune the track sample rate until the beginning of the sound lines up with the cursor. At that point, I can export the project as a *.wav or *.mp2, it doesn't really matter because Avidemux can re-encode the audio.

Back in Avidemux, under Audio->Main Track, I browse for the external audio track I just exported from Audacity. When I replay the video from the time noted at the beginning, the audio is perfectly in sync.

Finally, I leave the Video selector set to Copy and set the Audio selector to whatever codec that works with the selected format. The saved video works perfectly.