How to fix Audio Sync Drift using what MediaInfo calls "Audio FPS"?

[EmmB]

Now, I realize that the predominant view is that the audio track(s) of a video have no such attribute as FPS or "frame rate", but when I run MediaInfo on a particular video file, it's right there. Here's it's output for the Audio track:

Audio
ID : 2
Format : AAC LC
Format/Info : Advanced Audio Codec Low Complexity
Codec ID : A_AAC-2
Duration : 1 h 22 min
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 44.1 kHz
Frame rate : 43.066 FPS (1024 SPF)
Compression mode : Lossy
Default : Yes
Forced : No
Duration_Source : General_Duration


The reason I bring this up is because there's a considerable audio sync drift in that video. At the start it's not really perceptible, but within even 20 minutes it's extremely bad -- perhaps 10 seconds off. What I've been trying without success for at least a week is to eliminate the drift and get the audio/video to sync correctly.

Note that all the other mp4/mkv videos I've examined with MediaInfo showed common values for Audio FPS that match the video FPS, such as 29.97, 30, etc.

I've tried the following tools without success: BeLight, MeGUI, HandBrake, FFCoder, StaxRip, UsEac3to128 (GUI), Avidemux, MKVToolNix, TMPGEnc Video MasteringWorks 7, and perhaps others I don't recall. Most of them only provide basic correction for linear lip sync problems, but not drift. Those that appear to provide drift correction will only adjust the audio within an extremely narrow range (like 25 to 24, etc). None of even this subset could correct for such a large difference in audio "FPS".

Is there anything new as of September 2020 that could solve this problem?


Thanks!

[Cornucopia]

That framerate is a red herring. Perhaps it should be called "compressed bit packet rate" because the data is bunched up in packets and sent packet by packet, rather than constant sample by sample streams. Remember, compressed structures no longer bear a resemblance to their uncompressed sources (timebase, samplerate, bitdepth, etc). At playback, those packets can be fit in via their muxing cadence to stream in a pulse method, where a receiving buffer stores, reassembles them, and outputs them at the appropriate Sample rate (with a slight, but known and steady delay/latency).

Look in other areas - like the sample rate flag saying 44.1 when it was actually 48. Or saying 44.1 when it was actually 44.056 (the audio pulldown equivalent of 30 vs. 29.97).

To be clear: there is no such thing as FPS in audio! Ever.


Scott

[jagabo]

AAC frames have nothing to do with video frames. Just like they have nothing to do with picture frames, bed frames, car frames, etc.

https://wiki.multimedia.cx/index.php/Understanding_AAC#Frames_And_Syntax_Elements

If it's a linear drift, eg 1 second off after 1 minute, 2 seconds off after 2 minutes, 3 seconds off after 3 minutes... you can adjust the overall length of the audio to get it to match the video.

If it's a non-linear drive, eg in sync for the first 5 minutes, then 10 seconds off for the next 20 minutes, then 50 seconds off for the next minute... you'll have to edit manually adjust the audio in an editor.

[ProWo]

Demux your video and control with mediainfo the duration (length) of the video and audio stream. These should be of equal length. If not, adjust the audio stream to the duration of the video stream, e.g. with Audacity. Then remux the two tracks again.
If the audio track is from a different video with different FPS (e.g. 23.976fps source and 25fps target), you can adjust it with my clever FFmpeg-GUI.

[jagabo]

Audio and video don't necessarily start at the same time, end at the same time, nor have the same length.

[ProWo]

Audio and video don't necessarily start at the same time, end at the same time, nor have the same length.

This is correct, but in most cases they are the same length and start at the same time.

[Cornucopia]

Most cases of desync? I doubt that, @ProWo.


Scott

[ProWo]

Most cases of desync? I doubt that, @ProWo

Not of desynch, Scott. In the normal case.

[hello_hello]

That framerate is a red herring. Perhaps it should be called "compressed bit packet rate" because the data is bunched up in packets and sent packet by packet, rather than constant sample by sample streams. Remember, compressed structures no longer bear a resemblance to their uncompressed sources (timebase, samplerate, bitdepth, etc). At playback, those packets can be fit in via their muxing cadence to stream in a pulse method, where a receiving buffer stores, reassembles them, and outputs them at the appropriate Sample rate (with a slight, but known and steady delay/latency).

Look in other areas - like the sample rate flag saying 44.1 when it was actually 48. Or saying 44.1 when it was actually 44.056 (the audio pulldown equivalent of 30 vs. 29.97).

To be clear: there is no such thing as FPS in audio! Ever.


Scott

http://www.mp3-converter.com/mp3codec/frames.htm

Frames per Second

Just as the movie industry has a standard that specifies the number of frames per second in a film in order to guarantee a constant rate of playback on any projector, the MP3 spec employs a similar standard. Regardless of the bitrate of the file, a frame in an MPEG-1 file lasts for 26ms (26/1000 of a second). This works out to around 38fps. If the bitrate is higher, the frame size is simply larger, and vice versa. In addition, the number of samples stored in an MP3 frame is constant, at 1,152 samples per frame.

For the AAC audio in the opening post, MediaInfo is showing 1024 samples per frame (time domain) and 43.066 FPS. At 48k instead of 44.1k it'd be 1024 samples per frame and 46.875 FPS.

[hello_hello]

I've tried the following tools without success: BeLight, MeGUI, HandBrake, FFCoder, StaxRip, UsEac3to128 (GUI), Avidemux, MKVToolNix, TMPGEnc Video MasteringWorks 7, and perhaps others I don't recall. Most of them only provide basic correction for linear lip sync problems, but not drift. Those that appear to provide drift correction will only adjust the audio within an extremely narrow range (like 25 to 24, etc). None of even this subset could correct for such a large difference in audio "FPS".

The video and audio frame rates aren't connected, so what MediaInfo is showing doesn't relate to the problem.

Try MKVToolNixGUI but change the video frame rate (you can only stretch audio by a percentage with MKVToolNixGUI, and it tends not to work well anyway).
You probably won't need to change it by much. If the video frame rate is supposed to be 23.976 fps, try 23.9755, or 23.9765 fps, depending on whether you need to speed the video up or slow it down. A bit of trial and error might do it, assuming the video and audio really do match.
If the video is variable frame rate though, specifying a new constant frame rate with MKVToolNixGUI won't fix the A/V sync problem. It'll probably make it worse, so check that MediaInfo reports constant frame rate for the video before you bother changing the frame rate that way.

If you're an Avisynth user you can try the AudioSpeed function in my signature. It'll let you change the speed of the audio by any percentage. Once again, it'll probably take some trial and error, and of course you'll have to re-encode the audio if you do get it to line up with the video, but it might be easier than repeatedly remuxing with MKVToolNixGUI until you get it right. I'd create the script with AudioSpeed included and check the A/V sync by opening the script with MPC-HC.
You don't have to specify a numerator and denominator for changing the audio speed (as shown in the example) a single value will do, where 1.0 is the original speed (no change). ie AudioSpeed(1.002) or AudioSpeed(0.9998) etc.
Using the video duration and original audio duration as numerator and denominator might be a good starting point though (specified in seconds and milliseconds).

[jagabo]

You probably won't need to change it by much. If the video frame rate is supposed to be 23.976 fps, try 23.9755, or 23.9765 fps, depending on whether you need to speed the video up or slow it down.

If the OP's description is correct -- it will take a larger change than that. 10 seconds after 20 minutes is about 0.83 percent. Around 24.175 or 23.777.

[hello_hello]

You probably won't need to change it by much. If the video frame rate is supposed to be 23.976 fps, try 23.9755, or 23.9765 fps, depending on whether you need to speed the video up or slow it down.

If the OP's description is correct -- it will take a larger change than that. 10 seconds after 20 minutes is about 0.83 percent. Around 24.175 or 23.777.

I guess I should've done the math.

[Sartori]

Ive done this in the past with mpgs, demuxed, converted to wav, imported the video and audio into Vegas as separate stream (or any editor that allows shifting of the audio).
Playback the video/audio in the editor, ensuring it's aligned at the beginning (note the time of the point you are checking)
Go to the end of the video, see if it's misaligned there (note the time of the point you are checking)
Shift the audio on the timeline until the second point you checked is synced
Note on the timeline how much you moved it
Open the audio in Audacity
Select the audio between the points you noted down earlier on and stretch it by the amount you moved the audio by to make it sync
Save the audio as a wav again
Import it back into the video editor and it should all align on playback, the pitch will change but infinitesimally
Convert the audio back to aac and remux with the original video file

There are many ways to skin a cat ,but that's how I did it (back in the mid 2000's anyway)