Audio shifts on rendering - do I need to compensate?

[Bencuri]

This is something I noticed in Sony Vegas: you are ready with the project, do rendering to mp4 format. What I actually do is render the video by Debugmode Frameserver. Load the rendered video in Vegas, the old and the rendered line up perfectly. Every frame matches. Then I render the audio in aac, mux the rendered video and audio. Loading back to Vegas, the rendered audio is shifted compared to the old. If I compensate this shifting the audio before rendering, the new and old lines up perfectly.

Do I need to do this shifting before rendering, or the misalignment is only a glitch in Vegas, and in reality, the audio will play on time even if it appears shifted after rendering? Is there any way to objectively examine the position of the audio in the rendered file after muxing?

[hello_hello]

How much of a shift, and how are you muxing the video and audio?

Lossy encoders add padding (silence) to the beginning and end of the audio, and if I had to guess, that would probably be the cause.
https://wiki.hydrogenaudio.org/index.php?title=Gapless_playback

The only AAC encoder I know of that has an option to remove the padding is QAAC. Failing that, MKVToolnix compensates for AAC encoder padding when muxing, assuming the audio contains delay metadata, by applying an appropriate negative delay to compensate.

I rarely work with MP4s, but many I've seen have a positive video delay, I assume to compensate for the audio delay caused by the encoder padding, but there's at least two places where any padding delay might not be accounted for. When muxing the encoded audio and video, or when loading the encoded audio and video back into an editor.

As I said though, all of the above is just a guess as I've never used Vegas.

[Bencuri]

In this recent case it is about 3-4ms. Other times I noticed up to half a frame (in case of 25p). It depend on the format of the source as I noticed. This was originally mp4, so I guess that's why its not half a frame. And it is an extra delay. So the rendered is late compared to the original.

So do I understand correctly: when opened in the editor, the padding are overlooked and the sound is placed on the timeline as if there is no padding? What about players? I guess they take it into consideration, don't they?

I have just checked the rendered in Mediainfo, there is no delay reported there.

[rgr]

This is something I noticed in Sony Vegas: you are ready with the project, do rendering to mp4 format. What I actually do is render the video by Debugmode Frameserver. Load the rendered video in Vegas, the old and the rendered line up perfectly. Every frame matches. Then I render the audio in aac, mux the rendered video and audio. Loading back to Vegas, the rendered audio is shifted compared to the old. If I compensate this shifting the audio before rendering, the new and old lines up perfectly.

Do I need to do this shifting before rendering, or the misalignment is only a glitch in Vegas, and in reality, the audio will play on time even if it appears shifted after rendering? Is there any way to objectively examine the position of the audio in the rendered file after muxing?

Shifting forward 3 frames and silence in the last 3 frames? (The amount of shift depends on the fps).
1. Render the video (can be without audio).
2. Export the audio separately to a WAV file (PCM 48kHz, 32bit).
3. ffmpeg -i video.mp4 -i audio.wav -map 0:v -map 1:a -c:v copy -c:a aac -b:a 192k output.mp4

There's no other way, trust me.
Same procedure if you use the older Voukoder (free).
You can also export using Magix's built-in codecs; there shouldn't be any shift there.

Edit: And enable "Legacy H264" in Vegas options. Basically, start with this and see if you still have the offset. And it's best to check these offsets in VirtualDub.

Which version of Vegas exactly?

[Bencuri]

Vegas 12.0

[rgr]

Quite old. But apparently this bug existed back then. It disappeared with some version 20.xxx.

[hello_hello]

In this recent case it is about 3-4ms. Other times I noticed up to half a frame (in case of 25p). It depend on the format of the source as I noticed. This was originally mp4, so I guess that's why its not half a frame. And it is an extra delay. So the rendered is late compared to the original.

So do I understand correctly: when opened in the editor, the padding are overlooked and the sound is placed on the timeline as if there is no padding? What about players? I guess they take it into consideration, don't they?

I have just checked the rendered in Mediainfo, there is no delay reported there.

I generally use Avisynth+ for video, and if I convert the audio I generally convert to AAC using foobar2000 and the QAAC encoder, then mux with MKVToolnix. Regardless of the AAC encoder though, the audio itself is encoded in "frames" and MediaInfo should tell you the audio frame rate. The random file I just opened containing AAC showed the following:

Frame Rate: 31.250 FPS (1536 SPF)

So each audio frame contains 1536 samples and the frames go by at 31.25 fps (1536 x 31.25 = 48k audio).

If the encoder writes the amount of padding and/or any encoder delay to the m4a file it's usually saved in an iTunSMPB tag (an Apple thing).
I found a MediaInfo ticket that describes the tag here but I couldn't get MediaInfo to display it, other than using it's debug mode to drill down and show the hex values.
https://sourceforge.net/p/mediainfo/feature-requests/398/
Most audio players would pay attention to that data as it's required for gapless playback, so the encoded audio effectively has the same number of samples as the original, once any extra padding at the beginning and end is skipped.

When you mux AAC in an mp4/m4a container with MKVToolNix it reads the iTunSMPB info (if it exists) and drops an appropriate number of audio frames from the beginning. Just making up an example here, if the delay happened to be 2 and a half audio frames worth of samples, it'd drop the first three, which is a little too much, so when muxing it'd write half a frame's worth of audio delay to the MKV container to compensate. Or something like that. I remember when I used the Nero AAC encoder, an MKV after muxing the video and audio would usually have an audio delay of 9ms, which MediaInfo does display.

So I can tell you a little bit regarding how it works, but by the time you've read this you'll know as much about it as I do. I can't tell you anything else in respect to what other programs may or may not do, just that if there is some sort of small audio shift, that's probably where it originates.

[Bencuri]

I will try it with the ffmpeg command line.