MKVmerge audio stretch

[WeirdOzzie]

I`ve been having a major problem with MKVmerge recently. I`m making custom mixes with a 7 channel audio track on an existing video file, as in extracting the audio from the video, muting some of the channels, re-exporting the audio, and then muxing it into the original video as an additional track.

The trouble is, the newly added track is ever so slightly stretched. It starts out perfectly in sync, but by the end of the film it`s off by about 10 seconds. However, the stretching only occurs at playback, because when I re-extract the new track from the video and compare it to the original in the audio editor, they match up perfectly.

Furthermore, this has only been an issue with one particular film. I ran thorough tests on other files in an attempt to replicate the issue, but it only seems to affect the one. I`ve tried exporting the audio as several different file types, as well as extracting the video stream and remuxing it manually to no effect.

Why might this be happening? Has this been a problem for anyone else?

[manono]

You say both audio tracks - the original and the edited one - match each other. Do they also both match the length of the video? What happens when you mux back the original track after extracting it? Is it in synch all the way?

[sneaker]

If you look at the audio timecodes you will probably find they're not constant or constant with a slight difference to the audio frame length. When you do extract the original audio and mux it back in you should find it will be async as well (I guess that's what manono is expecting).

Try:
ffmpeg -i "input.mkv" -af aresample=async=1 "output.wav"
This will fill any audio gaps with silence.

If the problem aren't gaps but some kind of stretching it might be better to use a different aresample option, though. Or: adjust video fps instead.

[WeirdOzzie]

You say both audio tracks - the original and the edited one - match each other. Do they also both match the length of the video? What happens when you mux back the original track after extracting it? Is it in synch all the way?

If I extract the audio tracks, they match the video length perfectly. However, if I use MKVmerge to mux it into the video, it will slowly fall out of sync. If I re-extract the muxed track, the audio file looks exactly the same as it went in, which suggests that the problem exists in the muxing process.

If I extract and remux the original, the same problem occurs, furthering my hypothesis.

If you look at the audio timecodes you will probably find they're not constant or constant with a slight difference to the audio frame length. When you do extract the original audio and mux it back in you should find it will be async as well (I guess that's what manono is expecting).

Try:
ffmpeg -i "input.mkv" -af aresample=async=1 "output.wav"
This will fill any audio gaps with silence.

If the problem aren't gaps but some kind of stretching it might be better to use a different aresample option, though. Or: adjust video fps instead.

I`m not sure what you mean by "audio timecodes", since I can only view those in the audio editor, where everything acts normally. I doubt that what you`re suggesting will have any effect, since the audio file itself is not being distorted, and seems instead to be played back incorrectly when muxed into the video. Sadly, adjusting the video fps is not an option, because then the OTHER 2 audio tracks would fall out of sync.

[sneaker]

I`m not sure what you mean by "audio timecodes", since I can only view those in the audio editor, where everything acts normally.

The audio is divided into frames just like video. The Matroska container will assign each of those frames a timecode (explicitly or implicitly). These timing can be different from those you would see in an audio editor after demuxing the audio track since the demuxing will not use or copy the Matroska timecodes in any way. That's also why demuxing->remuxing will have the same desync. After demuxing your audio editor will "create" new timecodes by simply assuming each audio frame comes directly after the previous ones ends and the first frame will start at 0, essentially building a chain. Muxing raw audio using mkvmerge works by the same principle.
With the ffmpeg command I proposed any mismatches between the two methods - for example a gap implied by the timecodes from the Matroska container - will be handled by inserting or dropping audio samples. Synchronization should then be kept.

[sneaker]

Sadly, adjusting the video fps is not an option, because then the OTHER 2 audio tracks would fall out of sync.

Well, at least one of them should have the same defect, right? Did you try it with the 2nd track as well? For me this is a hint the audio tracks might be from 2 different sources, e.g. from a PAL DVD and an NTSC DVD. Measuring the exact difference at the end combined with the duration will likely reveal the fps of the video the async track originally belonged to. Then looking at its timecodes would reveal whether it was indeed stretched using that constant factor or if there are any gaps. If the former was used it might be better to use a higher "async" value for the resample filter so actual resampling is done instead of just inserting/dropping samples. If you feel more confident using mkvmerge you can play with a stretch factor for the audio there (I don't recommend this) or in AviSynth with different values for AssumeSampleRate().

If you upload the timecodes (mkvextract timecodes_v2) of all tracks and a MediaInfo log I could take a look.

[WeirdOzzie]

Well, at least one of them should have the same defect, right? Did you try it with the 2nd track as well? For me this is a hint the audio tracks might be from 2 different sources, e.g. from a PAL DVD and an NTSC DVD. Measuring the exact difference at the end combined with the duration will likely reveal the fps of the video the async track originally belonged to. Then looking at its timecodes would reveal whether it was indeed stretched using that constant factor or if there are any gaps. If the former was used it might be better to use a higher "async" value for the resample filter so actual resampling is done instead of just inserting/dropping samples. If you feel more confident using mkvmerge you can play with a stretch factor for the audio there (I don't recommend this) or in AviSynth with different values for AssumeSampleRate().

If you upload the timecodes (mkvextract timecodes_v2) of all tracks and a MediaInfo log I could take a look.

Upon further review, it seems that only muxing track 2 causes the defect. Extracting and remuxing track 1 does not result in any stretching. What I`m having a hard time understanding now is what data in track 2 could possibly cause it to mux incorrectly regardless of format?

I don`t know anything about the specific origins of the video or audio tracks, but the two tracks, as ORIGINALLY encoded in the file, are perfectly in sync. Only when I attempt to remux track 2 or any of its variants that I create do things go screwy. I think you may be suggesting this, but it seems like the original encoder made an adjustment to account for this problem. Unfortunately, you`re referencing tools and using terminology that I`m unfamiliar with, so I`m not sure what you`re asking for.

I also attempted a ham-fisted solution, by which I append track 2 to track 1, and then split them again at the merge point. I`ve used this before to give video files from different sources the same characteristics (which I later discovered mostly had to do with additional attached files). However, when I attempted to do this, MKVmerge complained that the track parameters are not the same, and therefore they cannot be appended. They`re both 6-channel FLAC files. What parameters might it be referring to?

[hello_hello]

It seems one of the audio tracks isn't continuous. There's probably a gap or two. Usually, it's extracted as though it's continuous, so the gaps aren't accounted for.

The ffmpeg command line sneaker offered will extract the audio while replacing the gaps with silence (I've never used it myself but I believe him). The only other method I know of that does the same is extracting the audio with eac3to, which is also a command line program. Possibly the simplest eac3to GUI for extracting is the HD Streams Extractor (there's also a version built into MeGUI under the Tools menu). It'll open the MKV and replacing any gaps with silence will be automatic. It'll probably default to extracting to the same format (ie AC3 will be extracted as AC3 so nothing is re-encoded) but it can convert while it extracts.

If you can extract the audio with either of those tools and it's still in sync when you remux it again, then the same should apply after you've edited it.

[WeirdOzzie]

It seems one of the audio tracks isn't continuous. There's probably a gap or two. Usually, it's extracted as though it's continuous, so the gaps aren't accounted for.

The ffmpeg command line sneaker offered will extract the audio while replacing the gaps with silence (I've never used it myself but I believe him). The only other method I know of that does the same is extracting the audio with eac3to, which is also a command line program. Possibly the simplest eac3to GUI for extracting is the HD Streams Extractor (there's also a version built into MeGUI under the Tools menu). It'll open the MKV and replacing any gaps with silence will be automatic. It'll probably default to extracting to the same format (ie AC3 will be extracted as AC3 so nothing is re-encoded) but it can convert while it extracts.

If you can extract the audio with either of those tools and it's still in sync when you remux it again, then the same should apply after you've edited it.

That sounds like the opposite of the problem I`m having. If there were gaps in the original that are being ignored when it`s extracted, then I would think that would make the extracted version shorter than the original, which it isn`t, and therefore make the remuxed version shorter, too, which is the reverse of what I`m getting.

I`ll try what you`re suggesting nonetheless, if I can manage to figure out how.

Update: Used the HD Stream Extractor in MeGUI. Same exact problem as before. Everything looks perfect until I attempt to mux it.

[hello_hello]

You could try.... and I'm fairly sure if this has any chance of working you'd have to be remuxing the same type of audio as the original. ie the extracted audio is AC3, it's edited, converted to AC3 again, remuxed.

gMKVExtractGUI can be used to extract the time-codes of individual streams, as well as the streams themselves. MKVCleaver can do the same. It should be fairly obvious how once you've opened an MKV. Extract the timecodes for the stream giving you problems. Add the edited stream to the MKV for muxing with MKVToolNix, select it, and under the timecodes tab add the extracted timecodes. Remux.

It seems somewhat unusual for the extracted stream to increase in duration after it's extracted. It's possible someone used the MKVToolNix timestretch function to make it match the video. You could do that yourself although stretching audio streams is prone to causing playback problems (when I say "stretch" I assume a negative value for stretching would effectively "shrink").

I wonder if rather than extract the audio, given you're editing anyway, opening the MKV with Avisynth and converting the audio to a wave file that way would ensure audio sync doesn't change. Anyone know if there's a clever way to do with Avisynth to keep the video/audio sync the same?

[WeirdOzzie]

You could try.... and I'm fairly sure if this has any chance of working you'd have to be remuxing the same type of audio as the original. ie the extracted audio is AC3, it's edited, converted to AC3 again, remuxed.

gMKVExtractGUI can be used to extract the time-codes of individual streams, as well as the streams themselves. MKVCleaver can do the same. It should be fairly obvious how once you've opened an MKV. Extract the timecodes for the stream giving you problems. Add the edited stream to the MKV for muxing with MKVToolNix, select it, and under the timecodes tab add the extracted timecodes. Remux.

It seems somewhat unusual for the extracted stream to increase in duration after it's extracted. It's possible someone used the MKVToolNix timestretch function to make it match the video. You could do that yourself although stretching audio streams is prone to causing playback problems (when I say "stretch" I assume a negative value for stretching would effectively "shrink").

I wonder if rather than extract the audio, given you're editing anyway, opening the MKV with Avisynth and converting the audio to a wave file that way would ensure audio sync doesn't change. Anyone know if there's a clever way to do with Avisynth to keep the video/audio sync the same?

My suspicions about the muxing being at fault were formally proven wrong, since the extracted audio played along side the original video are not in sync. The files only have the same length because some of the stretched audio file is truncated.

I extracted the audio timecodes from the video, but I can`t quite find where in MKVtoolnix I would re-insert them. There is a section labeled "timecodes", but everything in it is grey and inaccessible (see image). Am I looking in the wrong spot?

[hello_hello]

You'd need to select the audio stream in question in the bottom left pane first, so MKVToolNix knows which stream the timecodes belong to.

[WeirdOzzie]

You'd need to select the audio stream in question in the bottom left pane first, so MKVToolNix knows which stream the timecodes belong to.

Muxed the audio with the timecodes extracted from the original file. Still stretched. Did not otherwise edit the extracted track in any way.

[hello_hello]

That seems fairly impossible. I'm not saying that's not what's happening, just that it seems impossible.

In "try anything" mode now..... have you tried remuxing the extracted audio with the video without including any of the other audio, chapter or subtitle streams? Just to see if there's something else causing the problem.

All I've got left at the moment is to suggest accepting the audio's too long and try time stretching it, or time shrinking it in your case. Don't use the timecodes file and specify a stretch amount when muxing instead. I don't know the length of the video but assuming it's exactly 2 hours and the audio is out by 10 seconds at the end, a time stretch of 71900/72000 would be somewhere to start.
That's not exactly a large amount of stretching so it mightn't cause playback issues (the audio will stutter if you stretch it too much and for some audio types it's more successful than others in that respect).

If doing that with MKVToolNix appears as though it'll solve the problem and you want to stretch the audio itself before muxing, I assume most audio editors will do time shifting. Audacity has a time stretching DSP that lets you specify a percentage (I think up to three decimal places) or if you're an Avisynth user the TimeStretch plugin will do it for you. http://avisynth.nl/index.php/TimeStretch

[WeirdOzzie]

That seems fairly impossible. I'm not saying that's not what's happening, just that it seems impossible.

In "try anything" mode now..... have you tried remuxing the extracted audio with the video without including any of the other audio, chapter or subtitle streams? Just to see if there's something else causing the problem.

All I've got left at the moment is to suggest accepting the audio's too long and try time stretching it, or time shrinking it in your case. Don't use the timecodes file and specify a stretch amount when muxing instead. I don't know the length of the video but assuming it's exactly 2 hours and the audio is out by 10 seconds at the end, a time stretch of 71900/72000 would be somewhere to start.
That's not exactly a large amount of stretching so it mightn't cause playback issues (the audio will stutter if you stretch it too much and for some audio types it's more successful than others in that respect).

If doing that with MKVToolNix appears as though it'll solve the problem and you want to stretch the audio itself before muxing, I assume most audio editors will do time shifting. Audacity has a time stretching DSP that lets you specify a percentage (I think up to three decimal places) or if you're an Avisynth user the TimeStretch plugin will do it for you. http://avisynth.nl/index.php/TimeStretch

Ah, forgive me, I made an error previously while trying to mux the timecodes in with the audio, in that I applied them to the wrong track. Upon applying them to the correct track, everything lined up very nicely. Furthermore, the application of these same timecodes to the edited versions of the audio results in its proper alignment. Thanks to your help, I can confidently say that the issue has been solved.