Matching audio streams from two movies

[nji9nji9]

From time to time it happens that I have a movie in two different versions;
with different languages each.
I want to have the two audio streams in one container ofthe video.

The simple case is to exctract one of the audio streams (losslessly)
and add it to the second container.
I take AviDemux for that.

But often it happens that the audio streams don't match by time.
In that case I extract both audio streams,
then take an audio editor to match their start,
and after that eventually stretch/squeeze in time the to-be-inserted.
Then save in wav-format, to load it into the video container.
That works fine for most cases.

But sometimes... it doesn't:
Although having done a "perfect match" near start and near end,
at other times there is desync to the movie.
(Both movies have constant framerate mode).

Where am I wrong? How can I achieve sync?

[jagabo]

Movies released in different countries often have different cuts. For example, in the USA sex scenes are often edited out. In the EU extreme violence is often removed. So you can't just sync at the beginning and end, you need to edit the audio to account for those cuts.

[nji9nji9]

There are no cuts within; the movies are literally identical.

The desync is less than 1s, unregulary.

[davexnet]

scrub through the movie and find out where the sync is lost. Often it's at scene changes.

You can bring up both videos on the screen at the same time using an Avisynth script
and look for a place wher they go out of sync

[nji9nji9]

Hmm... I can't tell actually.

Perhaps some details may be useful?
Actually it's a 90mins movie.
The two versions differ in lenght (abt. 0.065% = abt. 4s).
Having extracted both audio streams, I first change the start of the to-be-added stream
that it matches the "master" audio stream (-0.58s in my case).
Next I choose a "prominent" comparing point in both streams near the end.
(I take spectrumgram view for that; and can determine to precision of 0.01 s).
For that I derive the delta of 4s from above.
At that point I would expect that both audios slightly go out-of sync.
(If you draw a diagram of time (x) to delta (y), it would show slightly and regulary going up.)
Your scenario (cuts in the movie etc.) would have "jumps" in the diagram, right?
I wouldn't have expected that... hmm...
But what if both effects occur: Slightly AND jumps?
Thinking aloud... it might be best to have an automatic audio wave shift detecting tool,
that determines the graph from above, so I can (or the program) match the audios...
Does something like that exist?
Probably not...
So... doing it manually... choosing not only 1 but several time points and see what
exactly is happening, and when?
Oh man!

[ProWo]

Load both audiostreams in an audio editor (audacity) and compare the peaks.

[nji9nji9]

OK, I did that manually (not very funny) for some prominent audio events (precision abt. 0.01s);
y = delta between audios; see attachment.

That's a strange progression...
up to 50min a steep constant incline, then less slope for some time, then no incline.

Each uniform incline could be repaired be squeezing.
(Delta > 0.1s would be noticeable).

But can all this be right?

[ListyDoM]

Is the frame rate of both videos (not audio!) the same? It seems that one has a frame rate of 23.976 and the other has a frame rate of 24. At 90 minutes, there is a discrepancy of 4-5 seconds. If they are different, change the audio using the clever FFmpeg GUI https://forum.videohelp.com/threads/396490-clever-FFmpeg-GUI-small-FFmpeg-UI-fully-por...llationб and then align them in an audio editor by peaks.

[nji9nji9]

Is the frame rate of both videos (not audio!) the same? [...]

Both videos are 25fps cfr

[ListyDoM]

Is the frame rate of both videos (not audio!) the same? [...]

Both videos are 25fps cfr

Hmm, are these original videos? Can you provide those two audio files here?

[ListyDoM]

Original, unmodified. Extract using inviska or gMKVExtract

[nji9nji9]

The audios are 120 + 80 MB.

But maybe MediaInfo gives some information:

General
Complete name               : D:\A.mp4
Format                      : MPEG-4
Format profile              : Base Media
Codec ID                    : isom (isom/iso2/avc1/mp41)
File size                   : 1.31 GiB
Duration                    : 1 h 28 min
Overall bit rate mode       : Constant
Overall bit rate            : 2 134 kb/s
Writing application         : Lavf58.29.100

Video
ID                          : 1
Format                      : AVC
Format/Info                 : Advanced Video Codec
Format profile              : Main@L3.1
Format settings             : CABAC / 3 Ref Frames
Format settings, CABAC      : Yes
Format settings, Reference  : 3 frames
Format settings, GOP        : M=3, N=50
Codec ID                    : avc1
Codec ID/Info               : Advanced Video Coding
Duration                    : 1 h 28 min
Bit rate mode               : Constant
Bit rate                    : 2 000 kb/s
Width                       : 960 pixels
Height                      : 544 pixels
Display aspect ratio        : 16:9
Frame rate mode             : Constant
Frame rate                  : 25.000 FPS
Color space                 : YUV
Chroma subsampling          : 4:2:0
Bit depth                   : 8 bits
Scan type                   : Progressive
Bits/(Pixel*Frame)          : 0.153
Stream size                 : 1.23 GiB (94%)
Codec configuration box     : avcC

Audio
ID                          : 2
Format                      : AAC LC
Format/Info                 : Advanced Audio Codec Low Complexity
Codec ID                    : mp4a-40-2
Duration                    : 1 h 28 min
Bit rate mode               : Constant
Bit rate                    : 128 kb/s
Channel(s)                  : 2 channels
Channel layout              : L R
Sampling rate               : 48.0 kHz
Frame rate                  : 46.875 FPS (1024 SPF)
Compression mode            : Lossy
Stream size                 : 80.7 MiB (6%)
Default                     : Yes
Alternate group             : 1

and the second:

General
Complete name               : D:\B.mkv
Format                      : Matroska
Format version              : Version 2
File size                   : 3.53 GiB
Duration                    : 1 h 28 min
Overall bit rate mode       : Variable
Overall bit rate            : 5 719 kb/s
Encoded date                : UTC 2019-12-17 19:49:08
Writing application         : MakeMKV v1.14.4 win(x64-release)
Writing library             : libmakemkv v1.14.4 (1.3.5/1.4.7) win(x64-release)

Video
ID                          : 1
ID in the original source m : 224 (0xE0)
Format                      : MPEG Video
Format version              : Version 2
Format profile              : Main@Main
Format settings             : CustomMatrix / BVOP
Format settings, BVOP       : Yes
Format settings, Matrix     : Custom
Format settings, GOP        : M=3, N=15
Format settings, picture st : Frame
Codec ID                    : V_MPEG2
Codec ID/Info               : MPEG 1 or 2 Video
Duration                    : 1 h 28 min
Bit rate mode               : Variable
Bit rate                    : 5 522 kb/s
Maximum bit rate            : 9 800 kb/s
Width                       : 720 pixels
Height                      : 576 pixels
Display aspect ratio        : 4:3
Frame rate mode             : Constant
Frame rate                  : 25.000 FPS
Standard                    : PAL
Color space                 : YUV
Chroma subsampling          : 4:2:0
Bit depth                   : 8 bits
Scan type                   : Interlaced
Scan order                  : Top Field First
Compression mode            : Lossy
Bits/(Pixel*Frame)          : 0.533
Time code of first frame    : 10:00:00:00
Time code source            : Group of pictures header
GOP, Open/Closed            : Closed
Stream size                 : 3.41 GiB (97%)
Language                    : English
Default                     : No
Forced                      : No
Original source medium      : DVD-Video

Audio
ID                          : 2
ID in the original source m : 189 (0xBD)128 (0x80)
Format                      : AC-3
Format/Info                 : Audio Coding 3
Commercial name             : Dolby Digital
Codec ID                    : A_AC3
Duration                    : 1 h 28 min
Bit rate mode               : Constant
Bit rate                    : 192 kb/s
Channel(s)                  : 2 channels
Channel layout              : L R
Sampling rate               : 48.0 kHz
Frame rate                  : 31.250 FPS (1536 SPF)
Compression mode            : Lossy
Stream size                 : 121 MiB (3%)
Title                       : Stereo
Language                    : German
Service kind                : Complete Main
Default                     : Yes
Forced                      : No
Original source medium      : DVD-Video

[nji9nji9]

I did more refinements on the delta graph (see attachment).

Then did 2 corrections:
- linear squeezing of the first 46 min
- cutting 1s at the end of that sequence.

When using that audio stream for the movie I encounter the expected effects:

Linear squeezing leaves up to 1s desync (at abt. 24 min).
Not-repaired sync at the second part (> 46 min) leaves noticeable desync (abt. 0.2 s) there.

All this seem to point that there must be something wrong in the idea...
"Non-linear desync"... what does it come from?
Maybe there is a connection with the containers/ video codecs?

[krykmoon]

and the second:

General
Overall bit rate mode       : Variable

Video
ID                          : 1
ID in the original source m : 224 (0xE0)
Format                      : MPEG Video
Format version              : Version 2
Format profile              : Main@Main
Format settings             : CustomMatrix / BVOP
Format settings, BVOP       : Yes
Format settings, Matrix     : Custom
Format settings, GOP        : M=3, N=15
Format settings, picture st : Frame
Codec ID                    : V_MPEG2
Codec ID/Info               : MPEG 1 or 2 Video
Duration                    : 1 h 28 min
Bit rate mode               : Variable

You said both videos were CBR.. but while the first video is CBR indeed, the second one seems VBR.

[nji9nji9]

Sorry, typo by me...

It was framefrate (CFR) that was asked for, not bitrate (CBR).

[jagabo]

Maybe "gaps" in AAC? https://forum.videohelp.com/threads/418934-The-demuxed-AAC-from-a-MP4-file-is-wrong

[ProWo]

I agree with jagabo.
Most likely, the MP4 file has problems with the timestamps.
I recommend converting the AAC file to AC3, but not from the extracted AAC file, but from the MP4 file.

[nji9nji9]

Finally... I managed to create a (nearly) satisfying result (max. desync < 250 ms).

To be sure I refined the difference graph some more (see attachment).
Then I removed the 1s gap (step in difference graph) @ abt. 46 min.
Next I squeeze linearly from beginning to point 5 (steep incline).
That squeezing removes a difference of 1.5s (@ point 5) for the rest of the audio.
This 1.5s offset is just about the average of audio > point 5, i.e. done!

I wrote "nearly" satisfying.
That's not only because of the unperfect desync max. of 250 ms.
(That should be less than 100 ms at least).

But most of all I would like to know what's the cause of all this.
(Actually it's not the first time I came to this problem...).

I can well image variable frame rate of a video might cause it (???);
but both are cfr in this case.
Moreover I had the idea that maybe the association of video/ audio
in the container (time points etc.) might cause it (???).
Knowing the cause it might be possible to restore audio perfectly and automatically?

Doing a manual workaround is very cumbersome:
First identify matching points in the two audios.
Then having a look at the graph (I provided a libreoffice calc for that (see attachment)).
If (!) the graph may be approximated by few piecewise straight lines (or steps),
the desync may be lessened by using the lines for squeezing/ stretching.
If in some cases there is really no better way than approximating,
it would be nice to have a tool that builds the graph automatically from the two audios.
Maybe even do the correction...

[nji9nji9]

I agree with jagabo.
Most likely, the MP4 file has problems with the timestamps.
I recommend converting the AAC file to AC3, but not from the extracted AAC file, but from the MP4 file.

The problem file is the second one (MKV, see above), audio = AC3.

The audio in the first movie (MP4 with aac) is perfectly in sync with the video.
Or do you mean that the problem is by extracting the aac as-is with ADM?

[ListyDoM]

I agree with jagabo.
Most likely, the MP4 file has problems with the timestamps.
I recommend converting the AAC file to AC3, but not from the extracted AAC file, but from the MP4 file.

not recommend working in an audio editor with any lossy format, only lossless formats such as wav or flac.

[davexnet]

You could just upload the video and the audio you're trying to sync and let somebody take a look

[nji9nji9]

Master movie at
https://archive.org/download/Der-Richter-und-sein-Henker_Krimi-von-Friedrich-Duerrenma...-TRAC_1978.mp4
Audio to sync:
see attachment.

I'm excited what is the cause and how to repair it optimal.

[ListyDoM]

It is preferable to have a second file, video rather than audio, as it is unknown whether an error occurred during extraction. However, at first glance, these are simply different versions of the video; for example, there are only three moments at the beginning where there is a shift. I suspect that there are many more further on. In this case, no automatic methods will help, only manual synchronization. I do this in VEGAS.

[nji9nji9]

Oh... I wouldn't have expected that!

The movie from archive.org is of different (better) scan, as the movie with the to-be-added audio (which is from DVD).

So the movies are of different cuts, with cuts of parts of seconds?
Why on earth did they do that?

Please nobody do that cumbersome task to syncronize the hundreds (thousands) of desyncs!
(It took me some time just to identify the 15 points.)

But I remember having an archaic version of Adobe's Audition (3.0).
Maybe the multitrack workflow can do some automatic syncing of audio tracks...

[ProWo]

Here is your german audiostream, ready to be muxed:
https://limewire.com/?referrer=pq7i8xx7p2

[nji9nji9]

My conclusion (hopeful for anyone's benefit):

If the desync of two audios is very irregular (as in this case),
desyncing precisely is very cumbersome "by hand".

(Maybe it's possible with propriety and costy Adobe Audition or DaVinci Resolve).

If the desync-pattern consist of nearly linear intervals and steps
(see my graphics and the libreoffice calc file in the zip above),
it might be acceptable to linearize (squeeze/ stretch) the intervals
and to remove the steps.
In my case I was kind of successful:

[Attachment 89183 - Click to enlarge]
2nd audio's start was matched before.

[nji9nji9]

Here is your german audiostream, ready to be muxed:
https://limewire.com/?referrer=pq7i8xx7p2

Can't access (membership required?)

[ProWo]

Here is your german audiostream, ready to be muxed:
https://limewire.com/?referrer=pq7i8xx7p2

Can't access (membership required?)

Try this one:
https://limewire.com/d/sDhGT#JySGEnSOeU

[nji9nji9]

I muxed it, and checked the result... it seems to be pretty good

Will you please tell your way to achieve it?
Is it more precise, or less cumbersome, ...
That would be most valuable to know.

[ProWo]

I muxed it, and checked the result... it seems to be pretty good

Will you please tell your way to achieve it?
Is it more precise, or less cumbersome, ...
That would be most valuable to know.

This case was very complex.
I used two tools:
1) - clever FFmpeg-GUI
2) - MPEG Video Wizard DVD 5.0

The first step is to insert the mp4 video into 2). Then right-click on the inserted video, audio, audio peaks. The program remains open.

Then I used 1) to convert the ac3 file losslessly into a pcm (wav) file, adjusting the length, and inserted the new wav file into 2).
Right-click on this inserted wav, audio, audio peaks.

Compare the peaks at the beginning and end and, if necessary, create a new modified wav file.

Repeat this procedure until the peaks at the beginning and end match.

The rest is fine-tuning in 2).


[Attachment 89199 - Click to enlarge]

[Attachment 89200 - Click to enlarge]