ffmpeg synchronize audio and video when offset is known.

[BartZLederman]

I have searched and found many similar questions, but not this one. I've experimented and can't come up with an answer.

I have a Standalone "Digital TV Converter Box". This receives broadcast TV, and can act as a DVR. Mine is a "viewtv", but it's available from dozens of different vendors. It records combined video and audio (and optionally closed captions / subtitles) into a BDAV format file. I want to watch the program on other platforms, and sometimes edit the file and separate programs, and the BDAV format is inconvenient (to say the least). ffmpeg can easily perform the first part of converting the file into something more useable:

ffmpeg -hide_banner -err_detect ignore_err -loglevel verbose -stats -i "channel09202022-0858.mts" -map 0 -c copy "channel-09202022-0858.m2ts"

Here is where my frustration starts. There is usually a delay between the video and audio file, and it's different every time. MediaInfo reports something this:

Audio ID : 4352 (0x1100)
Menu ID : 1 (0x1)
Format : AC-3
Format/Info : Audio Coding 3
Commercial name : Dolby Digital
Codec ID : 129
Duration : 3 h 7 min
Bit rate mode :
Constant Bit rate : 192 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 31.250 FPS (1536 SPF)
Compression mode : Lossy
Delay relative to video : -1 s 501 ms
Stream size : 258 MiB (9%)
Language : English
Service kind : Complete Main


This delay can confuse some editing programs. If I cut segments out with ffmpeg the audio and video don't quite start in the same place. I would really like to have ffmpeg re-synchronize the files, but have not been able to do it. I've tried -vsync with various options (1, 2, vfr) -async with various options including -async 1 which the documentation seems to indicate will synchronize the beginning, -fflags +genpts+igndts in various combinations, -af aresample=async=1 , in every combination I can think of. In every single case, ffmpeg reads and writes with no reported errors: but the new file still has a "Delay relative to video" carried through to the new file. I've spend quite a lot of time on this, and all I have to show for it is severe aggravation.

The only thing I've found so far that comes close is to convert the file once, get the value of the delay, and then do this:

ffmpeg -err_detect ignore_err -hide_banner -loglevel info -stats -async 1 -itsoffset -1.501 -i "Story-09202022-0858.m2ts" ^ -strict normal -map 0 -c:v copy -c:s mov_text -c:a aac -b:a 192k ^ -ar 48000 Story-09202022-0858.mkv

This usually (but not always) re-encodes the audio in a way that does not show any significant delay between video and audio. However, it requires manually entering the values every time, and it must be exactly as I've shown it. Since -async is going to be "deprecated" (for no good reason I can see), I tried the so-called replacement of -af aresample=async=1 in this example and it did not work.

Can someone please tell me if there is a way to get ffmpeg to look at the information in a file that says what the delay is and then "fix" it? It seems very odd to me that there isn't a way to fix this synchronization between video and audio when the value of the off is is already known and available within the file.

If not, is there at least a way to obtain the value of the offset so I can write a command procedure (I normally use Windows for this, and have everything in BAT files) that will automate the procedure?

As I've said, I've searched through many forums, and all of the examples I've found have to do with audio and video being different tempos, or different durations, or similar problems, none of which apply here. I've never seen an example of fixing the initial offset other than with -itsoffset.

If it matters, the ultimate output will be AVC (MP4) video and AAC audio. Unfortunately, just copying the audio in AC3 isn't going to work as there are some systems (such as DLNA servers and Roku Media Player) that can't handle it properly.

Thanks.

[october262]

I have searched and found many similar questions, but not this one. I've experimented and can't come up with an answer.

I have a Standalone "Digital TV Converter Box". This receives broadcast TV, and can act as a DVR. Mine is a "viewtv", but it's available from dozens of different vendors. It records combined video and audio (and optionally closed captions / subtitles) into a BDAV format file. I want to watch the program on other platforms, and sometimes edit the file and separate programs, and the BDAV format is inconvenient (to say the least). ffmpeg can easily perform the first part of converting the file into something more useable:

ffmpeg -hide_banner -err_detect ignore_err -loglevel verbose -stats -i "channel09202022-0858.mts" -map 0 -c copy "channel-09202022-0858.m2ts"

Here is where my frustration starts. There is usually a delay between the video and audio file, and it's different every time. MediaInfo reports something this:

Audio ID : 4352 (0x1100)
Menu ID : 1 (0x1)
Format : AC-3
Format/Info : Audio Coding 3
Commercial name : Dolby Digital
Codec ID : 129
Duration : 3 h 7 min
Bit rate mode :
Constant Bit rate : 192 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 31.250 FPS (1536 SPF)
Compression mode : Lossy
Delay relative to video : -1 s 501 ms
Stream size : 258 MiB (9%)
Language : English
Service kind : Complete Main


This delay can confuse some editing programs. If I cut segments out with ffmpeg the audio and video don't quite start in the same place. I would really like to have ffmpeg re-synchronize the files, but have not been able to do it. I've tried -vsync with various options (1, 2, vfr) -async with various options including -async 1 which the documentation seems to indicate will synchronize the beginning, -fflags +genpts+igndts in various combinations, -af aresample=async=1 , in every combination I can think of. In every single case, ffmpeg reads and writes with no reported errors: but the new file still has a "Delay relative to video" carried through to the new file. I've spend quite a lot of time on this, and all I have to show for it is severe aggravation.

The only thing I've found so far that comes close is to convert the file once, get the value of the delay, and then do this:

ffmpeg -err_detect ignore_err -hide_banner -loglevel info -stats -async 1 -itsoffset -1.501 -i "Story-09202022-0858.m2ts" ^ -strict normal -map 0 -c:v copy -c mov_text -c:a aac -b:a 192k ^ -ar 48000 Story-09202022-0858.mkv

This usually (but not always) re-encodes the audio in a way that does not show any significant delay between video and audio. However, it requires manually entering the values every time, and it must be exactly as I've shown it. Since -async is going to be "deprecated" (for no good reason I can see), I tried the so-called replacement of -af aresample=async=1 in this example and it did not work.

Can someone please tell me if there is a way to get ffmpeg to look at the information in a file that says what the delay is and then "fix" it? It seems very odd to me that there isn't a way to fix this synchronization between video and audio when the value of the off is is already known and available within the file.

If not, is there at least a way to obtain the value of the offset so I can write a command procedure (I normally use Windows for this, and have everything in BAT files) that will automate the procedure?

As I've said, I've searched through many forums, and all of the examples I've found have to do with audio and video being different tempos, or different durations, or similar problems, none of which apply here. I've never seen an example of fixing the initial offset other than with -itsoffset.

If it matters, the ultimate output will be AVC (MP4) video and AAC audio. Unfortunately, just copying the audio in AC3 isn't going to work as there are some systems (such as DLNA servers and Roku Media Player) that can't handle it properly.

Thanks.

try with $ ffmpeg -i file.mkv -itsoffset 3 -i file.mkv -c:a copy -c:v copy -map 0:v:0 -map 1:a:0 out.mkv
more info here - https://www.geekyhacker.com/2020/05/17/synchronize-audio-and-video-with-ffmpeg/

[BartZLederman]

Thank you for your response. However, it's basically what I've already done.

That article gives an example of a known 3 second delay between video and audio, which is what I've done in the example I showed that (mostly) worked.

I'm hoping for one that will pick up the delay from the file instead of my having to manually supply one.

Bart.

[october262]

Thank you for your response. However, it's basically what I've already done.

That article gives an example of a known 3 second delay between video and audio, which is what I've done in the example I showed that (mostly) worked.

I'm hoping for one that will pick up the delay from the file instead of my having to manually supply one.

Bart.

i did find this "delay relative to video" - https://www.reddit.com/r/ffmpeg/comments/q9iz69/delay_relative_to_video_what_is_it/

[Luke M]

To get the delay:

ffprobe -v 0 -show_entries stream=start_time -select_streams a:0 -of compact=p=0:nk=1 "video.mkv"

[ProWo]

For your example in post #1
Encode audio to aac:

ffmpeg -itsoffset -1.501 -i "Story-09202022-0858.m2ts" -map 0 -c:v copy -c:s mov_text -c:a aac -b:a 192k -ar 48000 Story-09202022-0858.mkv

keep ac3 audio:

ffmpeg -itsoffset -1.501 -i "Story-09202022-0858.m2ts" -map 0 -c:v copy -c:s mov_text -c:a copy Story-09202022-0858.mkv

[BartZLederman]

Either I didn't explain properly, or you missed the point.

I know how to get the two streams to synchronize by manually entering a value for itsoffset. I posted an example that (usually) works.

What I'm looking for is a way to synchronize the two streams WITHOUT using itsoffset. I want a command line that will make ffmpeg synchronize the two streams automatically by picking up the information that is already in the input file.

As a cross check, could someone tell me if there is something wrong with my explanation that is leading people to give me a "solution" that is identical to what I've already said I have done?

[ProWo]

As a cross check, could someone tell me if there is something wrong with my explanation that is leading people to give me a "solution" that is identical to what I've already said I have done?

It's not identical, but forget my post.

[october262]

Either I didn't explain properly, or you missed the point.

I know how to get the two streams to synchronize by manually entering a value for itsoffset. I posted an example that (usually) works.

What I'm looking for is a way to synchronize the two streams WITHOUT using itsoffset. I want a command line that will make ffmpeg synchronize the two streams automatically by picking up the information that is already in the input file.

As a cross check, could someone tell me if there is something wrong with my explanation that is leading people to give me a "solution" that is identical to what I've already said I have done?

see if this will work - https://www.anycodings.com/1questions/2312573/ffmpeg-automatically-synchronize-audio-and-video

[BartZLederman]

In a way, I've sort of given up on my original goal. However, I have found a reasonable work-around. I use a Windows batch file to process multiple files at a time.
I use a two step process to change the way the various streams are stored in the container file, which has the side effect of fixing (more-or-less) the timing issue.

FOR /F "tokens=*" %%G IN ('dir /b *.mts') DO ffmpeg -hide_banner -err_detect ignore_err -loglevel info -stats -benchmark ^
 -i "%%G" -map 0 -c copy "%%~nG.m2ts"

FOR /F "tokens=*" %%G IN ('dir /b *.mts') DO ffmpeg -hide_banner -err_detect ignore_err -loglevel info -stats -benchmark ^
 -avoid_negative_ts auto -i "%%G" -i "%%G" -strict very -map 0:v -map 1:a -map 0:s:? -c:s copy -c:v copy -c:a copy "%%~nG-Step1.m2ts"

You are probably wondering why the two-step process. The MTS input file has one BDAV stream, which mixes everything together. ffmpeg, and some other video processors have problems interpreting this: Avidemux, for example, can't read it. So the first step is to do a copy to an M2TS container, which separates the streams into individual streams.

The second step then does a copy again. It reads the input file twice: the first contains the data, and reading it the second time is used to pick up the timing information from the audio stream. This appears to put things back together in synchronization, because after this I can use ffmpeg or Avidemux or anything else I've tried and the video and audio appear to be in sync.

Since the two processes are "just" copying files and not re-encoding them, they run through very fast. So far it has addressed the problem for me.

[JN-]

Hi BartZlederman, I made up a small ffmpeg batch script util “AV Sync, a while back, it relies on you having some idea of the offset.
In an ideal world doing it automatically would be great, but in your thread header you state “when offset is known”.

I’m sure its not doing anything that you haven’t already tried.

If the video is longish and variable frame rate then its best to first convert to cfr.