How to find out which audio sample belongs to which frame in a video

[tim5]

Hello,

I am wondering how I would find out which frame in a video belongs to which audio sample. I was thinking about using ffmpeg. I want to use the information contained in the container to find the solution. In the end I would like to automate the process with C++ but for the beginning I am happy with ffmpeg used by command line.

Lets say: I know audio sample 39822 has the characteristic I am looking for. Using the information contained in the container, I want to find out to which frame the audio sample was recorded. (Key frames/ Samples?)

I was thinking ffmpeg could help me with this problem.

I tried to find the answer within the ffmpeg documentation but did not succeed.

Maybe you guys can give me a hint which tool (maybe even other than ffmpeg) to use or where to look specific.

Thanks in advance Tim

[Cornucopia]

they are totally independent. you match them by the common time.


scott

[jagabo]

Cornucopia is right. There generally is not a 1 to 1 correlation between video frames and audio chunks. If you use ffprobe to examine the video and audio chunks you'll see that. You have to work by time.

[Cornucopia]

Example:
audio sample 328165
@ 48kHz samplerate, this is 328165 / 48000 = 6.8367sec

video @ 25FPS (since you're in Deutschland and the math is easier)
25 * 6.8367 = Frame 170.91927, but since there is no such thing as a partial frame, it's just frame 170. Almost 171.

Scott

[tim5]

they are totally independent. you match them by the common time.


scott

but there must be some information contained in the container which audio sample belongs to which frame?! There should be a keysample which is in relation with one frame, isnt it?

[tim5]

Example:
audio sample 328165
@ 48kHz samplerate, this is 328165 / 48000 = 6.8367sec

video @ 25FPS (since you're in Deutschland and the math is easier)
25 * 6.8367 = Frame 170.91927, but since there is no such thing as a partial frame, it's just frame 170. Almost 171.

Scott

I already worked with these equations. But I would like to use the information contained in the container and not the fps and audiosamplingrate.

[jagabo]

You can use the timestamps for the video and audio chunks. But once again, there isn't a1:1 relationship between video chunks and audio chunks.

[tim5]

You can use the timestamps for the video and audio chunks. But once again, there isn't a1:1 relationship between video chunks and audio chunks.

thanks

[emcodem]

there isn't a1:1 relationship between video chunks and audio chunks.

At the best of my knowledge this depends on how the container wraps and indexes the video and audio chunks, e.g. MXF Frame wrapped means to have a guaranteed relation between audio and video chunks in the existing Index. Despite it is about chunks, which can also be called frames - instead of "Samples". (where again each frame/edit unit of course consists of a defined number of samples because we deal with fixed frame rate).
So in my head, it depends on the container if such a releation is given or needs to be recreated by the decoder (e.g. re-indexing ).


The question itself about Samples is a little confusing here because it implies that this is about uncompressed audio. Is this given?

Might be better @tim5 if you add some words about the container you are working with?