Delayed/streched mux

[ultradj83]

Dears,
for the same content I have 2 formats:

• HEVC 2160p@24fps / Dolby Digital Plus 7.1
• AVC 1080p@23.976fps / Dolby TrueHD 7.1

I want to create a mux with HEVC 2160p / Dolby TrueHD 7.1 by MKVtoolnix or other tool.

Different frame rate and a delay at start in the HEVC track make it go out of sync.
I need to delay the Dolby TrueHD 7.1 by 2988ms to sync the start with the HEVC track, then I need to find a frame rate that keeps tracks synced.

On MKVtoolnix it looks quite easy to delay the Dolby TrueHD 7.1 track by setting delay field to 2988, but I haven't found a way to keep the sync yet.
Of course I need to work on video as I can't stretch audio.

Any suggestions?

Regards

[Ennio]

Changing framerate of TrueHD needs decoding - shrink or stretch - resampling - recoding. Last step would require Dolby software. You can use FLAC instead, it's free but your playback chain has to support it.

Maybe it's easier to change HEVC video framerate instead? Ffmpeg can do this for you. After, mux with existing TrueHD and apply proper delay.

[ultradj83]

Changing framerate of TrueHD needs decoding - shrink or stretch - resampling - recoding. Last step would require Dolby software. You can use FLAC instead, it's free but your playback chain has to support it.

Maybe it's easier to change HEVC video framerate instead? Ffmpeg can do this for you. After, mux with existing TrueHD and apply proper delay.

Dear,
sure I can't touch Dolby.

I want to stretch video by changing fps.
Maybe using the Stretch field on MKVtoolnix?
But overall, how to calculate the correct value?
And what if it is a not standard value?

Tks!

[Ennio]

Don't go "stretching" in mkvtoolnix. Use ffmpeg to change framerate of HEVC but don't use the mkv container as input. Use the elementary videostream. So demux the HEVC mkv first.
If you need further help please say so.

[ultradj83]

Don't go "stretching" in mkvtoolnix. Use ffmpeg to change framerate of HEVC but don't use the mkv container as input. Use the elementary videostream. So demux the HEVC mkv first.
If you need further help please say so.

Very interesting.
Would it also be possible to make ffmpeg cut a certain number of frames at the beginning of the HEVC track?

Regards

[Ennio]

I don't know. This is something MKVToolNix can do - to some extend. It can do cuts only at keyframe-positions. I think you mean to do this to get rid of (most) delaytime? What you can do is load the HEVC mkv in Virtualdub. It has buttons to jump to next/previous keyframes. Figure out what keyframe suits you best as cutpoint and use that info to set MKVToolNix cut at that point. It wouldn't matter if a suitable keyframe would be after the delay-time; MKVToolNix is clever enough to also cut whatever is needed from the TrueHD track. Note that, if doing so, MKVToolNix likely will add a residual delay for TrueHD, as such audio cannot be cut at every frame but only at certain frames. A bit similar to keyframes in video, if you will.

[ultradj83]

I don't know. This is something MKVToolNix can do - to some extend. It can do cuts only at keyframe-positions. I think you mean to do this to get rid of (most) delaytime? What you can do is load the HEVC mkv in Virtualdub. It has buttons to jump to next/previous keyframes. Figure out what keyframe suits you best as cutpoint and use that info to set MKVToolNix cut at that point. It wouldn't matter if a suitable keyframe would be after the delay-time; MKVToolNix is clever enough to also cut whatever is needed from the TrueHD track. Note that, if doing so, MKVToolNix likely will add a residual delay for TrueHD, as such audio cannot be cut at every frame but only at certain frames. A bit similar to keyframes in video, if you will.

You're right, some delay is for sure because it's cut to keyframe.

Do you suggest VirtualDub, VirtualDub2 or VirtualDubMod?

[Ennio]

VirtualDub2.

When cutting in the beginning of audio, be cautious to how loud the edited audio will start. Mind your speakers. You don't want to start in the middle of an explosion or whatever. If you'd have a bowl of soup on your lap, you'll have to mop the ceiling

[ultradj83]

VirtualDub2.

When cutting in the beginning of audio, be cautious to how loud the edited audio will start. Mind your speakers. You don't want to start in the middle of an explosion or whatever. If you'd have a bowl of soup on your lap, you'll have to mop the ceiling

Will I won't cut audio (it's Atmos), I'll cut video then I'll check drift...

[Ennio]

FYI MKVToolNix can cut TrueHD Atmos perfectly.

[ultradj83]

FYI MKVToolNix can cut TrueHD Atmos perfectly.

Ehm... Yes but after cut it looks like the Atmos channel is no more available so it's played as normal TrueHD...

[Ennio]

Maybe you found a bug. I have cut Atmos many times without issues. You're using latest version?

[ultradj83]

Maybe you found a bug. I have cut Atmos many times without issues. You're using latest version?

Yes. 69.0.0. But might be also a problem out of mkvtoolnik. With mediainfo I see the Atmos, but it doesn't play.
Should I load the mkv or the hevc in vdub?

[Ennio]

MKV. It doesn't play where?

And reading back:

[...after cut it looks like the Atmos channel is no more available so it's played as normal TrueHD...

May I ask with what SW/HW you tested this? It is not important to Virtualdub as you only would use this to check video keyframes.

[ultradj83]

MKV. It doesn't play where?

And reading back:

[...after cut it looks like the Atmos channel is no more available so it's played as normal TrueHD...

May I ask with what SW/HW you tested this? It is not important to Virtualdub as you only would use this to check video keyframes.

Nvidia Shield Pro and Harman/Kardon receiver. Both can decode atmos.

I sat vdub in direct stream mode but when I save it says I might want to set a different input driver...

[Ennio]

Unless I am misunderstanding something, why would you save anything in Vdub? No need to set drivers or whatever either. The cutting itself is done in MKVToolNix.

The idea is to load the HEVC video in Virtualdub, so that in the preview window you can determine what keyframes are available in the area where you want to cut (using the keyframe search buttons). Once having that info, you don't do any saving. Just write down the keyframe numbers/timings.
Close Virtualdub. Open MKVToolNix. Now use the keyframe info to set the correct timings/frames in MKVToolNix's cut-function (tab Ouput --> Splitting). Start muxing.

As a test, you can cut out any random part of Atmos video in MKVToolNix. For testing, you don't have to search for keyframes. Just set some random (but valid) timestamps for the cutting part and start output. MKVToolnix will auto-cut on nearest keyframes. Play the resulting mkv on your equipment. Atmos will play fine (unless a new bug is introduced).

[ultradj83]

Unless I am misunderstanding something, why would you save anything in Vdub? No need to set drivers or whatever either. The cutting itself is done in MKVToolNix.

The idea is to load the HEVC video in Virtualdub, so that in the preview window you can determine what keyframes are available in the area where you want to cut (using the keyframe search buttons). Once having that info, you don't do any saving. Just write down the keyframe numbers/timings.
Close Virtualdub. Open MKVToolNix. Now use the keyframe info to set the correct timings/frames in MKVToolNix's cut-function (tab Ouput --> Splitting). Start muxing.

As a test, you can cut out any random part of Atmos video in MKVToolNix. For testing, you don't have to search for keyframes. Just set some random (but valid) timestamps for the cutting part and start output. MKVToolnix will auto-cut on nearest keyframes. Play the resulting mkv on your equipment. Atmos will play fine (unless a new bug is introduced).

OK, so this is the plan.

Got HEVC track keyframe by vdub, it's at 4s while exact cut point should be at 2.988s.
Cut the HEVC track to keyframe by mkvtoolnix with the TrueHD track delayed by 2.988s. But how to find TrueHD keyframes? (I'll also check one more time the Atmos after cut on this).
Then there could still be some drift to be managed by changing fps with ffmpeg, being the HEVC 24fps and the AVC 23.967fps maybe I just need to change the HEVC from 24 to 23.967fps.

Right?

Tks!

[Ennio]

I think you're making it too hard for yourself. Don't worry about TrueHD cut-frames. Ever. MKVToolNix will take care of that. In doing so, it will adjust the audio delay if need be.

Another thing is to realise that you have determined the audio delay, but for video that is still 24 fps. Where audio is for 23.976 fps. If you change video framerate to match audiospeed, note that audio delay has to be adjusted also. Think of it this way: the time to play the extra frames until "audio startpoint" is reached, will change when you change the speed with which these frames are played

I think it's a good idea to explain in short steps how I would do things. In your case, first thing is take the 24 fps HEVC video and change framerate to 23.976. With that result, you are sure that video and audio are speed-synced.

I determine audio delay with visual help. I run two instances of Virtualdub. In the first one I load the 23.976 AVC stream. In the second I load the 23.976 HEVC video. Now I go looking for same frames. Hard cuts in the footage (POV change) are good points to look, or in fast pace pans/tilts and action scenes. Again, important is to be 101% sure that you have exactly the same frames. Take the framenumbers and determine the difference. Multiply this difference with frametime (41.708 ms is precise enough for 23.976 video) and you have the (extra) delay that has to be set (can be postive or negative of course).

Now that I have the needed info, I can go create end-result mkv. Load all files in MKVToolNix, select proper streams, set audiodelay, proper cut-values and hit it.

[Ennio]

Correctio et addendum to MKVToolNix video keyframe cut.

If the cut framenumber you set is not a keyframe, MKVToolNix will take the first next one (where I remember it taking the nearest one). I think something changed, or I have been wrong all this time. Anyways, my apologies.

Also, note that Virtualdub starts framecounting at 0, where MKVToolNix apparently starts at 1.

Obviously, the above means you have to set the cut-keyframe number (found in Virtualdub) 1 or more lower in MKVToolNix. Hope I'm not too late with this.

[ultradj83]

Correctio et addendum to MKVToolNix video keyframe cut.

If the cut framenumber you set is not a keyframe, MKVToolNix will take the first next one (where I remember it taking the nearest one). I think something changed, or I have been wrong all this time. Anyways, my apologies.

Also, note that Virtualdub starts framecounting at 0, where MKVToolNix apparently starts at 1.

Obviously, the above means you have to set the cut-keyframe number (found in Virtualdub) 1 or more lower in MKVToolNix. Hope I'm not too late with this.

Thank you Ennio.

I'm following your suggestions and I will check the result asap.
By the way, as far as I noticed, MKVtoolnik really goes to closest keyframe, I know this because in mu case it is the key before my exact cut point.

Regards

[Ennio]

Well, I'm beyond astonished to hear that. Just today I did a test where keyframe in Virtualdub was 122 and with that value set in MTX, the cut was done at the next one. With value set to 121, the cut was dead-on. The test was done with AVC video btw. Maybe MTX cut-behaviour is different with HEVC. Or maybe I'm growing certified to certain medication...
OTOH, your findings do confirm that I have remembered correctly.

Anyways, good luck. If issues/uncertainties should arise, don't hesitate to mention them.

[ultradj83]

I think you're making it too hard for yourself. Don't worry about TrueHD cut-frames. Ever. MKVToolNix will take care of that. In doing so, it will adjust the audio delay if need be.

Another thing is to realise that you have determined the audio delay, but for video that is still 24 fps. Where audio is for 23.976 fps. If you change video framerate to match audiospeed, note that audio delay has to be adjusted also. Think of it this way: the time to play the extra frames until "audio startpoint" is reached, will change when you change the speed with which these frames are played

I think it's a good idea to explain in short steps how I would do things. In your case, first thing is take the 24 fps HEVC video and change framerate to 23.976. With that result, you are sure that video and audio are speed-synced.

I determine audio delay with visual help. I run two instances of Virtualdub. In the first one I load the 23.976 AVC stream. In the second I load the 23.976 HEVC video. Now I go looking for same frames. Hard cuts in the footage (POV change) are good points to look, or in fast pace pans/tilts and action scenes. Again, important is to be 101% sure that you have exactly the same frames. Take the framenumbers and determine the difference. Multiply this difference with frametime (41.708 ms is precise enough for 23.976 video) and you have the (extra) delay that has to be set (can be postive or negative of course).

Now that I have the needed info, I can go create end-result mkv. Load all files in MKVToolNix, select proper streams, set audiodelay, proper cut-values and hit it.

Dear,
I'm back on duty proceeding like per your suggestion.

Comparing key frames on virtualdub I noticed it was synced only close to the end.
Checking files I noticed that FFmpeg in the truth fails to change fps, it returns no errors but the output file is still 24fps.
My syntax is:
ffmpeg -y -r 23.976 -i input.hevc -c copy output.hevc

I can't find no ways to change fps...

Regards

[ProWo]

My syntax is:
ffmpeg -y -r 23.976 -i input.hevc -c copy output.hevc
I can't find no ways to change fps...
Regards

ffmpeg -i input -c:v libx265 -crf 26 - r 24000/1001 - an - sn output.mkv

[ultradj83]

My syntax is:
ffmpeg -y -r 23.976 -i input.hevc -c copy output.hevc
I can't find no ways to change fps...
Regards

ffmpeg -i input -c:v libx265 -crf 26 - r 24000/1001 - an - sn output.mkv

I think this will transcode, right?

I just need to change fps...

Regards

[ProWo]

Can be done by mkvtoolnix, but i think there will be problems with many players, especially TV players

[ultradj83]

Dear,
I'm back on duty proceeding like per your suggestion.

Comparing key frames on virtualdub I noticed it was synced only close to the end.
Checking files I noticed that FFmpeg in the truth fails to change fps, it returns no errors but the output file is still 24fps.
My syntax is:
ffmpeg -y -r 23.976 -i input.hevc -c copy output.hevc

I can't find no ways to change fps...

Regards

I succeeded in changing fps, no idea what was wrong, maybe it fails output to mkv, it works to mp4.

Now I have:
Frame rate : 23.976 (23976/1000) FPS
Original frame rate : 24.000 FPS

Going on in comparison in Virtualdub...

[ultradj83]

I think it's a good idea to explain in short steps how I would do things. In your case, first thing is take the 24 fps HEVC video and change framerate to 23.976. With that result, you are sure that video and audio are speed-synced.

I determine audio delay with visual help. I run two instances of Virtualdub. In the first one I load the 23.976 AVC stream. In the second I load the 23.976 HEVC video. Now I go looking for same frames. Hard cuts in the footage (POV change) are good points to look, or in fast pace pans/tilts and action scenes. Again, important is to be 101% sure that you have exactly the same frames. Take the framenumbers and determine the difference. Multiply this difference with frametime (41.708 ms is precise enough for 23.976 video) and you have the (extra) delay that has to be set (can be postive or negative of course).

Here comes the damn...

After converting the HEVC track from 24 to 23.976fps it's still not synced, also clearing initial offset.

I checked 2 POV change key frames:

HEVC frame 1872
AVC frame 1817
Gap 55

HEVC frame 215136
Avc frame 215055
Gap 81

Variable frame rate problem?

Regards

[Ennio]

First off, it goes without saying that source and target video must be exactly the same frame by frame. When you have framecount differences changing, chances are the videos are not the same. In which case crossmuxing audio/video is useless.
Or, and I have seen this a couple of times, either or both source and target video suffer from bad timecodes. At which rewriting timestamps for both may help you to get rid of changing difference in offset.

Note that checking "Fix bitstream timing info" in MKVToolNix only works for AVC streams. HEVC is not (yet?) supported.

I succeeded in changing fps, no idea what was wrong, maybe it fails output to mkv, it works to mp4.

Now I have:
[I]Frame rate : 23.976 (23976/1000) FPS

I was afraid of this. Framerate 23976/1000 is NOT good. To change framerate (no transcoding, just speed-change by rewriting timestamps) of HEVC video, you want the elementary videostream as input to ffmpeg. Don't use any container, both for input and output. So, demux your video first. Rename the demuxed 24fps *.hevc stream into "input.hevc".
Use ffmpeg with this command:

ffmpeg -i input.hevc -c copy -bsf:v hevc_metadata=tick_rate=(24000/1001):num_ticks_poc_diff_one=1 output.hevc

Output will have proper (24000/1001) framerate (and not 23976/1000). If muxing an elementary videostream with MKVToolNix I ALWAYS set the container header "Default duration/FPS" to proper value. Players (this includes VirtualDub) can show A/V sync and time issues when this header is not set properly. So in your case, set 24000/1001p.

Good luck, let know how things work out.

[ultradj83]

Ennio,
I extracted the 24fps hevc and lowered to 24/1001 as you suggested.

Now how do I load a hevc stream to virtualdub for key frames comparison without to use a container?

Regards

[Ennio]

Note that my said "never use container" concerns ffmpeg.

I never preview elementary stream in VirtualDub as I wouldn't know how trustworthy frametime values would be. It's my custom to always mux (intermediate) mkv and load that in the preview. Again - DO NOT forget to set proper "Default duration/FPS" value in MTX GUI.