Splitting into 3, editing 1, can't rejoin bc private codec data don't match

[sneaker]

For me this leads to "most successful"(*) appending:

ffmpeg -i "test.mkv" -c:v libx264 -x264-params keyint=24:bluray-compat=1:slices=4:level=4.1:ref=4:slices=4:fps=24000/1001 -bsf:v dump_extra=freq=k "partb.mkv"
mkvmerge -o "output.mkv" "test.mkv" "+partb.mkv"

(*)
- I used appending instead of cutting out a part because the short sample only has 2 idr frames (frames #0 and #19)
- "most successful" as in playing past the cut point as well as seeking forwards and backwards between the parts worked for me in both MPC-HC and VLC
- I didn't bother to reproduce SPS/PPS 100% identical. It's often impossible to do this between different encoding softwares. So I rely on the repeated SPS/PPS within the bitstream ("in-band") for the job. Should work with most players.
- The source already had SPS/PPS at every keyframe so this made things easier. Just had to make sure the re-encoded part did as well.
- I did NOT create different SPS/PPS IDs (although logically it would seem to make sense some players don't seem to like it... *sigh*..).

[Travillion]

For me this leads to "most successful"(*) appending:

ffmpeg -i "test.mkv" -c:v libx264 -x264-params keyint=24:bluray-compat=1:slices=4:level=4.1:ref=4:slices=4:fps=24000/1001 -bsf:v dump_extra=freq=k "partb.mkv"
mkvmerge -o "output.mkv" "test.mkv" "+partb.mkv"

(*)
- I used appending instead of cutting out a part because the short sample only has 2 idr frames (frames #0 and #19)
- "most successful" as in playing past the cut point as well as seeking forwards and backwards between the parts worked for me in both MPC-HC and VLC
- I didn't bother to reproduce SPS/PPS 100% identical. It's often impossible to do this between different encoding softwares. So I rely on the repeated SPS/PPS within the bitstream ("in-band") for the job. Should work with most players.

Incredibly helpful, thank you!

To make sure I understand what you did: you recoded test.mkv rather than cutting it into parts due to small # of idr frames. After recoding test.mkv with the above ffmpeg command (test.mkv->partb.mkv) you then joined test.mkv + partb.mkv and they play back/seek fine. I am at work right now and unable to view your completed video (hence my double-checking).

A few follow-up questions. You were able to determine early on that this stream uses openGOP--how can you tell if it's open or closed?
You were also able to identify the IDR frames from the i frames. How so? I use avidemux to find keyframes, but I am not sure if those are verified IDR frames or, like ffmpeg and mkvmerge, it mistakes i frames for IDR frames.
I recognize most of the ffmpeg parameters you used, but I will google those I don't know.

Thanks again!

[sneaker]

To make sure I understand what you did: you recoded test.mkv rather than cutting it into parts due to small # of idr frames. After recoding test.mkv with the above ffmpeg command (test.mkv->partb.mkv) you then joined test.mkv + partb.mkv and they play back/seek fine.

Yes, exactly. ( P.S.: I added 2 more notes to my previous post )

A few follow-up questions. You were able to determine early on that this stream uses openGOP--how can you tell if it's open or closed?
You were also able to identify the IDR frames from the i frames. How so? I use avidemux to find keyframes, but I am not sure if those are verified IDR frames or, like ffmpeg and mkvmerge, it mistakes i frames for IDR frames.
I recognize most of the ffmpeg parameters you used, but I will google those I don't know.

Basically I looked at it with a stream analyzer. I know on Blu-Ray you must have a keyframe at least every other second. I could see the third I frame already wasn't an IDR frame ("idr_pic_id" not set, out of order with following B frames that reference said I frame). With ffmpeg's trace_headers filter I found only 8 instances of "idr_pic_id". Divided by 4 because of the 4 slices thingy that meant those first two I frames I saw were the only IDR frames.
Now I have to admit this is neither a very systematic nor automated way. Maybe someone has a good suggestion to do this better. Also, stream analyzers are expensive (see post #2).

[poisondeathray]

Just to explain what you were seeing farther:


Those were not IDR frame boundaries. As sneaker pointed out, there are only 2 IDR frames near the beginning

That cut section 2 plays *almost* normally, because you have a recovery point "i" frames (these are non IDR frames). Notice I said "almost" . If you go frame by frame, there is actually 2 repeat frames at the beginning , because in display order there are 2 leading b-frames before the "i" frame . This is open GOP, and those are known as "orphaned" b-frames. You need to reference the previous section for that missing information. (The behaviour can be different for some decoders, some drop the 2 frames, some repeat the next recovery point as duplicate frames). Another way you can see this is open part 1 and part 2 with two instances of a player. There will be a gap between the last frame of part 1 and the first frame of part 2. If you're having problems opening 2 instances, try 2 instances vdub2. You will see a "jump" or between the end of part 1 and the beginning of part 2.

Joining the split original parts with mkvmerge works, because you reference the original IDR frame in part 1 just like the original. But when you re-encode part 2, you lose those references to the IDR frame in part 1 causing a glitch.

Look at the first part of the split part 2 in premiere or any program. You will have either dropped or glitched 2 frames at the start to begin with because of the way it was cut. So even if you could join, there would be errors or repeat frames, or dropped frames. If you look at "testparts-002film.mkv" there are garbled frames at the beginning - that's because of the opengop and missing information when it's decoded, then re-encoded

The main point is you need to cut on IDR boundaries, it's the most common error when trying to do this appending without re-encode. Unfortunately, mkvmerge, ffmpeg , ffprobe, avidemux all have problems with distinguishing true IDR frames. You need to use a stream analyzer to do it accurately

[Travillion]

I want to respond to poisondeathray and sneaker's two preceding posts, but I'm not sophisticated enough to grab quotes from multiple posts, so forgive me.

This is all very helpful information and I really appreciate you taking the time to assist in this. I am persistent and detail-oriented (the same attribute that wouldn't let me just reencode the whole video for a stupid project like this) so I can learn, but some of the explanations I find online go over my head and I make some silly mistakes. So again, thank you for hashing this out with me! And poisondeathray's explanation of the bad cut makes sense. I thought it was good but it is not.

It is clear to me now that I need to use a stream analyzer in order to correctly identify IDR frames before making any future cuts. I have h264_parse (either poisondeathray or sneaker linked that in post #2) and I have successfully collected sps/pps logs (there is 0 documentation for that program, btw....!). I'm assuming I can identify idr frames there, or I could figure out ffmpeg's trace_headers filter. Is that sufficient to distinguish true IDR frames? And say I have a cut at the end of a file (e.g., credits)--is there a way to "seek" to the approximate spot and find the closest idr frames there? It seems unlikely that I could extract the full stream analysis and be able to identify what part (time) of the stream it is. Then translating that idr frame to mkvmerge or ffmpeg command line for the cut.... in practice, how is this done? I guess that's why VideoCodec costs so much. But on the cheap, how do people do this? (by "this" I mean: find the idr frames near a predetermined time point, and then communicate those idr frames for a cutting program [e.g., frame # or timestamp?])

Sneaker said:

...So I rely on the repeated SPS/PPS within the bitstream ("in-band") for the job. Should work with most players.
- The source already had SPS/PPS at every keyframe so this made things easier. Just had to make sure the re-encoded part did as well.

Can you explain how you enforced the sps/pps data on every keyframe in the re-encoded file? Is there a setting to force that, or it was just luck that the source did and so the re-encode kept them? And how did you verify that the information transferred over?

[sneaker]

Can you explain how you enforced the sps/pps data on every keyframe in the re-encoded file? Is there a setting to force that, or it was just luck that the source did and so the re-encode kept them?

It has nothing to do with whether or not the source had it. I did it by setting -bsf:v dump_extra=freq=k. Note that this is not the only way. Had I outputted .264 instead of .mkv (i.e. H.264 elementary stream instead of mkv container) then ffmpeg (as well as x264cli) would do this automatically. Mkvmerge may also duplicate SPS/PPS IIRC but only starting from the second part and probably only with elementary stream input. So there is more than one way to achieve this.

And how did you verify that the information transferred over?

Whether or not the SPS/PPS are in the output .. you need a stream analyzer again. (Well, if you want to check.) And again: it didn't transfer over, it's new information.

[Travillion]

Can you explain how you enforced the sps/pps data on every keyframe in the re-encoded file? Is there a setting to force that, or it was just luck that the source did and so the re-encode kept them?

It has nothing to do with whether or not the source had it. I did it by setting -bsf:v dump_extra=freq=k. Note that this is not the only way. Had I outputted .264 instead of .mkv (i.e. H.264 elementary stream instead of mkv container) then ffmpeg (as well as x264cli) would do this automatically. Mkvmerge may also duplicate SPS/PPS IIRC but only starting from the second part and probably only with elementary stream input. So there is more than one way to achieve this.

And how did you verify that the information transferred over?

Whether or not the SPS/PPS are in the output .. you need a stream analyzer again. (Well, if you want to check.) And again: it didn't transfer over, it's new information.

Ah, I see, it wasn't SPS/PPS info coming over from the source, it was new info about the encode placed in SPS/PPS. Thank you for the explanation and alternative methods.

I can't wait to get home and sit down with this. I'll get the stream info for the 30-second source clip and comb through that, since I already know how many # of idr frames, etc. I'll begin to familiarize myself with spotting them

[Travillion]

One more follow-up question. It was determined that in my 30-second clip, there were two IDR frames, #0 and #19. So, that means both IDR frames were within the first second, correct (23.976 fps)? Isn't it typical for IDR frames to be spaced a maximum of about 10 seconds apart (250 frames)? So is my source clip (30s long) an exception or do I have a misunderstanding of IDR frames?

It was just the opening logo, so I can see how frame #19 probably comes in after the fade-in and sets up the scene, with the remaining frames containing difference data, but still I thought there would be more forced IDR frames within a 30 second clip.

[sneaker]

On Blu-Ray there are certain rules about keyframe placement. In most cases they mustn't be more than 1 second (i.e. 24 frames on a 24 fps movie) apart. Similar is true for DVDs and TV broadcasts (you don't want to wait 10 seconds for the picture to appear when you switch a channel). 250 frames is only the default of x264 which is not limited to Blu-Rays encodings. There aren't such strict rules for e.g. mkv creation.

Now when I said "keyframe" above I mean IDR frame OR "recovery point". The latter is basically an I frame which you can seek to and is referenced by the preceding GOP (this is what we call OpenGOP). It doesn't have to be an IDR frame every second, such recovery points are sufficient. So you could in fact have a 120 minute movie with only the very first frame being IDR and still have a perfectly valid Blu-Ray ready for mass sale!

[poisondeathray]

For other tools, other than various stream analyzers , DGAVCIndex and DGIndexNV (tools from neuron2/videoh/donald graft used in avisynth and vapoursynth) can accurately distinguish AVC IDR frames if you open the .dga or .dgi in a text editor

DGAVCIndex is depreciated/discontinued (but it's archived on this site), and DGIndexNV is technically payware (require donation) , and requires a compatible Nvidia card

The .dga is more difficult to read/interpret because the frame numbers aren't listed clearly . But it might be possible to run it through a macro or some text filters to format it into something easier to parse. There is no parse tool (IIRC DGIndex for MPEG2 has a d2v parse tool) . A full description of the dga/dgi syntax is listed in the manuals

[Travillion]

On Blu-Ray there are certain rules about keyframe placement. In most cases they mustn't be more than 1 second (i.e. 24 frames on a 24 fps movie) apart. Similar is true for DVDs and TV broadcasts (you don't want to wait 10 seconds for the picture to appear when you switch a channel). 250 frames is only the default of x264 which is not limited to Blu-Rays encodings. There aren't such strict rules for e.g. mkv creation.

Now when I said "keyframe" above I mean IDR frame OR "recovery point". The latter is basically an I frame which you can seek to and is referenced by the preceding GOP (this is what we call OpenGOP). It doesn't have to be an IDR frame every second, such recovery points are sufficient. So you could in fact have a 120 minute movie with only the very first frame being IDR and still have a perfectly valid Blu-Ray ready for mass sale!

So BR streams have either an idr frame or a recovery point every second. But the recovery points can contain data from the previous GOP, so splitting on one of these would cause the same recombine problems. In the case of my 30 second clip, I'm assuming the idr frames were at the beginning and every following second was a recovery point?

[Travillion]

For other tools, other than various stream analyzers , DGAVCIndex and DGIndexNV (tools from neuron2/videoh/donald graft used in avisynth and vapoursynth) can accurately distinguish AVC IDR frames if you open the .dga or .dgi in a text editor

DGAVCIndex is depreciated/discontinued (but it's archived on this site), and DGIndexNV is technically payware (require donation) , and requires a compatible Nvidia card

The .dga is more difficult to read/interpret because the frame numbers aren't listed clearly . But it might be possible to run it through a macro or some text filters to format it into something easier to parse. There is no parse tool (IIRC DGIndex for MPEG2 has a d2v parse tool) . A full description of the dga/dgi syntax is listed in the manuals

Thanks for the info! I'll check compatibility with my graphics card (it is Nvidia, but is also pretty old....).

[Travillion]

So... I thought it would be easy to read the h264_parse output for the test file, considering sneaker told me the IDR frames were #0 and #19.

When I look at the data, I can't find the second IDR frame (#19). I find 4 slices at the beginning with output like this (the first slice of the first IDR frame):
Nal length 16951 start code 3 bytes
ref 3 type 5 Coded slice of an IDR picture
first_mb_in_slice: 0
slice_type: 7 (I)
pic_parameter_set_id: 0
frame_num: 0 (16 bits)
idr_pic_id: 0
pic_order_cnt_lsb: 0

I know there are 4 slices per frame, so I figure these are the four slices for frame #0. Doing a "find" in the file I come across 4 more slices later on, but they look almost identical to the first 4 IDR slices e.g. (the first slice of the second 4, that is, the 5th IDR slice in the file):
Nal length 1500 start code 3 bytes
ref 3 type 5 Coded slice of an IDR picture
first_mb_in_slice: 0
slice_type: 7 (I)
pic_parameter_set_id: 0
frame_num: 0 (16 bits)
idr_pic_id: 0
pic_order_cnt_lsb: 0

I thought frame_num would give me the frame number...but it is 0 for each instance. In sum, there are 8 slices (2 clusters x4 slices) of IDR frame in the h264_parse output for the test.mkv file, and all of them say idr_pic_id: 0 and frame_num: 0. Am I misreading this, how do I find the frame number for the IDR frame?

Attached is the output from the h264_parse for the test.mkv file uploaded earlier (first 30 seconds of stream, untouched).

[poisondeathray]

So BR streams have either an idr frame or a recovery point every second. But the recovery points can contain data from the previous GOP, so splitting on one of these would cause the same recombine problems. In the case of my 30 second clip, I'm assuming the idr frames were at the beginning and every following second was a recovery point?

That 1 second rule is "maximum" . IDR "I" or non-IDR "i" frames can come earlier or closer than 1 second.

Yes, only frame zero, and 19 were true IDR frames . The other ones were non IDR

The other cases of BD (sneaker said "most cases") , use a 2 second GOP, but have lower bitrate limitations. These are very rare and usually not found on retail BD's, certainly not major Hollywood BD's , but still "legal"

Another tool, is selur's framecounter. It's accurate for IDR frames and prints a text in human readable format with frame numbers . It doesn't list "i" frames correctly, but for splitting purposes, what you only need is the IDR identification. In the attached , you will see it correctly identified zero and 19

http://forum.selur.de/topic12-framecounter-counting-frames-in-raw-streams-source.html

[Travillion]

Another tool, is selur's framecounter. It's accurate for IDR frames and prints a text in human readable format with frame numbers . It doesn't list "i" frames correctly, but for splitting purposes, what you only need is the IDR identification. In the attached , you will see it correctly identified zero and 19

http://forum.selur.de/topic12-framecounter-counting-frames-in-raw-streams-source.html

Thank you! I will look at this program. For whatever reason h264_parse is not correctly identifying the frame numbers for the idr frames (or more likely, I am not reading it correctly.) It correctly identified only two IDR frames, but listed both as frame_number: 0. frame_number: 19 is listed as a b frame:

Nal length 5900 start code 3 bytes
ref 2 type 1 Coded slice of non-IDR picture
first_mb_in_slice: 0
slice_type: 6 (B)
pic_parameter_set_id: 0
frame_num: 19 (16 bits)
pic_order_cnt_lsb: 29

[poisondeathray]

I guess avoid h264_parse for identifying IDR frames. But there are several versions floating around, one of them might be working correctly.

[Travillion]

I guess avoid h264_parse for identifying IDR frames. But there are several versions floating around, one of them might be working correctly.

The only thing I really needed was to be able to identify IDR frames (with accompanying frame number), and Cu Selur's little program is perfect for that. Now, I can just find the approximate place I want to cut, multiply # of seconds by 23.976 to get approx frame #, and find the nearest IDR frames to that. mkvmerge allows me to enter frame # for cutting so it should be a piece of cake now.

[poisondeathray]

If you go the avisynth/vapoursynth with DGIndexNV, you can get an overlay displaying the frame type (including IDR frames) and frame number by using show=true in the script. You can preview scripts in avspmod or vdub2, or even media players like mpchc, mpv, or utilities like ffplay/ffmpeg have avs support too. You can play back the script in realtime (or faster) as if it was a video. Since it's frame accurate and indexed, you know the exact frame number and you can actually "see" the frames instead of just a number. Also there is no seeking error or mixed up frames when scrubbing around that you can sometimes get with some programs. The DGIndexNV utility can also be used cut and mark in/out segments, but unfortunately you can't feed a script back into it to see the overlay

[Travillion]

If you go the avisynth/vapoursynth with DGIndexNV, you can get an overlay displaying the frame type (including IDR frames) and frame number by using show=true in the script. You can preview scripts in avspmod or vdub2, or even media players like mpchc, mpv, or utilities like ffplay/ffmpeg have avs support too. You can play back the script in realtime (or faster) as if it was a video. Since it's frame accurate and indexed, you know the exact frame number and you can actually "see" the frames instead of just a number. Also there is no seeking error or mixed up frames when scrubbing around that you can sometimes get with some programs. The DGIndexNV utility can also be used cut and mark in/out segments, but unfortunately you can't feed a script back into it to see the overlay

That sounds really nice, I'll check it out. I didn't yesterday because I think I have installed about 5 different programs/add-ons over the last week trying to make this work (UT Video codec, avidemux, VideoCodec, h264_parse, FrameDecoder, etc.) and was just burned out. But the functionality you described sounds perfect.

After some more tinkering yesterday evening, I am happy to say that I was able to complete this project. I certainly would have never arrived at this point without the assistance of poisondeathray and sneaker. Your knowledge and recommendations were invaluable. I was able to use FrameDecoder to find the IDR frames to make the appropriate cuts, and the encode settings sneaker provided worked like a champ. I still got the private data error when I rejoined the clips, which it seems will never go away, but the playback is pristine and flawless, no skipped/dropped/dummy frames at all. For all intents and purposes, it is exactly what I wanted: a seamless edit with minimal quality loss due to only reencoding the small edited segment. Not to mention, I feel like I have a much better understanding of the format and some of the editing tools. Thanks again!

[poisondeathray]

You shouldn't get any errors, not even codec private data whatever errors.

Did you remember to enter the SAR ?

[sneaker]

You shouldn't get any errors, not even codec private data whatever errors.

No, it's expected if he used my commands. As I said I didn't bother matching SPS/PPS 100% (often impossible if different encoding softwares were used). So mkvmerge will display a warning. But it will work anyways because SPS/PPS are in-band.

[Travillion]

Yeah, the only changes I made to sneaker's command was to add a crf value (it made sense to me to bring the overall size as close as possible to pre-edit in order to minimize additional compression--is that even logical?). It looks like slices=4 was set twice, but I haven't tried removing one of the instances yet. Was that an error or deliberate?

Getting the private data to match was way beyond my skillset, and I assumed from sneaker's comment, beyond anyone else's skillset. While I would love 0 errors, the functionality seems to be the same . Before, it wouldn't even play after the join point.

[poisondeathray]

You shouldn't get any errors, not even codec private data whatever errors.

No, it's expected if he used my commands. As I said I didn't bother matching SPS/PPS 100% (often impossible if different encoding softwares were used). So mkvmerge will display a warning. But it will work anyways because SPS/PPS are in-band.

I'd just use an elementary stream . I don't like any errors , even if they are negligible. I just tested with the mkvtoolnix instead of the "old" dos method, no errors reported, no warnings . I split at frame 19 (so 2 segments 0-18, 19-end) , inserted a 5 second test section in between the two

But I used x264, didn't have to use sps-id . sps-id method is the heavy hitter, for problems that even retail software like videoredo has problems with and cannot join. My take on it is if "regular"appending works, just use it. This is just regular appending. ffmpeg should work too

x264k --bitrate 10000 --bluray-compat --level 4.1 --ref 4 --keyint 24 --slices 4 --sar 1:1 -o output.264 input.avs

No errors. No codec private errors. No warnings. Plays ok in multiple players, seeks ok, indexes ok

[sneaker]

While mkvmerge doesn't display said warning if you use ES input the underlying problem remains. It makes no difference.

[Travillion]

I will have to look into that. Like I said, I prefer no errors.

I am still a novice at this, but my take-away from other threads where the elementary stream method was discussed is that this is, in some ways, another "hack." That it's a way of bypassing the error messages but that doesn't mean the private data actually match. I imagine that unless I could recode the file with the same encoder and settings the studio used, I will never get the private data to match. Appending the raw data instead of muxed data doesn't change that.

I genuinely want to know, so please correct me where I am wrong.

[edit: apparently I was typing this at the same time as sneaker, but I am slower]

[poisondeathray]

Fewer or no error messages make me feel better.

Re-encoding is the only way if you want everything perfectly matched and the underlying problems perfectly addressed in all cases

It's not really more difficult to use elementary streams.

The sps-id method is more of a "hack" . But it works when other methods fail . (Fail as in corrupt image, glitches, or non union) . I'd only use it as a last resort, not as 1st try. You don't have to use it here. It's more for when you have very different streams. BD is pretty standard (even with the open gop)

[sneaker]

Re-encoding is the only way if you want everything perfectly matched and the underlying problems perfectly addressed in all cases

I agree you should match the SPS/PPS 100% if possible. But often it's not. (Does your re-encode achieve this? I haven't looked closer but my guess is it doesn't either.)

It's not really more difficult to use elementary streams.

You missed the part where IT LITERALLY DOES NOT MAKE ANY DIFFERENCE. You know why it doesn't display a warning about different "CodecPrivate" data when you are inputting your ES files? Because ES files don't have Matroska "CodecPrivate" elements. Only Matroska files have those. This does NOT imply your SPS/PPS are identical/compatible.
https://matroska.org/technical/specs/index.html#CodecPrivate

[poisondeathray]

I never said SPS/PPS are identical or compatible ; You missed the part where I said fewer error messages makes me FEEL better - not that it "is better" . That's why I said re-encoding - this means re-encoding the entire thing - that's the only way to be sure in all cases. That's why I said it's not more difficult to use elementary streams , so why not use elementary streams and avoid the error message ? It certainly make me feel better