MP4/AVCC Strange i-frame size prefixes in video

[TheOmbudsman]

Hi all, new here.

I'm working on a program that can extract i-frames from potentially fragmented or corrupt video. In my test data set I've come across a video that I don't understand, which I've attached.

I thought that for MP4/AVCC, the NALU size is four bytes which prefix the start code (0x65 for an IDR frame). I have used ffprobe to give me a list of the I frames with their sizes, however when I scan through this particular video in HXD, I can't find any of these sizes referenced before each NALU, and each NALU is in fact much smaller. My program tries to extract these 0x65 prefixed NALUs and convert them to JPEGs, but it spits about a bunch of partial frames. I don't get this with any other video in my dataset which is a couple thousand videos, those spit out full frames.

If I had to make a guess I'd say that the i-frames are being split across multiple NALUs. I'd like to try and understand a bit more about this mechanism. Is there a way to tell how many NALUs go into each i-frame outside the metadata. Does the SPS/PPS specify this for example? Is it naive to assume that full i-frames exist within a single NALU?

Cheers

0.000000,6525,K_
0.200000,6517,K_
0.400000,8445,K_
0.600000,9599,K_
0.800000,9686,K_
1.000000,9423,K_
1.200000,10191,K_
1.400000,9962,K_
1.600000,9547,K_
1.800000,10164,K_
2.000000,10532,K_
2.200000,9999,K_
2.400000,10914,K_
2.600000,10381,K_
2.800000,8854,K_
3.000000,9110,K_
3.200000,10085,K_
3.400000,10805,K_
3.600000,10565,K_
3.800000,10230,K_
4.000000,10681,K_
4.200000,11174,K_
4.400000,15618,K_
4.600000,15747,K_
4.800000,15723,K_
5.000000,15718,K_
5.200000,15722,K_
5.400000,15719,K_
5.600000,15909,K_
5.800000,16077,K_
6.000000,15963,K_
6.200000,15960,K_
6.400000,15960,K_
6.600000,15960,K_
6.800000,15960,K_

[ProWo]

Maybe interesting:
https://stackoverflow.com/questions/66649466/how-to-differentiate-between-valid-and-in...d-nalu-in-avcc
and
https://stackoverflow.com/questions/53531003/formatting-an-avc-h-264-mdat
and
https://www.ietf.org/rfc/rfc3984.txt

[badyu17]

It's slice based encoding, instead of one slice for each picture, it uses 8 slices for each picture.

[TheOmbudsman]

Thanks for the replies

It's slice based encoding, instead of one slice for each picture, it uses 8 slices for each picture.

Apologies, but how do you know this? What markers in the file indicate this? Are the slices guaranteed to be in the right order?

Many thanks

[badyu17]

I'm using bitstream analyzer and with it you can see video structure. You can also use 'trace_header' option in ffmpeg to see similar info although in text form.

[TheOmbudsman]

I must apologise for the very late reply. Thank you for the help so far. When you have a frame which has multiple slices, what do the NALU's look like? Are all 8 slices prefixed with 0x65 to indicate they are key frames, or is it only the first?

Edit: I've found the answer to my own question. It looks like multiple slices both have their own 0x65's, but they aren't prefixed by a 0x67, 0x68 like the first slice of the IDR frame is

[badyu17]

No, they can't be prefixed with 0x67, 0x68 (sequence parameter set and picture parameter set) because slices are part of one picture, if that's what you are asking.

[TheOmbudsman]

No, they can't be prefixed with 0x67, 0x68 (sequence parameter set and picture parameter set) because slices are part of one picture, if that's what you are asking.

Thanks badyu, I really appreciate all the help.