I'm messing with the h264 code again... project?

[ChristianundCo]

It seems to be the holy grail to patch / cut / change image areas without re-encoding videos. I've been interested in this topic for years.

It's a matter of principle and I don't want any loss of quality, which is not absolutely necessary in digital technology.

Now my problem. I have HD material from restored films. But the recording service has its channel logo in it. The material is even encoded in the Blu-Ray standard. Tmpeg Authoring Works (which is very strict) accepts it and doesn't want to re-encode it.

The channel logo could be covered by a subtitle track using the old trick. But not every player interprets such a track in the same way. Sometimes you have flickering and when skipping the effect that the subtitle disappears and yet the logo is still visible. Also, if you select a different subtitle track, the logo cover disappears. Not the real thing.

The logo is completely in the black border and as I strongly suspect, the logo is only saved in the I-frames. (no motion vectors are affected)

Now I have done the following manually:

I deleted the first I-frame from the h264 elementary stream using a hex editor.

I encoded an I-frame with a covered logo in the same settings (!) as the original stream.

I used the hex editor to copy the new I-frame into the place of the old one.

And what can I say: it works! The logo is gone and the stream is intact. Every player (even the very critical ones) does not display a logo until the next I-frame.

So it works manually!

The only flaw is that I have completely re-encoded the I-frame. This should also be lossless. So patch the I-frame at the position and patch the position where the logo is black.

And I can't do that. I won't understand the 300-page PDF on the h264 codec in this lifetime.

There should be an editor that can patch I-frames without loss. Limited to my special case. (Logo in the margin, only I-frames, only h264, only black)

I could offer some money for it. But I don't know how many hours a codec programmer would need. And at 50 € per hour I'll soon be poor.

What do you think of the project?

[ChristianundCo]

Here is a pdf for that issue


https://www.researchgate.net/publication/224200981_Replacing_picture_regions_in_H264AV...pendent_slices

[ChristianundCo]

Today I found this tool: “h26forge”

https://github.com/h26forge/h26forge/blob/main/docs/EDITING.md

This allows you to edit h264. A py script is also loaded when the exe is called. The given example script contains the respective logic.
With this line:

h26forge modify --input orgSR.264 --output test1.264 -t transforms/slice_0_all_blue.py

I was able to make the first I-frame completely blue. (and thus the whole first GOP)

But I would have to understand the syntax how to color a few macro blocks black from I-frames. Who can help me to code the logic in the py script for that?

[ProWo]

You could also use clever FFmpeg-GUI and crop the video lossless, without recoding.
But it's not supported by all players.

Load your video, click main page, click various, click crop.
Insert the crop values, check with the preview button, and if the result is ok then click crop.
Done.


[Attachment 84058 - Click to enlarge]

[ChristianundCo]

Ok, thank you, but that doesn't help me. To many players I tried ignore that patch.

[_Al_]

So you want to find out where a data is stored in that ds object for a macroblock?
to iterate thru I-frames only is here:

if is_slice_type(ds["slices"][slice_idx]["sh"]["slice_type"], 'I'):
   #some crazy code to change a macroblock from that slice

how to reach macroblocs are a bit explained here where he iterates horizontaly, it looks macroblocs are in ds["slices"]:

for i in range(len(ds["slices"][0]["sd"]["macroblock_vec"])):

so find out, print this dictionary or just its keys, if there is a data in it, what is its key name :

ds["slices"][0]["sd"]["macroblock_vec"][i]

if is just quick look, it is quite complicated, you'd need a key for that data and then find out indexes (i) for your particular macroblock you need to change, if it is even possible. You'd need iterate those slices, not using zero index as above

[_Al_]

Or better try to modify https://github.com/h26forge/h26forge/blob/main/transforms/slice_all_pcm.py
only for certain x,y macroblocks, where you write YUV desired values (like for your black)
That might not be possible also, because he is changing the whole frame structure perhaps, but anyway, in that example is shown to iterate thru macroblocks

[ChristianundCo]

Thank you! I made a lot of tests yersterday. The script

def slice_all_pcm(ds):
    print("\t Setting all frames to all IPCM with increasing color")

    # UT colors converted from RGB 197, 87, 0 to YUV
    # using https://www.mikekohn.net/file_formats/yuv_rgb_converter.php
    # are 109, 65, 190

    y = 109
    u = 65
    v = 190

    from helpers import get_chroma_width_and_height

    (mb_width_c, mb_height_c) = get_chroma_width_and_height(0, ds)
    c_dims = mb_width_c * mb_height_c
    for i in range(0,1):
        for j in range(486,493):
            ds["slices"][i]["sd"]["macroblock_vec"][j]["mb_skip_flag"] = False

            ds["slices"][i]["sd"]["macroblock_vec"][j]["mb_type"] = "IPCM"
            ds["slices"][i]["sd"]["macroblock_vec"][j]["pcm_sample_luma"] = []

            for _ in range(256):
                ds["slices"][i]["sd"]["macroblock_vec"][j]["pcm_sample_luma"].append((y + 37 * i) % 256)
            ds["slices"][i]["sd"]["macroblock_vec"][j]["pcm_sample_chroma"] = []
            for _ in range(c_dims):
                ds["slices"][i]["sd"]["macroblock_vec"][j]["pcm_sample_chroma"].append((u + 37 * i) % 256)
            for _ in range(c_dims):
                ds["slices"][i]["sd"]["macroblock_vec"][j]["pcm_sample_chroma"].append((v + 37 * i) % 256)

    return ds

def modify_video(ds):
    return slice_all_pcm(ds)

I only want to edit the macroblocks 486 to 492 on first slice. But it seems that changing their color influence the following macroblocks down there.
And not just at the border the green content is orange too. (not really to see here but it does - believe me, I just covered it because of movie rights) The modified blue script does the same... (in blue)
Now I'm looking for a solution to set the pixels black without influence other pixels or mblocks.


[Attachment 84071 - Click to enlarge]

[_Al_]

quick AI google says:

In the context of H.264 video encoding, "IPCM" stands for "Intra-Pulse Code Modulation," which refers to a specific macroblock mode where the pixel values are transmitted directly without any prediction, transformation, or quantization, essentially sending the raw data for that block, typically used in situations where precise representation of a small area within the video is crucial, even if it means using more data than other encoding methods.

quite interesting that perhaps a precise pixel values for a macroblock could be inserted in video,
not much time these days, might come back later, just some points:
-can we inject just new macroblocks like that, I guess thats purpose of this testing?

for _ in range(256)

-would mean macroblocks 16x16, are we sure about macroblock size? c_dims as chroma dimmension should be calculated ok or you might override it to 64 (for YUV420, guessing it it is a size for chroma matrix for 16x16 luma)

-guessing size change for frame: For every frame and even only one macroblock, 256+64+64 bytes have to be inserted in video. Times 7macroblocks = 2.7kBytes. That is 2.7kBytes per frame on the top of video size. For 10 minute 30fps video, it would be: 10minx60secx30fps=18000, 18000x2.7kB = 48MB. So it looks that macroblock override (even if it works) increases size considerably (because of storing raw values)
-for injecting simple values:

            for _ in range(256):
                ds["slices"][i]["sd"]["macroblock_vec"][j]["pcm_sample_luma"].append(y)
            ds["slices"][i]["sd"]["macroblock_vec"][j]["pcm_sample_chroma"] = []
            for _ in range(c_dims):
                ds["slices"][i]["sd"]["macroblock_vec"][j]["pcm_sample_chroma"].append(u)
            for _ in range(c_dims):
                ds["slices"][i]["sd"]["macroblock_vec"][j]["pcm_sample_chroma"].append(v)

he is creating some values that are animated, changed per frame, otherwise I'm not sure what is going on

[ChristianundCo]

The 1. for-loop iterates only to i=0 so the old code "y+37 *i" is the same as y. But I tried your code and nearly the same effect with other colors.
Perhaps interesting the difference schema between coded and original I-frame: Here you can see which areas are effected only when I modify
macroblock "486" (it's where the logo starts, so your code with i =0 and j = 486)

[Attachment 84074 - Click to enlarge]


48 MB are no problem for 10 minutes with a 2 gigabyte file (of 30 minutes)


I counted the macroblocks in first slice which is the first I-frame with the code

print(len(ds["slices"][0]["sd"]["macroblock_vec"]))

the answer was: 8160 (makes sense for 16x16 because the macroblock pixels of video are 1920*1088)


I also had an idea to copy the values from a black macroblock of the border to a logo-macroblock
(number "0" or "2" or "8159" for example because mblocks start from the left corner on top )

ds["slices"][0]["sd"]["macroblock_vec"][486] = ds["slices"][0]["sd"]["macroblock_vec"][2]

but then the entire frame content is damaged.

[ChristianundCo]

I have looked at the code and understood most of it. The following code processes all I-frames and sets the luma PCM values of macro blocks 487 to 492 to black. These are a few macro blocks from the first line of the logo.

def slice_all_pcm(ds):


    y = 0
    u = 128
    v = 128

     c_dims = 256 
   
    
    for i in range(len(ds["slices"])):
       if ds["slices"][i]["sh"]["slice_type"] == 7:

            for j in range(487,493): #486 org
                ds["slices"][i]["sd"]["macroblock_vec"][j]["mb_skip_flag"] = False

                ds["slices"][i]["sd"]["macroblock_vec"][j]["mb_type"] = "IPCM"
                ds["slices"][i]["sd"]["macroblock_vec"][j]["pcm_sample_luma"] = []

                for _ in range(256):
                    ds["slices"][i]["sd"]["macroblock_vec"][j]["pcm_sample_luma"].append(y)
                ds["slices"][i]["sd"]["macroblock_vec"][j]["pcm_sample_chroma"] = []
                for _ in range(c_dims):
                    ds["slices"][i]["sd"]["macroblock_vec"][j]["pcm_sample_chroma"].append(u)
                for _ in range(c_dims):
                    ds["slices"][i]["sd"]["macroblock_vec"][j]["pcm_sample_chroma"].append(v)
                        
    return ds
        

def modify_video(ds):
    return slice_all_pcm(ds)

Unfortunately, some of the brightness information in the logo blocks is later reused in the actual image content. So, if I turn the brightness (luma) of the logo to dark, the image also becomes dark in some places. So sometimes they are connected.

Above on the difference image you can see that if I darken Macroblock 486, then other image contents also become darker.

The logo can therefore not be patched even when encapsulated in an I-frame.

In addition, h26forge serializes the entire video. It therefore writes the entire bitcode in “plain text” in a json file with hundreds of gigabytes for just 12 seconds of video. And these lines are then edited individually and written back to bitcode. This process is not optimized.

The CPU is bobbing around at 1% utilization.

The RAM (and virtual memory on SSD) is 460 gigabytes!

And 12 seconds take 15 hours!!!

Maybe someone will find another approach, but I can't get any further for now. But it was an interesting and instructive experiment. And in the 12 seconds I created, the logo is almost gone and the image is mostly intact and unaffected.

[ChristianundCo]

Now I'm going to continue a little further.

The fact that other macro blocks are influenced is due to the prediction. You can equalise this with the residual value for each pixel. So that you end up with 0 (black) in the final pixel. Supposedly you only need to negate the prediction values. This would make the pixel black, but the other macro blocks would not be changed. I'll take a closer look at this.

I've also already thought of something for later use on longer videos to overcome the poor performance.

I'll keep at it for now...

[ChristianundCo]

Finally a breakthrough today.🎉🎉🎉 I looked at the Python examples from 26forge and with some trial and error I created the following code.

##
# Makes all macroblocks of logo black*
#
##

def slice_all_pcm(ds):

for h in range(len(ds["slices"])):
* *
    for i in range(486,493):
* * * * * *
       for k in range(len(ds["slices"][h]["sd"]["macroblock_vec"][i]["luma_level_4x4_transform_blocks"])):
           ds["slices"][h]["sd"]["macroblock_vec"][i]["luma_level_4x4_transform_blocks"][k]["coded_block_flag"] = False

     for i in range(486+120,493+120):
* * * * * * * * * * * *
         for k in range(len(ds["slices"][h]["sd"]["macroblock_vec"][i]["luma_level_4x4_transform_blocks"])):
              ds["slices"][h]["sd"]["macroblock_vec"][i]["luma_level_4x4_transform_blocks"][k]["coded_block_flag"] = False
* * * * * **
* * * *
     for i in range(486+240,493+240):
* * * * * * * * * * * *
         for k in range(len(ds["slices"][h]["sd"]["macroblock_vec"][i]["luma_level_4x4_transform_blocks"])):
              ds["slices"][h]["sd"]["macroblock_vec"][i]["luma_level_4x4_transform_blocks"][k]["coded_block_flag"] = False
 * * * * * * * *
* * * * * * * *
     for i in range(486+360,493+360):
* * * * * * * * * * * *
         for k in range(len(ds["slices"][h]["sd"]["macroblock_vec"][i]["luma_level_4x4_transform_blocks"])):
                ds["slices"][h]["sd"]["macroblock_vec"][i]["luma_level_4x4_transform_blocks"][k]["coded_block_flag"] = False
* * * * * * * *
* * * * * *** *
    for i in range(967,972):
* * * * * * * * * * * *
         for k in range(len(ds["slices"][h]["sd"]["macroblock_vec"][i]["luma_level_4x4_transform_blocks"])):
              ds["slices"][h]["sd"]["macroblock_vec"][i]["luma_level_4x4_transform_blocks"][k]["coded_block_flag"] = False
* * * * *
return ds
* *
def modify_video(ds):
return slice_all_pcm(ds)

The logo has disappeared and other parts of the image are not affected. The basis is there.
More than 30 I-Frame at once are not possible. (Memory leaks) But I don't need more. I give h26forge only parts of 30 frames and combine the parts later.

[_Al_]

That's great. So in manual it is referenced as "remove_all_frame_residue" to get rid of some references.

def slice_all_pcm(ds):
    mv_ranges = (
                 range(486,493),
                 range(486+120,493+120),
                 range(486+240,493+240),
                 range(486+360,493+360),
                 range(967,972)
                 )
    for h in range(len(ds["slices"])):
        mv = ds["slices"][h]["sd"]["macroblock_vec"]
        for mv_range in mv_ranges:
            for i in mv_range:
                for k in range(len(mv[i]["luma_level_4x4_transform_blocks"]))::
                    mv[i]["luma_level_4x4_transform_blocks"][k]["coded_block_flag"] = False 
    return ds

Why those macroblock vectors are selected with 120 step?

[ChristianundCo]

One line has 120 mbs. And so logo is 5 lines high. I was lazy so I jumped to next line with +120