MP4 with H264,AAC cutting and replacing frames

[eymerich]

Hi,

I'm dealing with MP4 file (H264, AAC). In the first GOP (between 2 I-frames) I need to replace some frames, re-encode it and join with the rest without re-encoding the entire file.
Let's say the file is 22 minutes. The first GOP is 480ms made of 12 frames. I need to replace some of them, re-encode this 480ms into a single file and join it with the rest (22m - first 480ms)
1) I'm able to cut the 22m into the 2 pieces (480ms, and the rest), avidemux or mp4box
2) I'm able to extract the frames using ffmpeg
3) ** I'm able to encode in h264 the first 12 frames (virtualdub) **
4) Join the newly created video with the big 2nd part. MP4Box (into mp4 container)

Problem: the joint file does not work! It is made of the 2 video streams but only the first is played...
Is it the step 3 not producing something correct!?? If played along looks fine...

Can you help me and tell me what parameters I need to use to encode the h264 to have the same encoding of the original file?

Or there is a better way/software to do the job?

Here is the source file with mediainfo

General
Complete name : C:\Videos\Conversion\2x09.mp4
Format : MPEG-4
Format profile : Base Media
Codec ID : isom
File size : 219 MiB
Duration : 22mn 57s
Overall bit rate : 1 334 Kbps
Writing application : Lavf52.36.0

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : Baseline@L3.0
Format settings, CABAC : No
Format settings, ReFrames : 1 frame
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 22mn 56s
Bit rate mode : Variable
Bit rate : 1 202 Kbps
Width : 640 pixels
Height : 480 pixels
Display aspect ratio : 4:3
Frame rate mode : Constant
Frame rate : 25.000 fps
Resolution : 24 bits
Colorimetry : 4:2:0
Scan type : Progressive
Bits/(Pixel*Frame) : 0.157
Stream size : 197 MiB (90%)

Audio
ID : 2
Format : AAC
Format/Info : Advanced Audio Codec
Format version : Version 4
Format profile : LC
Format settings, SBR : No
Codec ID : 40
Duration : 22mn 57s
Bit rate mode : Variable
Bit rate : 128 Kbps
Channel(s) : 2 channels
Channel positions : L R
Sampling rate : 48.0 KHz
Resolution : 16 bits
Stream size : 21.0 MiB (10%)


Thank you a million, I'm going crazy to solve the problem!!!
Fabrizio.

[poisondeathray]

what did you do with the audio? vdub can't encode to aac , so the segments probably don't match, and that's probably throwing off mp4box, and likely why you are getting 2 files instead of appending

what does mediainfo say about your encoded intro segment?

[eymerich]

Thanks for the reply.
I extract the audio from the original mp4 file using MP4box into a separate .aac file.
Then I work on the video part and I did what i've described.
Then my original idea was to add the audio stream to the joint video stream. But I can't get the video to work (without any audio stream)!
I don't know it is the proper way to proceed, what you would do to achieve the goal?

Bye,
Fabrizio.

[poisondeathray]

It might be your mp4box version. Some are reported to be buggy. Maybe try a different build

What happens when you join the video only files with avidemux, mkvmerge, or tsmuxer? You can try dos joining as well

[eymerich]

Hi,

thanks for the help.

I tried different combinations with avidemux, virtualDub, mp4box, mkvmerge, ffmpeg
My process flow:
1) splitting MP4 (with H264, AAC) using avidemux in 2 parts into 2 MP4 files
2) ffmpeg to extract frames into PNG from the first part: GOP of 12 frames (from first I-frame to the frame before the next I-frame)
3) Replacing unwanted frames
4) encoding the 12 PNG files into a video using x264 through Avidemux. MP4 Container
4.1) The resulting video is playable and looks correct.
5) Join the newly created MP4 with the second part from point (1) using mkvmerge. It gives me this warning
<<
Warning: The track number 1 from the file 'Y:\Videos\Conversion\Test\2x09_piece_part2_video. mp4' can probably not be appended correctly to the track number 1 from the file 'Y:\Videos\Conversion\Test\2x09_beg_vdub.mkv': The codec's private data does not match (lengths: 38 and 40). Please make sure that the resulting file plays correctly the whole time. The author of this program will probably not give support for playback issues with the resulting file.
>>
And, of course, the resulting video doesn't play correct (only first 12 frames are ok)

I encoded the 12 pictures with virtualDub... same result!

I tried mp4box with -force-cat option but the result is still not working.

After changing container with mkvmerge and analyzed the 2 mkv with mkvinfo I confirmed CodecPrivate to be different...

Do I need to use a different x264 version?
What can I do?

Thanks to you all!
Fabrizio.

[poisondeathray]

What does mediainfo say about your encoded 12frame mp4 ? To append the segments, everything has to match; I suspect you used the wrong encoding settings

[eymerich]

Again, thanks for your support. I really appreciate it.

Here is mediainfo for first 12 frames (I did them with virtualdub, fourCC is avc1 like it is for the original video, the container is AVI)
-------------------
General
Complete name : Y:\Videos\Conversion\Test\2x09_beg_vdub.avi
Format : AVI
Format/Info : Audio Video Interleave
File size : 75.7 KiB
Duration : 480ms
Overall bit rate : 1 292 Kbps
Writing library : VirtualDub build 32661/release

Video
ID : 0
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L3.0
Format settings, CABAC : No
Format settings, ReFrames : 1 frame
Codec ID : avc1
Duration : 480ms
Bit rate : 1 151 Kbps
Nominal bit rate : 1 225 Kbps
Width : 640 pixels
Height : 480 pixels
Display aspect ratio : 4:3
Frame rate : 25.000 fps
Resolution : 24 bits
Colorimetry : 4:2:0
Scan type : Progressive
Bits/(Pixel*Frame) : 0.150
Stream size : 67.4 KiB (89%)
Writing library : x264 core 67 r1145bm d2e1e1c
Encoding settings : cabac=0 / ref=1 / deblock=1:0:0 / analyse=0x3:0x13 / me=umh / subme=7 / psy_rd=0.0:0.0 / mixed_ref=0 / me_range=16 / chroma_me=1 / trellis=0 / 8x8dct=1 / cqm=0 / deadzone=21,11 / chroma_qp_offset=0 / threads=2 / thread_queue=2 / nr=0 / decimate=1 / mbaff=0 / bframes=0 / keyint=250 / keyint_min=25 / scenecut=40 / rc=2pass / bitrate=1225 / ratetol=1.0 / qcomp=0.60 / qpmin=2 / qpmax=51 / qpstep=4 / cplxblur=20.0 / qblur=0.5 / ip_ratio=1.40 / aq=1:1.00
-------------------

Before joining operation I change container from AVI di mkv using mkvmerge.
The only think I spot is a different "Format profile".... but I don't know how to select "Baseline@L3.0" to match the original video... and I don't know if it can be the problem!

Bye,
Fabrizio.

[poisondeathray]

If you were using the non vfw version you would add: --profile baseline

I think you can add that to the commandline box for the vfw version

You can also try using the regular encoder instead of the vfw version

[eymerich]

Thank you,

what is the regular encoder you are suggesting me?
What software should I use to do the job?

The "--profile baseline" is not recognized by virtualDub x264 encoder...

Fabrizio.

[poisondeathray]

The regular one is the commandline version of x264 www.x264.nl

There are several GUI's for it if you don't want to use commandline e.g. megui, xvid4psp, ripbot264

Most of them will take avisynth input so you could use ImageSource() to input your stills

[eymerich]

I’m back.
I had to give up my MP4 cutting project for a while. For anyone who’s interested I found a solution. YES! Let’s summarize the task
1) I’ve got an MP4 file with a video track in h264 and an audio track in AAC.
The entire video is a TV recording. In the middle there is the program I would like to extract, call it my_prog. Since the starting and ending frames of the program are not I-frame, I need to cut the file in the middle of a GOP (between 2 I-frames). The audio can start some frames before the piece of video I’m interested in, and can end later.
2) Here is the steps I followed
2.1) Extract all frames of the first GOP (where my_prog starts) into pictures to remove or replace them
2.2) Do the same with last bunch of frames (where my_prog ends). The first frame must be an I-frame, but not the latest.
2.3) Extract the video in between (starting with an I-frame)
2.4) Extract the audio
2.5) Encode first and last bunch of frames using the same encoder used for the middle video portion
2.6) Concatenate the 3 video pieces and mux with audio

Tools and software to achieve the goal:
Avidemux, ffmpeg, wavepad (free version is enough), mp4box, mkvmerge, megui

3) Extraction
3.1) with Avidemux I seek the points for video cutting, very easy to navigate through I-frames
3.2) wavepad to trim the exact starting and ending times of audio track
3.3) ffmpeg does the actual job of extracting frames of the first and the last GOP (PNG files), video in the middle and audio track in raw format. Video is in MP4 as original
3.4) MP4box to extract middle video from MP4 container into raw format .h264
3.5) Verify cutting points of audio track with wavepad and determine audio delay if required for perfect synchronization (as original)

4) Elaborations frames: remove and substitute them so that the entire video length will be the same of the audio track

5) Encoding
5.1) MEGui to encode in h264 and output into rawavc the 2 groups of pictures (beginning and ending)
5.2) mkvmerge to join the 3 video in raw format h264 and mux them with audio track in AAC into mkv file
5.3) Avidemux to convert mkv into MP4 container

DONE!

It plays fine and audio is ok.
But if I reach the end of the video (with any player or even with avidemux) and I go back, the rendering is wrong.
I’ve checked the original video (ending with a P-frame) and go back from the last frame is rendered ok.

Any idea or suggestion?

I would like to thank poisondeathray for his very precious advices.

Eymerich

[eymerich]

Hi,

Following the steps of my previous message I obtain a MKV file with H264 video, AAC audio inside.
I would like to specify better the "wrong" behaviuor of the final (joint) file
I'm using avidemux to verify the file.
The video is made of 3 joint parts: part_1, part_2, part_3.
part_1 and part_3 are rebuilt from images using x264.
1) If I press right arrow from the initial frame, the movie plays fine. If I stop somewhere in part_2 and go back (press left arrow), everything is still ok. Once part_1 is left behind I can jump over I-frames too.
2) If I jump over I-frames by pressing up arrow from initial frame and stop somewhere and then go forward or backword (left or right arrow) the rendering is not correct (shows same frame with some corrupted block, then another one, ...)
3) If I save in MP4 format the result is the same.

Any idea of what is causing this? And how to fix it?
If it can help
- ffmpeg complains when trying to change container mkv -> mp4: ... error, non monotone timestamps 144 >= 144
- mkvmerge used to attach the 3 parts (generated in raw format) didn't report any warning.

All suggestions are welcome!
Eymerich

[poisondeathray]

Not sure, I would have to examine it.

It might be your part2 hasn't been cut correctly. The GOP structure is slightly different using x264; not all I-frames are reference frames, only I frames that are IDR frames serve as reference. It might be you excluded the IDR frame in part2

In addition, there is the stream order is not necessarily the same as the display order. ie. Displayed frames can in a different order than they are coded.

I've done similar joinings before without issues, but using avisynth, imagesource() and x264cli , joining with either mp4box or mkvmerge

You could also ask for help at doom9 forums, where many of the programmers "live". The address has changed http://85.230.118.136/

[eymerich]

Here again,

Have some more clues, I hope useful!
I have tried many combinations of the mentioned tools. And performed a lot of checking the test playback.
Actually I'm using a command line batch to run the commands. I've used the GUI only to understand what parameters should I give.
Let's work with the first 2 parts (images sequence + video)
1) Extracting frame is ok (ffmpeg)
2) Encoding them with x264 into RAW is ok.
3) The middle video sequence (part_2) is ok. Avidemux can navigate it in any manner (my validation tast!). It's made of I and P frames only. Start of course with an I-frame.
4) extraction of RAW track of part_2 with mp4bix is ok. (encapsulation back into MP4 is producing a working video again)
5) Encapsulation of encoded image sequence into MP4 with mp4box is ok. (avidemux navigation test ok)
To summarize: the 2 parts seem ok: in MP4 (mp4box) or MKV container (mkvmerge). I assume raw video track should be ok too.
The pain comes in when I join them. mp4box or avidemux don't produce a working result: stuttering when part_2 comes in.
mkvmerge joins the first 2 parts (inputs are the raw h264 file) into mkv container. No warnings nor errors.
The result is nearly ok.
The part_1 is made of 39 frames (I-frame + 38 P-frames)
On the final result (either mkv or mp4) avidemux jumps on frame 52 instead of 40 (pressing UP to jump over I-frames)
Then
a) if I advance one frame, no ok
b) If i go back (frame by frame) to hit I-frame 40, then the video plays fine backwards/forwards.
c) If I advance from initial frame to frame 40 then I can navigate on part_2 and it's fine.
I need in anyway to hit I-frame 40 (first frame of part_2) to have correct reproduction while randomly navigating (UP/DOWN/LEFT/RIGHT/END) on part_2.

If it makes sense I can upload the small video.
Maybe I made some mistakes in earlier steps that come out when joining the 2 video tracks.... who knows?!?!?

Thank you so much for all of your time.
Eymerich.

[poisondeathray]

Just 1 point: avidemux is not necessarily good for testing. It's not necessarily frame accurate. I've noticed it's quite buggy even on "normally" encoded clips (i.e. not appended) when doing non-linear seeks.

[eymerich]

Addendum

I've just discovered that frame 52 is actually an I-frame even if it is reported by avidemux to be P-frame if jumped in directly (press UP).
By stepping on frame back and forth again (right/left) the "frame type" becomes I (00)

[poisondeathray]

Addendum

I've just discovered that frame 52 is actually an I-frame even if it is reported by avidemux to be P-frame if jumped in directly (press UP).
By stepping on frame back and forth again (right/left) the "frame type" becomes I (00)

This is what I am referring to. It's not necessarily frame accurate, with non linear seek - even on a "regular" (non appended) clip

If you have progressive content, you should use DGAVCIndex for encoding which is frame accurate. To analyze, there are no good free tools for stream analyzers, but I use elecard streameye. You can have a look at the trial. It's quite expensive, but worth it if you do a lot of testing and debugging. And the encoded order is usually different from the display order.

DSS2 and ffmpegsource2 using avisynth can be frame accurate with the right container. You could use ffdshow's OSD and DSS2() to determine frame type for example. But this is still not 100% guarenteed. The only 100% accurate way is with a low level stream analyzer

[eymerich]

poisondeathray you are THE MASTER.

I gave myself a chance to understand what's wrong being less the novice in video encoding art with elecard streameye in trial mode (first 200 frames).
I've just discovered what probably is wrong, given my limited knowledge.
I found frame 40 (that's the first I-frame of part 2) to be NOT a key frame, while the rest of I-frames are.
If I open the part 2 alone (mp4 container) its first frame (that joint later becomes 40) is I-frame and IS a key frame.
I don't know why when joint with part 1 the same frame becomes NOT KEY.
This probably explains why avidemux is skipping it. It is not a key-frame!

Any suggestion?

I'm very grateful to you.
Let me use my mother tongue: Grazie.

[poisondeathray]

Just make sure you are cutting on IDR frames, not I frames

For sections that you encode yourself , you can override frametype placement and manually specify frametype with a qpfile. e.g you can say frame #67 is an IDR frame, frame #68 is an I frame etc...

Note, even when you play an uncut stream in avidemux (i.e. a normal stream), it still does the "skipping" on non-linear seeks! This means on a regular stream, if you scrub back and forth, you can lose frame accuracy.

The only way to keep frame accuracy is with an indexing application e.g. dgavcindex or streameye analyzes each frame individually. So make sure you identified your GOP boundaries correctly. Remember coded sequence can be different than display sequence - there is a toggle for this in streameye.