How to get ffmpeg/x264 to use same parameters as input?

[Gyan]

I'm trying to accomplish a smart rendering workflow using ffmpeg/x264. In short, get ffmpeg to segment the input file. Locate the segment(s) with changes to be made. Re-encode altered segments with same parameters as source. Concatenate altered + original segments.

Problem: the resulting video turns garbage when it first traverses from an original to altered segment or vice-versa. Most likely, because the altered chunks and original chunks differ in some bitstream parameters, or possibly some GOP boundary issues are at play.

So,

1) is there a way to tell ffmpeg/x264 to generate a concat-with-source-compliant output?

2)If it's GOP related, what do I look for in the source and what do I tell the encoder?

Thanks,

Gyan

[Selur]

1) is there a way to tell ffmpeg/x264 to generate a concat-with-source-compliant output?

No ffmpeg has no such option.
What you want requires some analyzing of the source (and even then it's probably more a guessing than exact science unless you are really thorough with your the analysis)

2)If it's GOP related, what do I look for in the source and what do I tell the encoder?

Source needs to be cut into GOP segments. (remember not I-frames but IDR-frames are important here)
The GOPs which are partially needed then need to be re-encoded and later merged with the input.

Here's roughly what I do in MKV Merge:

1. analyze the source to figure out where are the IDR frames
   This can be done using ffprobe or, if the source if raw content or demuxed, with h264_parse. (there are also other tools which can do this)
2. let the user specify where he wants to cut the content
   Combined with the input from 1. this gives the GOPs which can be dumped, the GOPs than can be kept without modification and the GOPs which need to be partially reencoded
3. cut the content and drop the unneeded parts
   This normally can be done for audio&video in one go. (interlaced or vfr video can be tricky, subtitles might need to be processed separately)
4. Now the audio and video (and other content like: subtitles and other attachments) need to be separated and analyzed. In regard to what settings need to be used for reencoding. (Normally the audio can be simply cut, subtitles are tricky and attachments need to be kept)
   To analyze the content one can use:
   a. MediaInfo (should be enough if the content was already encoded with x264 and contains a SEI with encoding setting)
   b. h264_parse (can only analyze raw H.264 content)
   c. ffmpeg (needs to be run with very verbose output and is a pain to interpret, but might be better suited for exotic containers)
   (often all three are needed)
   and come up with x264/ffmpeg settings to create a compatible stream. (interlaced/telecine content can be really tricky, SPS/PPS groups might be a problem) Be aware that x264 only can create MBAFF and not PAFF content. Also if your content is vfr, this whole thing can end up really problematic.
5. once GOPs which needed to be partially re-encoded are re-encoded their additional streams audio, subtitles,... need to be remuxed to a new container
6. once these parts are back in a container, the kept GOPs and the re-encoded parts need to be rejoined
   Depending of the content and the container this can be done with ffmpeg, mkvmerge, MP4Box, tsMuxer or similar tools.

Out of curiosity:
Are you planning to write a (freely available) gui for this? Or is this meant to end in some (batch/terminal) script? Are you planning to release the source code of what you are doing to the public? Is this a commercially or privately motivated project?
Assuming you are planning to write real program:
a. Are you planning to use libav&co directly or are you just using existing tools like ffmpeg&co?
b. What programming language do you plan to use?

=> All in all the general goal isn't hard to archive (btw. this was the motivation behind me writing MKVCutter a while back), the problem is to do this properly and handle cases where the content is interlaced (normal interlaced, telecine, mixed content) and has a vfr video stream. (subtitles, attachments and exotic audio streams can also cause some problems)

Cu Selur

Ps.: Unless you have some basic understanding about H.264 raw data structures you might want to read up on that or you will probably run into real problems when you start trying to figure out what settings are needed to match the input content.

[Gyan]

Hi Selur,

Thanks for your reply. Seems to clear some things up.

A few questions:

How do I identify IDR frames in ffprobe output? I see 'key_frame' (1/0) and 'pict_type' (I/P/B) fields. I do not see any 'IDR' string. In the file I'm working with, GOP is M=2, N=90 (fps=30.12). So each segment that ffmpeg generated (X0,X1,X2...) is exactly 90 frames long i.e. 1 GOP. And each of these segments contains exactly 1 keyframe, which is the very first video frame. Now I want to force a keyframe at frame 41 of X0, chop off from that frame, and merge with X1, X2 onwards. Since the video is coming garbled when it reaches X1 portion, I think it's because the decoder doesn't treat first frame of X1 as an IDR frame. How do I force that?

Much thanks for your help. This is not for any programming project. I'm only developing the method for my use as a video editor, since I currently will be starting a project where I'll need to trim many videos without re-encoding as far as possible. So this technique will be used when the trim point is not a keyframe. If it works, it's a better and quicker option than re-encoding the whole trimmed portion.

Gyan

[smrpix]

There are commercial products that do this quite well, among them videoredo and TMPGencMPEGSmartRenderer. If you're doing a lot of this it may be worth the investment.

[Selur]

How do I identify IDR frames in ffprobe output? I see 'key_frame' (1/0) and 'pict_type' (I/P/B) fields. I do not see any 'IDR' string.

key-frame would be the interesting part (assuming ffmpeg can differentiate between I and IDR frames)
Unlike other MPEG-based formats I frames do not have to be key-frames for H.264.

In the file I'm working with, GOP is M=2, N=90 (fps=30.12). So each segment that ffmpeg generated (X0,X1,X2...) is exactly 90 frames long i.e. 1 GOP.

No, the maximum GOP size used is 90 frames, there might be shorter GOPs. That is why you have to collect the key frame positions.

I think it's because the decoder doesn't treat first frame of X1 as an IDR frame. How do I force that?

No that's because the first frame isn't a key frame and thus refers to other frames and if you cut before that frame references get lost and thus the frame can't be decoded properly (the decoder assumes it is a key frame ) and you end up with garbled content.

Cu Selur

[Gyan]

The issue is that all the I type pictures are 90 frames apart, so I guess it's fixed GOP. And all the I type frames are flagged as key_frame=1 including the first frame of X1.

smrpix: I will consider videoredo if this doesn't work out.

Gyan

[Selur]

The issue is that all the I type pictures are 90 frames apart, so I guess it's fixed GOP

Sure, my point was that this does not have to always be the case.

[Gyan]

Yeah, I know that GOP length can be variable; I was just clarifying for this particular file.

But, like I said, first frame of X1 segment is flagged as a key_frame and yet the video is garbled from that point onwards, even when it gets to other segments. What could be the issue?

Gyan

[Selur]

a. source might be corrupt
b. something went wrong during cutting
c. ffprobe did mistake an I frame for a key frame

[Gyan]

b has to be the answer, if it's one of these three choices, because the individual segmented file X1.mp4 plays fine on its own. As do the other segments. But I can't see what went wrong during cutting.

[Selur]

But I can't see what went wrong during cutting.

Since you haven't posted any details about what you are doing neither can anyone else.

[Gyan]

Sorry, thought I had linked to my thread at ffmpeg forums:

http://ffmpeg.gusari.org/viewtopic.php?f=11&t=1793

[Selur]

a. I'm not sure '-f segment' works the way you think it works.
b. 30.12 fps might be an indication that your content is vfr, which might cause additional problems
c. there is no clue why you choose the x264 settings you did

-> I doubt the segmenting and concatenating will work the way you intent it to.

For the fun of it you could remux the content to mkv, then cut it using MKVCutter and check if the output is okay.

[Gyan]

a. I have checked 3 or 4 segments with ffprobe and each of them reports the first frame (and only the first frame) as key frame = 1. And of I type. And there are no other I frames within each segment.

b. Mediainfo reports "Frame rate mode" as constant for the source X.mp4.

c. I chose the x264 settings because the source is profile Main, Levels 3.0, CAVLC coding, and Mediainfo reported ReFrames as 2. Whereas without those parameters, ffmpeg outputs profile High, CABAC, ReFrames 4. I'm just trying to match source/output parameters in case that's the problem.

Gyan

[Selur]

. Mediainfo reports "Frame rate mode" as constant for the source X.mp4.

MediaInfo only checks the first few time codes. To be sure all time codes have to be checked.
But vfr is probably not the problem of broken reencoded content.
I still think the problem is the segmenting part, but may be the folks over at the ffmpeg forum can tell you more.