GOP...Pandoras Box

[monkey man]

Maybe someone can shed some light on this issue. I understand I,P,B frames and there involvement in the GOP. I understand for fast action and lot's of cuts it's good to have a small GOP. But what about quality. Does a larger GOP yield a better picture? I'm in the process of transferring a high quality S-VHS live performance to DVD. I'm trying my best to equal the source. I'm stumped on the GOP. I want the best picture possible but my lack of experience in this subject has me stumped on the best setup. I'm using TMPGE plus and DVDit to author, so I know the largest GOP I can use is 18 (NTSC). Could someone give the facts on proper GOP setup for my application.
MM

PS I have read Rui del-***** Tempe guide but it only elaborated on the function of individual I,P,B frames.

[PigOnWing]

There seems to be not too many replies so far - probably due to the subject being discussed upon over and over in the past (search the forum for GOP) - so I might as well try to step in.
However I'm stepping in from the point of 'a newcomer to another' - I'm not expert on this subject either, just seem to be on the same 'esperimenting pond' as you.

Please correct me regarding anything in this post which may be wrong, as may well be the case.

Sofar, as much as I understand, the general rule regarding GOP is that there's no general rule.
The major issue to grasp with GOP, is that it doesn't really have an effect on filesize - larger GOPs won't reduce the size, smaller GOPs won't increase the size: the file size is affected only by the bitrate. You specify how much bits to allocate per second (for example, 1150Kbits for video+224 Kbits for audio), then multiply it by the total time - that's the filesize, regardless of GOPs.

[Following section explains what you seem to already know, feel free to skip]
The idea behind GOP, is that if you'll take the idea of 1150Kbs per second, and just encode the frames like that, you'll get - we'll use dort-of-NTSC for the calculation - 1150Kbs/30fps=38.3Kbit=4.8Kbyte per frame.
Which isn't a lot.

So instead of having 30 frames each occupying <5KB, GOP is using one 'Key' frame (I-frame) - which hold all frame data - then adds to it 'doughter-' frames, which only have the 'difference from the keyframe' (P-frame), and further frames which hold only the information different between one P-frame to the other (B-frame). Like saying:

Which
*it
  s
#+?

[Feel free to stop skipping at any time]

So basically, you would care less about bitrate/fps no more, but bitrate/Gps - which is a very rough calculation.
In GOP, you'd care mostly about I-frame,s of course. They'll hold most data.

In ~30fps, using a 15-frame GOP (4 P-farmes, 2 B-farmes), you'll get 2 GOPs per second = 575Kb per GOP. In 24 fps, use a 12-frame GOP (3 P-frames, 2 B-frame) to get the same effect.

Larger/smaller GOPs will cause, at least in the calculations in my head, inconsistency in Kb allocated per GOP throughout the encoding. Same goes for 25 fps - Using 12-frame GOP will get you 2 GOPs in _most_ seconds, but once in 25 seconds you'll get 3 I-frames (note: Not complete GOP, just I-frames) in one second, dropping the overall Kb per rough-key-frame to ~380Kb.
Using larger GOPs in 25 fps will be more inconsistent.

Thing is, this whole 'consistensy' thing might really be only in my head, or in on-paper calculations. The fact that I didn't find the Holy Grail of mathematical-parsing of GOPs in 25fps, doesn't mean that it looks bad. It may mean just what it is: That mathematically, it could have looked better.

Then again, mathematically, I could have used a 5-frame GOP (2 P-frames, 1 B-frame), and it would have given me a very, very consistent 230 Kb (28.75 KByte) per GOP. I'm not too sure I'd like that.


Then again - GOP need not be that consistent. Using 'detect scene change', in TMPGEnc's settings, actually inserts an I frame whenever there's a 'totally new frame', starting a fresh GOP - and ending the previous one earlier. Again, that would suggest that some GOPs are less equal than others, and that on some seconds these GOPs may get less bitrate - but it does work, and (in theory? I never tested it otherwise, actually) does look better.

The search for the Holy Grail of GOPs - especially for PAL - continues, but alongside I begin to wonder how important is it.

-- Piggie, wondering if this kind of message fits this forum area at all.

[KoalaBear]

MPEG quality is directly related to bitrate, not GOP structure.

Does a larger GOP yield a better picture?

No. The longer the GOP, the more prediction errors will accumulate between frames, the more degraded the picture will become.

I want the best picture possible but my lack of experience in this subject has me stumped on the best setup.

The default GOP is optimal.

Sofar, as much as I understand, the general rule regarding GOP is that there's no general rule.

The general rule is that you want an I-frame to occur twice a second on the average.

GOP is using one 'Key' frame (I-frame) - which hold all frame data - then adds to it 'doughter-' frames, which only have the 'difference from the keyframe' (P-frame), and further frames which hold only the information different between one P-frame to the other (B-frame).

Uh... almost.

In the key/delta frame system, a delta frame contains the absolute difference between two source frames; the data comprising the difference is retained.

MPEG takes this a step further by forming a P (predicted) frame which tells the decoder how to shift blocks of memory around in order to construct a new frame from an old one. The delta itself is discarded. B (bidirectional) frames are similar, except their purpose is to average the difference between predicted frames. This is where prediction errors start to accumulate, so it's necessary to send a fresh I-frame at regular intervals to reset the blocks upon which those predictions are based.

The search for the Holy Grail of GOPs - especially for PAL - continues, but alongside I begin to wonder how important is it.

There is no Holy Grail of GOPs. They have nothing to do with picture quality per se. The only time they need to be changed is when the recording is intended for a particular purpose -- editing (I-frames only), strong random access (closed GOPs), etc. For all other purposes you really can't improve on the default, but where you can, the encoder will handle that process automatically.

[PigOnWing]

The general rule is that you want an I-frame to occur twice a second on the average.

Which is exactly what I was aiming at. However, that doesn't necesseraly fits with -

The default GOP is optimal.

Default GOP - in which application? TMPGenc's default GOP is an 18-frame GOP, actually. Which will never yield a 2-GOP per second; as such, it may get two I-frames per second, or it may get one.

ATI MMC's default GOP, if I'm not mistaken, is 15 frames for NTSC & 12 for PAL (which is pretty much what I'm suggestion, albeit through my lack-of-experience. I think I've seen a screenshot of Panasonic MPEG encoder where it states a 15-frame GOP, too. So 18-frames seems, rather, a little strech.

There is no Holy Grail of GOPs. They have nothing to do with picture quality per se.

Which is pretty much my sentiments. 'Searching the HOly Grail of GOPs' seems to be an issue, judging from former forum messages, that gets a hot debate every once in a while - but I did add to that exclamation of mine that "I wonder how important is it".


For what it's worth, a commrecial PAL VCD I checked has a general GOP of 12 frames. So unless someone will find such a Holy Grail that will cause a change in GOP structure to give results similar to doubling the bitrate, I see no reason to be too picky about any further adjustments.

(So why the hell am I typing so much about it? )

-- Piggie

[monkey man]

Thank you very much. I did search the forum but it seemed that all issues with the GOP and quality of the picture were skated around. It seemed that most people were only concerned with optimal compression not picture quality. Both of you answered my question right to the point. The further elaborations also answered other questions I had about GOP. My DVD player lets me know the structure of the GOP in commercial DVDs. It seemed like everyone is using a 12 (IBBPBBPBBPBB) frame structure. The performance that I am transferring has lot's of cuts and some pretty fast moving scenes. So using the scene detect setting on TMPGE along with the smaller GOP (IBBPBBPBBPBB) should work just fine for my project. After spending so much money on DVD-R production I just want to make sure I have covered all bases. Like I said in my first post I just want to equal the quality of the original performance on S-VHS, which I think should be possible.

thanks again
MM

[PigOnWing]

OK, this does interest me: Do you mean that most DVDs you checked had a 12 frame GOP, or that most PAL/NTSC film DVDs you checked had it? Or did you check only PAL/NTSC film ones, and not 29.97 fps NTSC ones? This will go actually against the 'generally, 2 I-frames per sec' statement above: 12-frame GOP will give many 3 I-Frames per sec.

Then again, as said before, maybe this is spending too much time over neglectable settings - and comparing it to DVD... well, do remember that the normal settings of 8-8.5Mbps, as opposed to 1.15-2.6Mbps we're dealing with (VCD-SVCD), makes the whole comparison, well, uh, ridiculous. We're talking about 4 to 8 times the allocated space here, for each - sec, GOP, frame, whatever.

-- Piggie, going in circles

[KoalaBear]

My DVD player lets me know the structure of the GOP in commercial DVDs. It seemed like everyone is using a 12 (IBBPBBPBBPBB) frame structure.

NTSC DVDs are almost always made from motion picture sources (24fps) which are interpolated to 30fps on playback. I'd wager that if you found a video source DVD (like the one you're making) the GOP length would be 15. But as long as the bitrate is high enough to sustain a high-quality picture, you really can't go wrong.

The general rule is that you want an I-frame to occur twice a second on the average.

However, that doesn't necesseraly fits with -

The default GOP is optimal.

TMPGenc ... will never yield a 2-GOP per second; as such, it may get two I-frames per second, or it may get one.

An MPEG recording isn't lock-stepped to the GOP, it's just a shorthand that denotes proportions between frame types. The encoder sets the actual length of each GOP according to the contents of the recording. Load an MPEG-1 file in VirtualDub and have a look at the GOP structure for different scenes. You'll see what I mean.

ATI MMC's default GOP, if I'm not mistaken, is 15 frames for NTSC & 12 for PAL

Again, the encoder isn't synchronized to the GOP. The default structure prevails only in those segments with the lowest degree of activity, where one or two I-frames per second is all that's needed to maintain the integrity of the motion prediction system.

So 18-frames seems, rather, a little strech.

Not really. Change it to 12 or 15 if that makes you feel better, but it won't make any difference. :)

So unless someone will find such a Holy Grail that will cause a change in GOP structure to give results similar to doubling the bitrate, I see no reason to be too picky about any further adjustments.

Remember the concept of "conservation of mass?" Rearrangement of matter doesn't change its weight. You can dissolve a pound of salt in a quart of water, yet it will still be a pound of salt, right?

By the same token, rearranging a GOP doesn't alter the bitrate. If there aren't enough bits to reproduce a recording without artifacts, those artifacts will remain no matter how you rearrange the GOP.

(So why the hell am I typing so much about it? :)

You're starting to put the pieces together, which is great. And one of the best ways of testing your understanding of these concepts is explaining them to others. :)

[PigOnWing]

The general rule is that you want an I-frame to occur twice a second on the average.

However, that doesn't necesseraly fits with -

The default GOP is optimal.

TMPGenc ... will never yield a 2-GOP per second; as such, it may get two I-frames per second, or it may get one.

I'd like to point out, that this special editing-out actually caused a bit of a mis-quote. TMPGEnc's Default GOP is being referred to, not TMPGEnc in general, of course. In other word, an 18-frame GOP is referred to.

An MPEG recording isn't lock-stepped to the GOP, it's just a shorthand that denotes proportions between frame types.

Yes, so noted. I quote myself, from my inital post:

[...]GOP need not be that consistent. Using 'detect scene change' [...] inserts an I frame whenever there's a 'totally new frame' , starting a fresh GOP - and ending the previous one earlier.

That is, exactly, why I doubt 'The search for the Goly Groap' has any significance. We're at total agreement here.

That said, Knowing that it has little effect, taking into account that if 'new data' (= scene change) take action, the GOP will be auto-modified, I would still like the 'General' GOP - excatly what you refer to, regarding the "segments with lowest degree of activity" - to be consistent, bitwise. Which is exactly where we - both, again - aim at a ~2 Gop per frame.

Which, however you look at it, is not 18 frames. Be the result significant or not.

All this, still sums up to what I said and you have chosen to quote -

So unless someone will find such a Holy Grail that will cause a change in GOP structure to give results similar to doubling the bitrate, I see no reason to be too picky about any further adjustments.

This does not mean I expect such a thing to surface. This is, actually, to state that I don't expect it. We do say the same thing here, eventually.

(So why the hell am I typing so much about it?

You're starting to put the pieces together, which is great. And one of the best ways of testing your understanding of these concepts is explaining them to others.

Finally, someone who understand where my horrible, annoying ditactic approach is coming from, and doesn't necesseraly goes for a show to throw at me.

-- Redundant Piggie

P.S.
Ever tried playing with multi-nested quotes? Hey, it's fun. You can burn a whole afternoon on these.

[KoalaBear]

Mr. Monkey Man got the answer he needed, so the discussion is sort of academic at this point. But I'm happy to discuss the details if you'd like.

TMPGEnc's Default GOP is being referred to, not TMPGEnc in general, of course. In other word, an 18-frame GOP is referred to.

Yes. But what's this got to do with the price of tea in China?

[...]taking into account that if 'new data' (= scene change) take action, the GOP will be auto-modified, I would still like the 'General' GOP - excatly what you refer to, regarding the "segments with lowest degree of activity" - to be consistent, bitwise.

That wouldn't be practical.

GOPs aren't used to divide bitrate among frames except in the abstract sense. I > P > B, but the frame sizes themselves aren't fixed in any way. The encoder is free to generate GOPs of any length provided (a) they are shorter than or equal to the length of the default GOP, and (b) the sum of the sizes of the frames doesn't exceed the maximum bitrate.

You could force the encoder to respect a more rigid GOP structure (say, by disabling scene detection) but in most cases you'd be damaging the quality of the recording unless you could guarantee a high average bitrate to compensate.

In practice, the only time this is ever done is when a multi-angle DVD MPEG is created. The I-frames have to be synchronized in order to achieve seamless angle switching during playback, and even then the GOPs are shortened in order to minimize latency.

Which is exactly where we - both, again - aim at a ~2 Gop per frame. Which, however you look at it, is not 18 frames. Be the result significant or not.

I don't understand. Do you mean ~2 GOPs per second? That's a rule of thumb, not a commandment. :)

[PigOnWing]

Do you mean ~2 GOPs per second?

Yup. Me bad.

That's a rule of thumb, not a commandment.

Yup again. Which is exactly why -

I would still like the 'General' GOP [...] to be consistent, bitwise.

(Underline added.

By 'General' I mean exactly for the 'Rule of thumb' - not always. Of course. the GOP could and should be shortened when needed (scene change) - but that's what I would aim generally.

I can't see how 'won't be practical' even comes into it.

That's exactly where the Tea in China has to do with everything. If we mention TMPGEnc's default GOP, which is an 18-frame GOP, we will never get 2 GOP / sec, which is what we aim as a general 'rule of thumb', which can still be shortened on a case-by-case basis, but that has nothing to do with the 'general rule of thumb'.

(Going through these issues in backward order, did I manage to convince you that for the past 3-5 posts we've been aiming to say the exact same thing? I know it's been hard for me to acknowledge, but it seems that just because you're right, it doesn't necesseraly means I'm wrong. )

[KoalaBear]

it seems that just because you're right, it doesn't necesseraly means I'm wrong. :wink: )

I see what you're saying, and you're absolutely right. But my point isn't to "prove you wrong," rather, to discuss the down-and-dirty details of GOPs for the purpose of bringing the facts into the open. I don't see that as an antagonistic, zero-sum, win/lose kind of exchange.

Let's assume for example you're 100% right: the TMPGenc default GOP is sub-optimal because it doesn't guarantee two I-frames per second, thus changing it to a length of 12 (PAL) or 15 (NTSC) is the "holy grail," the ultimate in GOP efficiency for that encoder.

Would that invalidate my point(s) in whole or in part?

Why the guy who wrote TMPGenc chose 18 as the default GOP length is beyond me. Maybe he didn't know any better; maybe he thought an extra prediction cycle would create better video at the same frame rate; maybe it was to avoid a degenerate outcome of the prediction algorithm. I don't know. What I do know is that I wouldn't change it unless I had a justifiable reason for doing so -- PAL/NTSC FILM (L=12), editing (I-frames only), segmentation or animation (I, P only), but never for the purpose of gaining a compression advantage, because it simply can't be done that way.

I can't see how 'won't be practical' even comes into it.
That's exactly where the Tea in China has to do with everything.

I gathered from your previous post you'd like to take a symmetrical approach to encoding, with roughly half the bandwidth per second devoted to the first GOP, the remainder to the second GOP.

That's just not a practical goal.

GOPs aren't designed for symmetric distribution of bitrate, they're designed for distribution of compression. There are times in which no compression (P/B) frames are warranted; there are times when the mildest compression (GOP=IPB) will suffice. To the extent you have to force an MPEG encoder to respect a particular structure, the default GOP isn't so much the rule as the exception.

A GOP of L=18 may deliver as many or as few I-frames per second as a GOP of L=15; it really depends on the material being encoded and the degree to which the encoder's choices are overridden. As to whether L=18 is best for TMPGenc, yeah, I'll err on the side of the designer and say that's optimal for his encoder.

Again, that doesn't mean you're right or I'm wrong or vice versa; it's a simple exchange of ideas.

Right?

[PigOnWing]

But of course. That's the main point. Just wanted to express it, to make sure.

And now, to business.

I gathered from your previous post you'd like to take a symmetrical approach to encoding, with roughly half the bandwidth per second devoted to the first GOP, the remainder to the second GOP.

When there's no need for shorter GOPs (no scene-change, no high motion, no nothing). Only in the scenario of plain, simple movement, which won't alert the encoder to outbound the default, general GOP structure.

What I do know is that I wouldn't change it unless I had a justifiable reason for doing so -- PAL/NTSC FILM (L=12), editing (I-frames only), segmentation or animation (I, P only), but never for the purpose of gaining a compression advantage, because it simply can't be done that way.

(Both my underlines)
Regarding the second underline - I agree completely, I hope I never suggested otherwise. If I did, that was a slip of the keyboard - my whole inital approach was to explain that this is totally irrelevant in regard to the compression (compression, being filesize, that is, allocated bits per sec. It does affect the compression of I-frames - shorter GOP = more I-frame = lesser bits per each. That's not much gaining or losing compression as quality - I think this is another point we agree on).

But regarding the first underlined part - ah, but here it lies. If you won't change it unless you have a good reason to, then such a good reason can exist.

Moreover - if you supply such a reason as an MPG in PAL/NTSC film, and state the GOP you'd change it to as a 12-frame one - going with the 'rule of thumb' we both use, as '2 GOPs per sec' - according to that, logic would say to have a 15-frame GOP for NTSC.
Which is pretty standard, according to what I see other encoders use.

But -

As to whether L=18 is best for TMPGenc, yeah, I'll err on the side of the designer and say that's optimal for his encoder.

If so - why change it for PAL/NTSC film? The PAL/NTSC film templates still use an 18-frame GOP.
If changing it in these templates - why not change it in NTSC?

For some time, I understand that the default templates provided with TMPGEnc aren't being referred to as a 'Holy Grail' templates for VCDs/SVCD - on the contrary, one of the first suggestions people get when they say "I encoded a VCD using the default template and it sucks" is "play with the settings, the default template isn't optimized". The GOP sturcture is a setting, stored in a template - it has nothing to do with 'default' settings for an encoder.

Taking almost each other component in the templates as 'something to tinker with', why not the default suggested GOP? Especially when it goes against a 'general rule of thumb' that we both seem to like...

-- Nagging Piggie

[KoalaBear]

I'll level with you, Mr. Piggy:

I think you're less interested in an informative, mutual discussion of GOPs than in preserving your opinion to be "correct." That's okay, but remember that the only thing that can reliably said about a man who believes he's a Denver Omelette is that he's in the smallest possible minority. :)

But regarding the first underlined part - ah, but here it lies. If you won't change it unless you have a good reason to, then such a good reason can exist.

Mr. Piggy, am I wrong, or did you earlier say --

Please correct me regarding anything in this post which may be wrong, as may well be the case.
Sofar, as much as I understand, the general rule regarding GOP is that there's no general rule.

The general rule is that you want two I-frames per second. That's all. The rule can be overriden according to taste or circumstances, but it doesn't cease to be a rule simply because it has exceptions.

one of the first suggestions people get when they say "I encoded a VCD using the default template and it sucks" is "play with the settings, the default template isn't optimized".

Mr. Piggy, I apologize for other people's ignorance, but I can't take responsibility for your gullibility. Believe what you like, and perhaps when you're tired of chasing your tail we can talk about other things, yes? :)

Taking almost each other component in the templates as 'something to tinker with', why not the default suggested GOP?

Go right ahead. In fact I encourage it. When you discover something that breaks the laws of physics as we understand them, do let us know, mmm'kay?

Take care.

[PigOnWing]

Of course not. If I'll find something like that, I'll sell it to the highest bidder and make millions. :P

And now to business.

First, yes, I'll have to admit, I do sin occasionally - sometimes - ok, frequently - while thinking that I actually understood something, in trying to get 'approvals' from other people, who already know about this subject.

But the idea isn't just to "preserve my opinions no matter what", but to make sure that I indeed have it right - and if not, to understand why.

Please correct me regarding anything in this post which may be wrong, as may well be the case.

- But once corrected, I want to know the ins and outs of the whys. Moreover: As we already agreed that there isn't a 'right or wrong' here, but that it's merely a subjective case-by-case basis, I still want to know the reasoning behind each such opinion. The question of whether I'll accept or use it for my own encoding, is irrelevant: that's the part that is subjective.

So, at the end, after agreeing on the 'exceptions on a scene-change case', and after we agreed that 'a rule of thumb isn't a Holy Grail', and after agreeing that it won't have a major impact on the quality, we still have it -

The general rule is that you want two I-frames per second.

And it seems we both, generally, like that rule-of-thumb.

Based on that rule of thumb - a [General] 12-frame GOP for PAL/NTSC film, I can understand; a 15-frame GOP for NTSC, I can understand; an 18-frame GOP - I don't.

I'm not suggestion it's 'Wrong'. I'm not suggesting it will yield worse or better results one way or the other. Heck, I didn't conduct testings to claim either way.

All I'm saying, is that I don't understand it. Theoretically, not based on 'I actually got better/worse results from this or that'. Based simply on the 'rule of thumb'.

So yes, on one hand, you could say that I'm 'trying to get people to tell me I'm correct'. But on the other, you could say that I just want to understand the reasoning behind the different settings - all different settings - regardless of them being correct, wrong, both or neither.

-- Piggie, who always fears that his tone passes much more agressively than intended

P.S.
I've had this post phrased so much better yesterday. Then lost it to a network failure. Darn.

[Nelson37]

Interesting discussion. I was told that using a GOP with fewer B frames, as in "IBPB" as opposed to "IBBPBB" would result in lower quality due to the number of high-quality frames for a given bitrate. However, does it not follow that if the bitrate is high enough, more accurately representing a higher number of quality frames will give a Better Video?
I find that if I have a clean original, at bitrates from 2000 to 2400, using fewer B frames (and therefore more I and P frames) looks better to me, for DVD playback.
Also, although I know bitrate determines all, encoding same 600MB clip at IBBPBB and IBPB gave about a 3MB difference (4pass CCE), possibly slight averaging difference, but definitely looks better. Smoother motion, better picture quality.