Is IBPBPBPBPB the best all-purpose GOP structure?

[SaSi]

BJ_M,

Interesting finding in the GOP structure. I never thought of checking this before. Will do.

The setting you say reads VCSD (??).

Rgds

[BJ_M]

correct

[Sulik]

I just want to put my two cents in, as there is a lot of mis-information in these threads:

1. A low-delay stream can't have B-frames, BUT a stream without B-frames is NOT automatically a low-delay stream (there are many other restrictions that go with low_delay=1). DVD streams with no B-frames are totally legal.

2. Scene changes on B-frames are not that bad, just not very efficient. How it affects the rate control of the encoder is another story. Even if your scene change happens on a B-frame, the B-frame can still reference future frames through backward prediction. Also don't forget that a P or B frame can be entirely intra-coded, just like an I-frame, and doesn't make such a big difference at high bitrates.

3. IBBP vs IBP is debatable. There are trade-offs: having too many B-frames also means that the reference frames will be further apart from each other, resulting in a loss of compression efficiency for P-frames.

Again, to test this for a specific sequence, simply encode with both GOP settings using CBR, and pick whichever one has the lowest average quantization and/or highest PSNR.

Personally, I use ATI MMC with the standard NTSC GOP setting 0.5s, IBBP.
Recent versions of MMC are above and beyond older 7.x versions (I actually get lower Qps than with tmpgenc in many cases).

MMC can be buggy at times, but I have yet to see anything that produces better real-time captures for under $300 (yes, I looked at the happauge and just because the encoder runs on a DSP and doesn't use any CPU doesn't make it a good encoder)

[aamir12345678]

Just a quick question. Why is it necessary for you to mess with the GOP structure? If you have more high-motion scenes with a particular video, why don't you just up the bitrate? Furthermore, unless you use CQ or CQ-VBR, won't changing the GOP structure actually degrade quality? In a 2-pass VBR or CBR, you have a fixed bitrate that is being achieved. If you decrease compression by decreasing the # of P or B-frames in a GOP, you'll need a higher bitrate to achieve the same quality. Am I seeing this right? What's going on exactly?

[BJ_M]

i'm not sure to whom (or which post) the last two posters are adressing because nether statement fits in to the rest of the thread as to it's content or are disputed ...

[aamir12345678]

My question was asked to understand which GOP structure to use.

[BJ_M]

aamir12345678 --

yes you are right , for example an all I frame requires a higher bit rate

[aamir12345678]

BJ_M, then what's the advantage of using only I & P pictures with high bitrates like 7000-9000. Why do people suggest that? Is there any benefit to it?

[lordsmurf]

BJ_M, then what's the advantage of using only I & P pictures with high bitrates like 7000-9000. Why do people suggest that? Is there any benefit to it?

Editing.
High action.

[petar]

BJ_M, then what's the advantage of using only I & P pictures with high bitrates like 7000-9000. Why do people suggest that? Is there any benefit to it?

Editing.
High action.

True that a GOP without B frames is easier to edit but MPEG2 is meant to be a final product and one shouldn't encode thinking about editing it afterwards. Quality should be the first concern.
High action may benefit but it's debatable. Each frame is 1/25 (1/30 for NTSC) of a second. How much can the P picture benefit by referencing the previous frame than one that's 1/25 of a second away? On the other hand you're spending more bitrate (P use more bitrate than B) which means your compression is less effective.

Aamir12345678, if you want my advice, use the the default GOP structure. A good encoder will modify it if required by a high action scene. The DVD designers decided to use the MPEG2 Main Profile for a reason. I'm sure they exploited all posibilities before they made the decision, having in mind the max bitrate and the DVD size. The MPEG2 Simple profile is the same but it doesn't use B pictures and if the DVD image would have benefited from having no B pictures they would have decided to use it instead.

MPEG without B pictures has its use. Decoding B pictures is very resource-consuming on the decoder (the way an IBBP GOP is decoded: first I picture, then P picture, then back to decode the B pictures, then the next P picture...) and it's not possible in some situations.
As part of my Master degree I had to do a research on MPEG2 and I remember that a major obstacle DVD had at the time (I'm talking some 10 years ago, no DVDs were available yet) was that the PCs has no "power" to decode it (I think the PCs topped at Pentium 120Mhz at the time and even with a hardware decoder you needed at least a P90). However that's not the case anymore so there's no reason why you wouldn't use B frames in a DVD.

Quality is a subjective topic. As such I'm sure many people find that the image looks better to them when they modfy this or that but the theory behind MPEG2 (or MPEG1) is that B pictures should provide better overall quality of the video.

[BJ_M]

BJ_M, then what's the advantage of using only I & P pictures with high bitrates like 7000-9000. Why do people suggest that? Is there any benefit to it?

Editing.
High action.

True that a GOP without B frames is easier to edit but MPEG2 is meant to be a final product and one shouldn't encode thinking about editing it afterwards. Quality should be the first concern.
High action may benefit but it's debatable. Each frame is 1/25 (1/30 for NTSC) of a second. How much can the P picture benefit by referencing the previous frame than one that's 1/25 of a second away? On the other hand you're spending more bitrate (P use more bitrate than B) which means your compression is less effective.

.

thats simply not true -- i.e All new sony and other digital formats in the same vein (IMX, MX, TS and others) are in fact mpeg2 types and made for editing ...

you are also forgetting about how a mpeg file works in terms of playback ...

[petar]

thats simply not true -- i.e All new sony and other digital formats in the same vein (IMX, MX, TS and others) are in fact mpeg2 types and made for editing ...

MPEG2 was never designed for editing in mind (except for video sequencies coded only with I pictures, there was not even a requirement for full random access, never mind editing). But, as the technology advances and hardware gets more and more powerful, it's easier and easier to edit it, so it's true that editing mpeg2 is not a big deal these days.
However, I believe the topic of this thread was regarding the best quality with PC encoding hence not using B pictures to get better editability should not be a priority.

[SaSi]

The whole issue with optimizing GOP structure vs bitrate and encoding method (CQ vs VBR) just boils down the encoder (s/w) quality and "cleverness".

Take a drastic example: A video of still countryside (picture yourself a hillside with green trees and tall grass) lasting 1 minute. Plain still nature. This doesn't require a high bitrate.

Now, imagine that in the middle of this video clip, a fast horse runs along from left to right covering almost a quarter of the screen in size. The horse takes 2 seconds to run from side to side.

The previously still video that could be encoder with good quality at under 5Mbps CBR, suddenly will suffer low quality during the crossing of the horse.

This means that the horse sequence requires a higher bitrate.

The question is: Will the MPEG Encoder s/w or H/W recognise this need and will it recognize the full extent of the need to kick the bitrate up to (say) 8Mbps while encoding the horse?

I've seen that Mainconcept is a bit "Bitrate shy", even at 2pass VBR. Encoding such sequences with 2passVBR @ 0/3000/8000 it will very seldom use up the max bitrate.

On the other hand, there is the issue of clever spending the bitrate. The heart of the compression is practically motion search, which minimizes the content of the B and P frames allowing for a low bitrate budget.

But since motion search takes time, we tend to use either moderate settings (and complain about quality) or extremely high settings (and complain about overnight encodings.

[BJ_M]

Mainconcept is a bit "Bitrate shy" is true , so is procoder , tmpgenc goes the other way and spikes the data rate, CCe is not bad but requires a higher low value (pretty well i stick with CQ with cce almost always and mastering quality with procoder and settings all over the place with MC depending on the project) .. MC also likes a higher low value though it doesnt always abide by it ..



procoder 1.5 has a bug where it makes to long a gop once in a while and a few other weird things when you make a lot of changes to its presets. thought i would throw that in as it really pissed me off today.

field based encoding works better on interlaced material than frame based , but supposed that some dvd players dont like this, this setting is in mc and procoder. i woudl liek to know what players do and do not support field based encoding ...

[flaninacupboard]

it was my understanding that this was an "old" bug on the first DVD players whereby they decoded in the wrong colourspace when faced with a frame based file. i'm sure i remember seeing some shots somewhere, you ended up with lines running through your material.

Actually, perhaps this was limited to PAL progressive scan?

oh, just ignore me.

[SaSi]

I 'd like to come back to the GOP structure topic with some recent test results.

I took 1,500 frame long clip of a formula 1 race capture. The content is almost one lap round the track, with some close shoots and lot's of panning. The frame size is 352x576@25fps (PAL)

I encoded with Tmpgenc with CQ=60 and various GOP formats. The results are below:
GOP Structure............Av.Bitrate...I-Frame....P-Frame....B-Frame
IIIIIIIIIIIIIII..............5,960kbps..30,000kB
IPPPPPPPPPPPPPP.......5,605.........72,500.....25, 000kB
IBBPBBPBBPBBPBB.....4,729.........63,500.....45,00 0......11,300kB
IBBBPBBBPBBBPBBB...4,163.........58,400.....47,000 ......11,600
IBBBBPBBBBPBBBB....3,796..........52,000.....47,00 0......12,700
IBBBBBPBBBBB..........3,385..........45,000.....46 ,500......12,600

Sorry about the format of the list, can't seem to achieve a tabular format.
The interesting part is that the more B frames I used the lower bitrate I got. And all the clips were encoded at the same quality.

I used the last one (IBBBBBPBBBBBP...) version and made a DVD with DVD-LAB (50 sec long). It played fine on a Denon DVD player.

So, it appears, the more B frames I use, the best compression ratio can be achieved at no apparent expense to quality.

One previous posting indicated that increasing the number of consecutive B-frames would cause them to inflate in size, countering the compression effect.

Well, their size increases, slightly, however the overall effect is a constant drop in bitrate requirement when more B-frames are added.

[flaninacupboard]

That's very interesting. did you encode at CBR with those GOPs to check for any quality difference?

[petar]

Very interesting indeed. SaSi, have you tried the same with a full 720x576 resolution? The reason I'm asking is that with half resolution you have plenty of "spare" bitrate. Doesn't matter how far the B and P pictures are from their references, if you have enough bitrate to give them, they'll be fine.

Increasing the number of B frames will always lead to better compression, therefore more efficient use of bitrate, therefore more bitrate for the whole video, therefore better quality (or same quality with smaller size). But you are bound to hit a limit for a given bitrate. If you look at your example when you put 5 B frames in-between, the P picture is so far from the I that it uses the same bitrate, i.e. it doesn't have much to reference to.

The I picture is an overall (direct or in-direct) reference to the whole GOP. The fact that you didn't notice any quality loss in the last example suggests that you still have enough bitrate so the I picture quality doesn't suffer.

It would be interesting to repeat this test for full DVD resolution or half the max bitrate.

[SaSi]

That's very interesting. did you encode at CBR with those GOPs to check for any quality difference?

Didn't do any CBR because quality is so subjective I don't even trust myself with deciding between two clips.

My idea was to "let" Tmpgenc make Constant Quality encoding, so that it tried to keep as much as it wanted do. The more quality the more bitrate.

Unless, ofcourse, it doesn't have to.

I am doing a test with full PAL resolution and will post the numbers shortly.

[SaSi]

Some newer info from tests, for anyone still following this. I'll try to make it short and clear.

1. The more B frames used in encoding, the better compression we can achieve for a given quality. In general - Some encoders are better than others in this and provide a more substantial saving in file size

Therefore, for encoding in Constant Quality, both with Tmpgenc and MC, the file sizes were smaller in this order (first is larger):
IIII...
IPPP...
IBPBPBP...
IBBPBBPBBP...
IBBBPBBBPBBBP...
IBBBBPBBBBPBBBBP...
IBBBBBPBBBBBPBBBBBP...
Using more than 6 B frames in a row seems to cause negligible savigns in file size.

2. The best GOP structure is probably IBBPBBPBBPBBP... (as if the development committe didn't know what they were saying)
Why?

As the B frames must be decoded after the "following" P frames are loaded and decoded, the more B frames we use, the more time it takes for the decoder to retrieve the P frame, decode it and then go back to the "preceding" B frame(s) and decode it (them).

Although my PC seems to have no trouble decoding almost any stream type, my standalone DVD player showed an increasing flickering --> struttering as the B frame number increased.

So, I'm personally sticking (returning) to IBBPBBPBBP... with variable GOP size.

3. For MainConcept users, try the following encoding settings:

- Select 2 Pass encoding,
- In Advanced settings, in the GOP structure section select VCSD
- In advanced video settings, in Motion search pixel movement Enable motion search and select 12 for horizontal and 6 for vertical.
- In motion search mode try 8
- Finally, check the "User Quant Matrices" and edit them. Use the values provided by CCE 266/267 for Very Low Bitrate.

These settings gave me a very good tradeoff between speed and quality. Even in low bitrates (2800~3200kbps for Full D1 PAL) the picture was not showing the noisy and pixelated effect that Mainconcept typically generates.

For Tmpgenc users:

Try checking the "Force Picture Setting" checbox in the GOP Structure settings and then scan the video source to mark scene changes with a sensitivity of 150. It will help the encoder create I-Frames in these places and significantly reduces pixelation in scene changes happening during high activity parts.

[adam]

I woudl liek to know what players do and do not support field based encoding ...

My Apex 5131 doesn't. I get black picture unless I'm ff'ing or rw'ing...really weird. Sucks too because DV looks substantially better to my eyes with field based setting in Procoder.

[flaninacupboard]

That's plain weird, do you have any commercial discs that cause the same problem?

[BJ_M]

yea - that really does suck ....

it DOES look better ....

its hit or miss on what can play it back -- but it seems for compatability , frame based stays ..

[adam]

flaninacupboard

We're tallking about field based encoding, not just using field pictures, aka an interlaced stream. Field based encoding isn't supported in the DVD standard, and so like BJ_M said its a hit or miss with compatibility. My Apex misses.

With interlaced sources field based encoding looks MUCH better. Procoder can actually switch back and forth between field and frame based encoding depending on the source, but alas....

[ronnylov]

I have read somewhere that you need Procoder 1.5 to encode field based because the earlier versions had some bug. Then you must use full DVD resolution (like 720x576 PAL) and use top field first setting. If you are encoding DV which normally is bottom field first then you must change the input to top field first (can be made with avisynth) and encode it with top field first field based encoding in Procoder 1.5. No other resolutions than 720x576 worked on the standalone players with top field first when encoding field based. With this settings quite many DVD players could play it without problems.

I haven't tried it myself yet.