TEST: Multi-pass VBR vs. CQ VBR

[therick]

Alright, here we go...

I know many of you on this board followed the "Battle of CBR/VBR/CQ Modes"...I know I did and I even got involved at one point, but then stepped to the side and let people who know more than I duke it out. The main reason for this post and for the tests (which you will see below) was simply my own curiousity. I have not been on the board that often lately, but what I did take before I left was a lot of info about capturing from different sources, and that Tmpegenc was for MPEG-1 and CCE was for MPEG-2.

This is a simple test between ONE mode of encoding, VBR-all of these tests created VBR streams. I have tried to make it as simple as possible. Three METHODS of VBR encoding were tested:

TMPEGEnc's CQ mode (MPEG-1/kwag's 2 CD template)
TMPEGEnc's 2-pass mode (MPEG-2)
CCE's 2-pass mode (MPEG-2)

Now the reason I chose these three is simple-

1. I wanted to see if there was something to kwag's new GOP method for "near-dvd" quality.
2. I wanted to see if TMPEGEnc's 2-pass VBR encoding was still as slow and as terrible as it was when I switched to CCE for VBR MPEG-2 streams (back around the 2.0 release).

Now I plan to run these tests a little differently than other tests. I am not going to interpret the results. I ran bitrate viewer ONCE...to get the average from the CQ encode to get the average for the 2-pass encoeds. I am not going to post picutures of bitrate viewer or say that one file was better than the others. I will leave that to others.

Some of the argument over previous tests had to do with source material. The source for this test was the X-Men DVD, ripped with Smartripper and frameserved via DVD2AVI. So the source is excellent.

Here are the applications I used:

Smartripper
DVD2AVI
AVISynth
TMPEGEnc 2.56
CCE 2.5

My computer specs are listed on my profile-you will notice that I am by no means breaking any encoding records. Only an Athlon 950 MHz and 512 MB of RAM.

Now as for the clip I chose. I chose the scene early in the movie where Logan picks up Rogue in Canada. It seemed like a good clip to use, there are slow head shots, tight close ups, and fast action. The clip is right under 3 minutes (2 min 45 sec/3977 frames) and principally I thought that three things in the clip would test the encoders(and modes) well:

1. The snow that Storm flings at Sabretooth.
2. The fire that we see start in the back of Logan's winnie.
3. The explosion of Logan's winnie.

Here was my method-I used AVISynth to frameserve the DVD2AVI project file to TMPEGEnc and CCE.
Here are the scipts:

for tmpegenc:-------------------------------------

LoadPlugin("MPEG2DEC.DLL")
mpeg2source("xmen.d2v")

Crop(8,60,712,420)
CropBottom(60)

BilinearResize(704,240)
AddBorders(0,120,0,120)

trim(25921,29897)

#ReSampleAudio(44100)

and for cce:--------------------------------------

LoadPlugin("MPEG2DEC.DLL")
mpeg2source("xmen.d2v")

Crop(8,60,712,420)
CropBottom(60)

BilinearResize(704,240)
AddBorders(0,120,0,120)

trim(25921,29897)

ReSampleAudio(44100)

---------------------------------------------------

So both encoders (and all three modes) had the same, DVD qulaity source.

Now, before I get to the creation of the streams, they are all 704x480 (for comparison to kwag's CQ method), run at 23.976 fps, and have NO audio.

Alright, here is how my encodes went:

The first encode used the above .avs file into TMPEGEnc with kwag's 2 CD template loaded. For details that is an MPEG-1 encode in CQ mode with the Q at 70. I simply loaded up the template, so all the defaults were in(min=300/max=2300). The encode took approximately 14 minutes and 46 seconds. The resulting file was 26.4 MB.

here is the file: it's name is : xmen_test(kwagTmpeg).mpg

and it is located on my ftp site:

robbins.dns2go.com l/p : vcdhelp/vcdhelp

***Now this is the one time in the experiment that I used bitrate viewer. I got the average from the CQ file and exited. The average for the clip was 1238 kbps.

The second encode used the above .avs file into TMPEGEnc with 2-pass VBR MPEG-2 NTSC-Film settings and identical settings to the CQ encode (300-1238-2300). The encode took 21 minutes and 12 seconds and the resulting file was 24.9 MB.

here is the name of the file: xmen(Tmpeg2pass).mpg

and it is located on my ftp site:

robbins.dns2go.com l/p : vcdhelp/vcdhelp

The third encode used the above .avs file into CCE with 2-pass VBR MPEG-2 settings and again, identical bitrate settings to the CQ file- min. 300, max. 2300, and an avg. of 1238. The creation of the .vaf file took 4 minutes and 2 seconds and the transcoding took 3 minutes and 58 seconds, so that is 8 minutes even for the total encode time and the resulting file was 24.5 MB.

here is the name of the file: xmen(CCE2pass).mpg

and it is located on my ftp site:

robbins.dns2go.com l/p : vcdhelp/vcdhelp

So there are the files for your analysis...but I am not done yet. I KNOW the two multi-pass files are smaller than the CQ file. I am doing the math right now to bring the averages up so that they are the same size as the CQ file. Hopefully I will be able to post links to them tomorrow.

Please post any problems that you have with my methods-I do not claim to know everything (which is why I have decided to leave the evaluation of these results to others). I will gladly run more tests(within reason, of course) if they are necessary or will help clarify the issue.

I realize that quality is subjective...but I personally would like to see these results evaluated by virtualis, kwag, adam, Martyn1980, anyone who wants to.

Thank you and good night.

[injunpana]

As far as I'm concerned your method was set up fairly. As to when to compare the results, now there is the dilemma.

Do we compare the results now with them all using the same min-ave-max bitrate parameters even though they don't have equal filesizes? If we do this, we would be breaking one of the parameter constraints which would be they should all be of equal filesize.

OR

Do we adjust the minimum or average or maximum bitrate for the TmpGenc and CCE VBR encodes until the filesize equals that of the CQ? In this case the filesizes would be the same but we would be breaking one of the other bitrate parameters.

I'm afraid it'll be hard to get a fair comparison unless the three encoding methods all happened to produce the same filesize with the same min/ave/max bitrate constraints.

[adam]

Personally I say judge the clips by their size, so adjust the avg to match the CQ file. The reason I say this is because encoders are inherantly slightly inconsistent so the final output size will always be a little off.

The CQ file is obviously impossible to predict exactly, but if it were "as" predictable as x-pass vbr, well then the resulting filesize would still be a little off. The fact that its size is larger is simply a result of it being encoded first and the other encoding modes not being able to match it exactly, so if you were to give the disadvantage to cce's or TMPGenc's 2-pass vbr than that would be penalizing those modes due to CQ's unpredictability. Also don't forget that bitrate viewer may be incorrectly reporting the avg, though it would be easy enough to figure it out on your own.

The test here is to determine the quality differences between CQ and x-pass vbr, with cce thrown into the mix. Since the quality is largely determined by the total amount of bits used during the encoding I think this should supercede all other factors.

Ultimately, I don't think that slight size discrepancy is going to make that much difference but since I'm on dial up I'm going to wait for the next round of samples.

I do think another test at higher bitrates would be good too because from my experience cce doesnt perfom nearly as well at ~1.2mbits as it does at 2.5mbits or even high bitrates like 5-6mbits, compared to TMPGenc of course. How about it therick, you up for it? I like your tests, they are simple and fair.

[adam]

Wait a sec, I just re-read the original test and I have a big problem with it. The CQ encode is mpeg1 and the other two are mpeg2...sorry but I don't see how this test can really prove anything. This seems more like a comparison of how well mpeg1 and mpeg2 perform at similar bitrates and when using different encoding modes.

[injunpana]

Oh yeah, I missed that too. Adam's right. The results won't mean anything.

[kinneera]

I would also argue that 3-min clips are too short for variable bitrate modes to demonstrate effective bitrate distribution, unless those clips are taken from a full encode.

[therick]

Ok, guys-points well taken. Yeah I wasn't sure about how put the files on equal ground, but I think my first inclination was right, I am just going to up the average until we have the same file size.

And as for kwag's template CQ file-will his modified GOP have any effect on MPEG-2? Could I just switch the encode to MPEG-2 in TMPEGEnc? If so I can just do that encode over again...

@adam - I agree that the bitrate is too low, but that is what kwag's template picked...so we should set up the parameters for a true DVD quality encode...I am down with that-the only problem is that I don't encode DVDs...so I will need a little help with what settings others use to encode DVDs.

@kinneera - what would be an acceptable length of time...10 minutes? 15? Get back to me and that will be the length of the DVD encodes...

Now to make that CQ file MPEG-2...be back in a bit....

[MoovyGuy]

And as for kwag's template CQ file-will his modified GOP have any effect on MPEG-2? Could I just switch the encode to MPEG-2 in TMPEGEnc? If so I can just do that encode over again...

If you want to use Kwag's stuff with Mpeg2, use his KDVD templates instead.

Although they share the same Q.Matrix with the KVCD templates, the GOP is different as the rules for DVD compatible Mpeg2 are very strict.

[Gtechman]

Why not compare Kwag's mpg1 vs CCE mpg2?? I'm sure that a lot of folks (myself included) are interested in only interested in 3 things: filesize, quality, and if it will play in my DVD player. Why not just get the filesizes similar and judge for quality??

[injunpana]

And as for kwag's template CQ file-will his modified GOP have any effect on MPEG-2? Could I just switch the encode to MPEG-2 in TMPEGEnc? If so I can just do that encode over again...

I think kwag's template uses also a modified Q matrix as well as the modified GOP arrangement. Also that this template is geared primarily for mpeg1 so it might not work as well for mpeg2. I'm not really sure of this, I just want to throw it in there so you would be aware of the possibility.

[therick]

Alright, I used kwag's 704x480 template and switched it to MPEG-2...right or wrong the file's name is: xmen(kwagMPEG-2).mpg

and it is located on my ftp site:

robbins.dns2go.com l/p : vcdhelp/vcdhelp

I think, though, that the Q matrix and GOP settings DO make a difference, as injunpana mentioned. The encode took a little longer than the previous kwag encode, but it is 30.2 MB-giving it a definite advantage over the other modes' tests. But since it is probably not a good representation of kwag's work then I guess it doesn't matter...

[adam]

Why not compare Kwag's mpg1 vs CCE mpg2?? I'm sure that a lot of folks (myself included) are interested in only interested in 3 things: filesize, quality, and if it will play in my DVD player. Why not just get the filesizes similar and judge for quality??

Because you can't encode to two totally different formats and try to make any judgements about the encoders that produced them. You need to eliminate as many variables as possible, and the format that you encode to is a MAJOR variable.

therick you don't necessarily have to go up to 6mbits just use something a little more reasonable like at least 2mbits. Using kwag's regular template and just setting it to mpeg2 should make a fair comparison. I don't see why you wouldn't get the same benefits as with mpeg1, as far as the gop modifications are concerned. Also I realize that Kwag's template is a part of this test but I think its really limiting, as far as the tests parameters. We are stuck with a non-standard resolution which gives horrible bits per pixel ratio compared to standard SVCD or VCD and the bitrate is already very low. Many people already agree that TMPGenc beats cce at low bitrate, so I really think tests at a higher bitrate are a must in comparing these encoders.

Also I think I have a suggestion that will make the test much more fair...make it a blind test. Tell us how you set up each encoder but don't tell us which sample came from where. This is the only way to make the quality judgments unbiased.

[therick]

great idea adam! I didn't even think of that! I tell you what-gimme a day and I will post three clips- Sample #1, Sample #2, and Sample #3.

I agree with what you say about the unstandardness (is that a word?) of it all but I did want to test kwag's template...but he has a DVD standard template as well, that I didn't know about, so how about I use that and then use TMPEGEnc's 2-pass for a DVD stream along with CCE's 2pass for a DVD stream...is that to everyone's liking? Lemme know so I can get started...

[adam]

No I don't think thats a word

Sounds good, only I think another comparison at ~2.5 mbits would be much more telling than one at 1.2mbits since most people who encode mpeg2 probably use alot higher bitrate than 1.2mbits. For instance, I think I will probably prefer TMPGenc's output at 1.2 but prefer cce's at 2.5.

And just one more minor suggestion. For any comparison other than the "kwag dvd template vs cce" raise the image priority to 22-24. The default of 17 is too low except on very high bitrate encodes.

[therick]

my image priority default default is 25...I am running 2.5, does that make a difference?

[kwag]

@therick:

As everyone has agreed that TMPEG looks better at MPEG-1 and lower bit rates, then go ahead as suggested, and do a comparison with CCE's MPEG-2 AND KDVD Full D-1 ( which is also MPEG-2 ).
That will compare "apples and apples" and may the best encoder win ( which I already know who it is 8) )
The KDVD template is set to 720x480 MIN=300, MAX=5,000Kbps CQ mode.
So go ahead and compare that with the same parameters on CCE.
That should be a fair test for everyone.
Please download the latest KDVD Full D-1 template, as it also integrates the KVCD's Q.Matrix.The KDVD and KVCD Plus templates were sincronized today. So update your template before conducting the test. If you want to test our Q. Matrix with CCE, go ahead, but I haven't tried that. You'll probably have better results with CCE's own Q. Matrixes. And it's fair that we encode with KVCD's Q. Matrix, as CCE also has it's own optimized Q.Matrix. So there shouldn't be an argument like "ah, but KDVD is using another Q. Matrix", because CCE is also using another one.
Fair enough?

-kwag

[therick]

alrighty kwag, I just downloaded the template for the first time today, so I think that I should be alright. I probably won't get to run the tests until later on tonight, but I will post them in the exact same way I posted the first round.

[kwag]

alrighty kwag, I just downloaded the template for the first time today, so I think that I should be alright. I probably won't get to run the tests until later on tonight, but I will post them in the exact same way I posted the first round.

You can verify that you have the latest, if the quality setting is set at 70. The previous KDVD templates were set to 65.

And be sure to encode with the luminance level set to 0-255 in CCE, because the default is 16-235. The KDVD's default is 0-255.

-kwag

[Martyn1980]

I hope this isn't going to go the way of the last one

Im downloading the samples therick but is it normal to be 0nly getting about 15k a second on the downloads, im on 512k cable. ive just got the first kwag one which seems kinda irrelevant now im not going to download any more until during the night my connection is much better then, it might be that.

And Kwag, i wish you wouldn't come across as so biased, it kinda spoils things, and as for the tests for DVD are we aiming for filesize here or are you strangling my Maximum bitrate again which on DVD seems strange when you have a maximum of 9.80mbps.

[adam]

Yeah I don't understand why you would use a max of 5mbits for dvd. Kwag if you want to keep the avg and filesize down then lower the quality level. In order for the vbr allocation to even be all that useful your avg would have to be like 3mbits, which is hardly representative of a dvd.

Make it min 300, max 9mbits, and shoot for an avg bitrate of around 6mbits. I think this would be fair.

[kwag]

Yeah I don't understand why you would use a max of 5mbits for dvd. Kwag if you want to keep the avg and filesize down then lower the quality level. In order for the vbr allocation to even be all that useful your avg would have to be like 3mbits, which is hardly representative of a dvd.

Make it min 300, max 9mbits, and shoot for an avg bitrate of around 6mbits. I think this would be fair.

The max of 5,000Kbps was chosen for a balance between space/quality. The goal of the KDVD template was to put around 4 hours in a DVD(+-)R at full DVD quality. I missed!. The KDVD gets between 4.5 to 5 hours per DVD! With TMPEG, and now with the new Q. Matrix, even on high action scenes the bit rate barely reaches 5,000. The CQ was chosen to maintain a quality which is equivalent to a DVD, and it does. Just visualize the CQ this way. The CQ value is the treshold point, the "low watermark", and the MAX is the "high watermark" point. If I lower the CQ from the current point, quality drops on a sharp curve. If I increase above the current point, the quality barely increases, but the file size does. So the CQ is set at exactly the crossover point of quality vs. file size. If we were to plot a graph with two curves, a quality curve and a file size curve, this would be the intersection point. So sliding the CQ value above 70, in this case, the file size rockets but the quality curve starts to flatten. Below this point, the file size drops rapidly, and so does the quality. To it's set pretty much at the saturation point. Viewed from a quality perspective.

-kwag

[Martyn1980]

Maybe im being too fussy, but ive been trying encodes using CCE and Tmpeg on 6000, 7000,8000 and even 9000kbps CBR and neither can match the original, even if the filesize is larger than the original, so the word DVD quality usually makes me laugh, but not many people have their nose pressed against the screen, watching the same clip over and over again

And surprisingly Tmpeg holds up very well at high bitrates too, the gap between the two encoders at high bitrate mpeg2 isn't as big as most people make out, and considering the price difference, ,it really is amazing.

[injunpana]

Maybe im being too fussy, but ive been trying encodes using CCE and Tmpeg on 6000, 7000,8000 and even 9000kbps CBR and neither can match the original, even if the filesize is larger than the original, so the word DVD quality usually makes me laugh, but not many people have their nose pressed against the screen, watching the same clip over and over again

Unlike you, I don't let myself get obsessed trying to get that "high quality" to the point of watching the entire movie clip over and over looking for artifacts or blurring. I collect dvds so there is no need for me to do that.

I do make xvcds for the purpose of sharing the movie to some friends and the quality of a vcd just don't cut it.

As long as I get a quality where the video is crisp and clear and there is no presence of blocks, that is good enough for me. I already wouldn't be embarrassed sharing it to friends knowing that they would enjoy the movie without being distracted by a poor quality.

[Martyn1980]

I too collect DVD's i do this more like a hobby if i really want a film i just buy it, im just doing this to see how far i can push it, and see what the best quality i can get is, ive done stuff which other people say looks as good as the original but ive hated it.

I think most people do this for fun anyway because is it really worth all the hassle to say a few pounds/dollars/euros or whatever your currency is, its more the fun and satisfaction of accomplising it and showing people and saying "i did that"

[adam]

The max of 5,000Kbps was chosen for a balance between space/quality.

It was my understanding that the CQ level and the max bitrate were relative to each other. Raise the max bitrate by %1 and lowering the CQ value by %1 will offset this. I haven't tested this thouroughly though and I'm not quite sure how to.

CQ is a pretty unconventional encoding mode so I can understand using unconventional settings. The problem though is that these settings don't translate well to other encoding modes.

In this encoder comparison Kwag's templates are simply another variable. To truly test the encoders against one another you have to eliminate this variable.

I think pitting KSVCD or KDVD against cce at comparable settings is a very worthwhile test. For anyone who uses Kwag's templates its certainly worthwhile to know which encoder performs best at these settings and at these lower bitrates. For someone like me this test is not very telling at all though. Right or wrong I would never use a max of 5mbits for DVD or an avg of say 1mbit for SVCD.

If you want to test Kwag's templates vs CCE at comparable settings, that's one test. If you want to compare TMPGenc vs CCE than you need to use more standard settings. I would suggest using the standard SVCD and DVD templates in TMPGenc and making comparable encodes in CCE.

Why not test TMPGEnc vs CCE both with Kwag's templates and without? I'm sure that not everyone who uses TMPGenc uses Kwag's templates and they do limit the parameters of the test significantly.

The discussion that inspired this test is very broad. It cannot be encompassed in 3 short samples, but however the test is done I will give my honest opinion of every sample. In the end I know exactly what's going to happen. For any set of samples theres going to be disagreement as to which one is the highest quality. That's what happens when you test something as subjective as visual quality.

[Gtechman]

Adam

It really doesn't sound like you will ever be satisfied with any test. To me, using the default settings in each is not a good test because we know we can get better results by tweaking them. Kwag has worked to tweak TMPGeng. Why don't you tweak CCE?? I haven't used CCE, but I thought you couldn't use some of the settings Kwag uses in his templates.

[kinneera]

And be sure to encode with the luminance level set to 0-255 in CCE, because the default is 16-235. The KDVD's default is 0-255.

??? The luminence level should be matched to the colorspace, regardless of encoder/template. If it's a DVD source, it almost certainly should be set to TV colorspace in DVD2AVI and 16-235 in both encoders.

[kwag]

And be sure to encode with the luminance level set to 0-255 in CCE, because the default is 16-235. The KDVD's default is 0-255.

??? The luminence level should be matched to the colorspace, regardless of encoder/template. If it's a DVD source, it almost certainly should be set to TV colorspace in DVD2AVI and 16-235 in both encoders.

NTSC DVD's are recorded as CCIR601. That is the digital standard television signal, with a scale from 0 to 255. That's why I suggested to change to 0-255, because the sample mentioned for the test, is from a DVD. 16-235 is used for viewing on PC monitors.

-kwag

[adam]

Adam

It really doesn't sound like you will ever be satisfied with any test. I'To me, using the default settings in each is not a good test because we know we can get better results by tweaking them. Kwag has worked to tweak TMPGeng. Why don't you tweak CCE?? I haven't used CCE, but I thought you couldn't use some of the settings Kwag uses in his templates.

Tweak CCE? You're totally missing the point. I have no problem with Kwag's gop modification or the modified Quant scale, the problem is his bitrate settings. If you only compare Kwag's templates to CCE's output then you are putting a severe cap on the total bitrate used in the encode and it has already been established that TMPGenc probably outperforms CCE at low bitrates.

Bottom line, if you are only comparing Kwag's templates to CCE than a typical SVCD encode is going to have an avg bitrate of 1.5mbits tops, and a dvd encode will have an avg bitrate of 3.5 to 4mbits tops. These are not representative of these formats at all and I personally would like to see a test that actually applies to the formats that I create.

I would like to see a test between TMPGEnc and CCE at 2 mbits avg, 2.5mbits max for SVCD and at 6mbits avg, 9mbits max for dvd (lower max for compatibility with dvd recordable media.) These are the most logical bitrate settings to use for encoding these formats in x-pass vbr yet if Kwag's templates are used in the tests than these bitrate settings ARE NOT POSSIBLE TO TEST!

No I will not be satisfied with any test...I will only be satisfied with a number of tests. Like I said, all testable parameters of these two fine encoders cannot be encompassed in 3 short samples.

If CCE is designed to perform optimally at 6mbits avg then don't limit the avg to 3.5mbits! Make a test that actually tests something.

Tweak TMPGenc all you want but just give me a test at higher bitrates!

[kwag]

Adam

It really doesn't sound like you will ever be satisfied with any test. I'To me, using the default settings in each is not a good test because we know we can get better results by tweaking them. Kwag has worked to tweak TMPGeng. Why don't you tweak CCE?? I haven't used CCE, but I thought you couldn't use some of the settings Kwag uses in his templates.

... it has already been established that TMPGenc probably outperforms CCE at low bitrates.


If CCE is designed to perform optimally at 6mbits avg then don't limit the avg to 3.5mbits! Make a test that actually tests something.

Tweak TMPGenc all you want but just give me a test at higher bitrates!

@adam and All:

Then there's no test to be done. You've just admitted that TMPEG outperforms CCE at low bit rates. So for putting movies in one ( or two ) CD's with better quality, TMPEG is the way to go! Case #1 closed.

So you want to compare CCE and TMPEG's MPEG-2 at high bit rates? Hell, then just encode CBR at 7,000+ and there won't be any difference between any of the two encoders. Oh, wait, yes there is. About $2,000 less for TMPEG. So case #2 closed too.

Conclusion:
There's no sense in doing any more tests. TMPEG is better for low bit rates. Tried and proven. And at higher bit rates, there's no difference between encoders. So people, what's you're choice for MPEG-2 DVD encoding? The way I see it, this is a WIN WIN situation for TMPEG.

-kwag