2009 MPEG-2 Encoder Test!

[simps]

*Sorry for the bad english.
**Don't forget to vote the poll!



Personal note: Frames are also saved on my Gmail account!

Frames backup 25.4MB [Mirror 1]:
http://www.easy-share.com/1905095196/frames.rar

Frames backup 25.4MB [Mirror 2]:
http://rapidshare.com/files/230615654/frames.rar.html



1-Encoders tested:
HCEnc v0.23 Free
QuEnc v0.72 Free
TMPGEnc v2.525 Free
CCE SP v2.7
Canopus Procoder v3.0
Main Concept Reference v1.61



2-Encoding time (normalized):
CCE SP 1.00
HCEnc 1.05
Main Concept Reference 1.15
Canopus Procoder 1.28
TMPGEnc 4.10
QuEnc 5.85



3-File size:




4-Real bitrate (Kb/s):
CCE SP 3999
HCEnc 3994
Canopus Procoder 3994
TMPGEnc 3994
Main Concept Reference 3976
QuEnc 3888



5-Results (my view of it):
Best: CCE
Very good: HCEnc (very close to CCE)
Good: Procoder / QuEnc (too bad QuEnc is so slow)
Not so good: TMPGEnc / Mainconcept



6-Encoders:




7-Setup:
Source is a 7Mbit/s x264 rip of my blueray copy of The Dark Knight.
Frames are 720x480 (16:9) for standard NTSC DVD.
Avisynth for frameserving
VBR (two pass) 0 - 8000 with average of 4000Kb/s.
DC-Precision set to 9.
Standard DVD gop structure.
All settings on each encoder set to max. quality.
I know all of this encoders very well, so I believe I created a fair scenario for the comparison.



8-Frame 800:

Source:


CCE:


HCEnc:


Main Concept Reference:


Canopus Procoder:


QuEnc:


TMPGEnc:




9-Frame 1174:

Source:


CCE:


HCEnc:


Main Concept Reference:


Canopus Procoder:


QuEnc:


TMPGEnc:




10-Frame 1934:

Source:


CCE:


HCEnc:


Main Concept Reference:


Canopus Procoder:


QuEnc:


TMPGEnc:




11-Frame 3504:

Source:


CCE:


HCEnc:


Main Concept Reference:


Canopus Procoder:


QuEnc:


TMPGEnc:




12-Frame 3545:

Source:


CCE:


HCEnc:


Main Concept Reference:


Canopus Procoder:


QuEnc:


TMPGEnc:




13-Frame 3805:

Source:


CCE:


HCEnc:


Main Concept Reference:


Canopus Procoder:


QuEnc:


TMPGEnc:




14-Frame 4152:

Source:


CCE:


HCEnc:


Main Concept Reference:


Canopus Procoder:


QuEnc:


TMPGEnc:




15-Frame 6040:

Source:


CCE:


HCEnc:


Main Concept Reference:


Canopus Procoder:


QuEnc:


TMPGEnc:




16-Frame 8944:

Source:


CCE:


HCEnc:


Main Concept Reference:


Canopus Procoder:


QuEnc:


TMPGEnc:




[]'s
Simps

[Soopafresh]

Thanks for doing that! That's a lot of work. I find myself being drawn to the Procoder output, probably because of the contrast and saturation looks greater than the others.

[poisondeathray]

Thanks for the tests.

IMO, the job of an encoder is to faithfully represent the frames sent to it, unless you set it not to do so, that should be the job of pre-processing and filters. So in that respect, the Procoder output is the worst. The detail in the blacks are gone with the levels shift. There must have been some levels / contrast / saturation setting not set correctly?

The CCE output is shifted, and your forgot to fix the line shift setting. Did you forget to disable low pass filter as well? the output doesn't look as crisp as it should, you can tell especially by looking at the "school bus" letters

HCEnc looks the best to me. TMPGEnc is pretty bad in terms of artifacting and image quality. In the school bus frame, there is quite a bit of haloing and artifacting by mainconcept (edge of the bus and corner of building), and the picture is not as crisp if you look at the background buildings compared to HCEnc - what AQ setting was used for HCenc?

Quite of color descrepancy between the shots and the source. What colormatrix did you use? Maybe post the script used?
Maybe you could redo the CCE and Procoder with proper settings?

[simps]

poisondeathray,

Every encoder was tunned for max. quality. For example, procoder in mastering mode, QuEnc in extreme slow mode, HCEnc in best profile mode, etc. Some encoders have different settings, but my rule was to take advantage of every feature of every encoder, respecting of course the bitrate and other stuff.

About procoder, that is just the way it behaves. Procoder in mastering mode always mess with contrasts and saturation like that. I don't like it too.

And you are right about CCE. I really forgot to disable the LPF. I am used to CCE 2.5, and it has a different menu. For some reason I decided to use CCE 2.7, and that setting just escaped from me

HCEnc was tunned with AQ=2 and Lum_Gain=1.

I just added two more frames (3504 and 6040) into post #1.

Hope it helps.

[]'s
Simps

[simps]

I am re-doing CCE without the LPF and line shift.
Will post soon...



Done. CCE is fixed now, this are the CCE settings used now. 1st post updated with fixed CCE frames.




[]'s
Simps

[edDV]

Encoders should maintain levels vs. originals so contrast or saturation differences point to a problem.

Samples should be in motion with lighting or texture torture tests. For consumer application, noisy sources should be tested. It goes on and on for thorough testing.

One should look for artifacts in motion areas or where slight gradients of luminance or chroma exist. All this should be done at various output bit rates.

[manono]

What are the scripts? It doesn't look to me that the colorimetry was adjusted for the downconversion from Hi-Def to Std-Def.

[simps]

I said I wasn't going to rank the encoders, but here we go:

Best: CCE
Very good: HCEnc (very close to CCE)
Good: Procoder / QuEnc (too bad QuEnc is so slow)
Not so good: TMPGEnc / Mainconcept

This is the result in my opinion.

[]'s
Simps

[2Bdecided]

Encoders should maintain levels vs. originals so contrast or saturation differences point to a problem.

I think the Procoder issue is well known. It assumes it's being fed different levels - which, in some specific instances, it could be. I think it's wrong, but...! There are work-arounds.

Samples should be in motion with lighting or texture torture tests. For consumer application, noisy sources should be tested. It goes on and on for thorough testing.

"Thorough" testing would include interlaced video with challenging content

Then we'd see some serious differences, and TBH at 4Mbps I think you'd see all those encoders falling apart in a still image comparison.


Good work though simps - that must have taken ages!

Cheers,
David.

[MysticE]

Nice test. I find it interesting how good the old and basically abandoned QuEnc looks.

[Delta2]

There is some more tweaks that can be done to CCE

I don't use that settings :

Closing Gops that according to CCE manual it degrades image quality,

Quantizer Characteristics, should be 1 from HD->SD because it keeps details, and a average value (26 +/- ) from SD->SD

IT all depends on the source

As for the colometry for CCE, YUY2 is the recommended one, and it is the best to use....but slows down CCE a lot, and I mean a lot, so I usually use YV12

A good and sharpen downsizer plus a good QMat also makes miracles

So I think that some CCE images can be improved

[simps]

Delta2,

You were right. I just re-did CCE with Open GOP's (don't think this had an effect), but I also did it with Quantizer Characteristics -> 1.
And results are much better now. CCE was already the winner, but now it is even better. Just look at Frame 800 and Frame 8944. You can see a big improvement.

First post is already updated, with frames of CCE running from settings below.
Let me know if there is another setting there that can improve, and I will give it a go again. I still have all the setup on my PC, it is easy to re-do some encodes.

Thanks.

[Delta2]

As far as I can see it is what I use in AVI2ISO, 95% of the users of that soft are CCE dependant

nevertheless they are free to change everything, either in AVISYNTH script during runtime or via ECL template

I only notice that 10 pass setting, and that is just useless, more than 3+1 you barely notice any change, is just a waste of time

And also V/C ratio should be 25 ( I use 20 ) in order to give full VBR quality advantage over CBR, the full advantage of using all bitrate scope ( intervals )...do you remember your high motion problems about HD ?

CBR in CCE is foolish, I rather use VSO, or some ipod converter

[simps]

Delta2,

I tried with V/C = 20, and the result was not really the same as with V/C = 30, but very close to it, just some pixels changing arround the frame, but I didn't feel it looked better. I actually found that it looked better with V/C = 30, so I am leaving it that way and calling it best for the moment. But than again, it was pretty much the same result, just some very few pixels changing arround the hole frame, and for me, the frame with V/C = 30 looked better.

[lordsmurf]

Sorry to be negative here, but these tests are worthless.

It is impossible to demonstrate MPEG quality with stills, especially not when your source is a DVD (or Blu-ray or download, or whatever else).

The same frame may not necessarily be a I, P or B frame from encoders to encoder. Beyond that, the motion of a video is more important than the static still.

This is the kind of test a newbie makes, and it was already discussed here on this site almost 5 years ago, as well as in professional magazines at the time.

This very concept of "stills=useless" is often mentioned in professional video magazines, because it's true. They do it as a disclaimer, when using a sample of a video encode to demonstrate the complaint. Many times, the temporal compression differences are also mentioned.

The only way to truly test with a still is to encode with zero temporal compression, but that sort of defeats the purpose of testing MPEG encoders.

Try again, with something that didn't come off a DVD. Use DV footage. And then encode multiple clips at multiple resolutions, with various bitrates, and analyze the output from there.

I've done this, it's a bitch, it takes much time.

I don't want to seem so negative, but somebody that doesn't know what they're doing tends to really confuse and deceive other people who also don't know what they're doing.

Good effort, but seriously, try again.

And next time, post your images on Videohelp itself, attach the images to your forum post. That imageshack.us site takes too long to download, and these will expire eventually, rendering this most even more useless as time goes on.

[fritzi93]

I suspect that, (watching the files in motion rather than as stills), QuEnc will rate slightly lower and HCEnc slightly higher. Just my own empirical impressions from using both. [shrugs]

But I defer to superior knowledge and will watch this thread with interest. BTW, there have been exhaustive tests posted here in the past. I remember one in particular a couple of years ago.

[Bjs]

You left out Bbmpeg

[simps]

lordsmurf,

Frames used here were either all I or all P or all B for every encoder.
Most of them are B. I really did pay attention to that.
60% of the frames I picked were during fast motion on the movie, 40% during slow motion.

I do have all the .mpg files for real moving action comparison, but those files add to more than 1GB and I will not post that. I have seen them all, and the frames comparison just do a good justice (in this case) to it. CCE won here, and would won if you were watching the .mpg too. So it is good enough.

I still have the setup in my HD, I can show you any frame you like. I encoded from frame 0 to 9805. Pick any frame on that interval and I can show you. I just can´t garantee they will be all I or P or B. The ones I posted, I can garantee they are all the same kind, because I checked. And they are about 66% B (worst one).

This is not supose to be a definitive test or anything very special. This is just some amature comparison between the encoders, not intended to be the full scale thing. If anyone want to see some comparison, they can come here and see, it is just that, and it server for this porpous. Does it have limitations? OF COURSE, tell me where I said this was the real complete comparison? I never said that, you are assuming that I believe this is some kinda of "real deal" of a comparison, and then making your comment. Looks kinda patetic hum?

About your negative comment on this, I would expect that, considering we were discussing in another thread. So, nothing really special here. Nice try, but really, YOU should try harder

PS: I think it is a good idea to upload the frames to vhelp instead of imageshack. I will do that. Thanks for helping

[simps]

You left out Bbmpeg

I can give that a go. Will try tonight, and post it here.

[disturbed1]

You left out Bbmpeg

Please study the settings on bbmpeg. It's a bit more advanced than the others you've tested here. Though the older Mainconcept encoders did have the same options hidden away, which they should considering the same guy coded both apps

I like what lordsmurf said. It's pretty much spot on. He left out the fact that the stills are also compressed yet again.

The only proper way to accurately test quality is by play back on it's intended device by it's intended audience. After enough hours go by, you're eyes will start to water, and you'll realize that even if there is a best encoder, it is not best by that much. Unless you royally screwed up the settings somewhere.

These tests have been done years ago on this forum. And most likely every member that joined before 2003 has done their own tests as well. There are encoders that stand out in progressive encoding, interlaced encoding, and a couple that do well on both. CCE seems to like progressive streams, ProCoder and TMPG like interlaced streams. Mainconcept does a good job with both.

frames used here were either all I or all P or all B for every encoder.

This is impossible unless you set up proper ecl's, even then I doubt this is the manner in which this test was conducted. [s:2f291e88b4]5[/s:2f291e88b4], [s:2f291e88b4]4[/s:2f291e88b4], [s:2f291e88b4]3[/s:2f291e88b4], 2 encoders will not all encode frame $$$ as an I frame. Unless you set the encoders up to use the exact same IPB sequence with closed GOPs, and no scene change detection. If those are the settings you used, the quality isn't as high as it should be and you are not taking full advantage of the products.

[minidv2dvd]

to me this is just another warez user trying to show off his ill gotten crap he thinks he knows how to use. if you wanted to test current encoders it wouldn't be 3 and 4 year old versions of cce, procoder and tempgenc.

[d4t]

Amazing work! I was searching for a good roundup like this for a long time!
Best one so far. HCenc has my vote!

btw, you achieved some GREAT quality frames for 4000kb/s, I am VERY impressed. Good job mate.

[lordsmurf]

Amazing work! I was searching for a good roundup like this for a long time! Best one so far. HCenc has my vote! btw, you achieved some GREAT quality frames for 4000kb/s, I am VERY impressed. Good job mate.

You must not have looked very hard. A really good round-up of MPEG and H.264 encoders was done in several magazines at the end of last year. I don't recall if it was VideoMaker or DV anymore, I'd have to dig out my notes (I ripped out the pages). MainConcept won both of those, too, FYI. It also included a number of professional hardware and software methods not found in this post.

The test in this post would only be taken seriously by somebody else that doesn't know what they're doing.

[d4t]

Amazing work! I was searching for a good roundup like this for a long time! Best one so far. HCenc has my vote! btw, you achieved some GREAT quality frames for 4000kb/s, I am VERY impressed. Good job mate.

You must not have looked very hard. A really good round-up of MPEG and H.264 encoders was done in several magazines at the end of last year. I don't recall if it was VideoMaker or DV anymore, I'd have to dig out my notes (I ripped out the pages). MainConcept won both of those, too, FYI. It also included a number of professional hardware and software methods not found in this post.

The test in this post would only be taken seriously by somebody else that doesn't know what they're doing.

I've read this again and find it to be a good comparison test, and I can't find a better one on the internet. Results are just what I would expect for progressive sources, with CCE and HC on top of it.
Also found you to be very arrogant.

simps, again, thanks for the hard work. Haters will always be arround, just ignore.

[lordsmurf]

Says the person with 2 posts, who joined simply to praise simps, the original poster who put down his bat and went home (or so he says)

My bet says that d4t = simps

[SatStorm]

Smurfs don't bet. Papa Smurf don't allow it!
So, you gain nothing for your bet!

[lordsmurf]

Haha ... good to see you're still around Satstorm.

[deadrats]

to me this is just another warez user trying to show off his ill gotten crap he thinks he knows how to use. if you wanted to test current encoders it wouldn't be 3 and 4 year old versions of cce, procoder and tempgenc.

your comment is not only way out of line but also idiotic beyond belief, and i'm being nice about it.

this guy took the time to put together this test and for the most part all he has received is people ragging on him. i would be most interested in knowing what logical gymnastics you performed to conclude that this guy is a warez user if by your own statement he used "3 and 4 year old versions" of the various encoders.

i just did a quick check on 2 very well known torrent sites and i found the latest versions of procoder, tmpg enc express, tmpg author 4, sony vegas, adobe's suite, and a ton of other commercial video editing and encoding applications. if he really was a warez user that wanted to "show off his ill gotten crap" he would just have run the tests with the latest versions of all the apps available on torrent sites, so your accusations are based on faulty reasoning.

now if you want to offer some constructive criticism of his test, that i believe would be warranted, primarily due to the fact that he started with x264 bd-rips encoded at 7 mb/s as his source and set mpeg-2 at an average 4 mb/s as his target, in short he went from a more efficient codec at a higher bit rate to a less efficient codec at a lower bit rate, albeit at a significantly lower resolution, that would be fine.

but for you to engage is warrant-less character assassination is just douche beyond belief.

[minidv2dvd]

who needs a warrant, the versions he listed are out of date and useless for comparisons. if he had "legal" copies they would have been updated to the latest available and therefore useful for comparisons.

we covered all these years ago and this is just junk. i will agree with your assessment of his source material though, it's not good enough.

[lordsmurf]

Sorry, but I'm inclined to agree with minidv2dvd on this one. Anybody that actually owns this software, and has experience using it, would know better than to create a "test" like this.