Is x264 the best?

[hello_hello]

These forums, and forums like it around the net, are filled with people asking questions like "I always use CRF 18 (or whatever number) for my encodes and have been happy with the results but for some reason movie 'x' is resulting in any of the following - poor quality, bigger size, etc, why is that".

Taking that argument to it's logical conclusion, every Bluray video you buy should be encoded with virtual perfect quality. That's not the case though.

The discussion seems like it's heading in a familiar direction. x264 can encode at fairly high quality at lower bitrates than other encoders and people use it that way, but being better at low bitrates somehow makes it a bad encoder. Or when it's not perfect that makes it bad encoder etc.

Here's the reality, P and B frames by definition are based on I frames, it's to your benefit to have very quality I frames and it's to your benefit to have as many as possible, I typically like to have a GOP length of half frame rate. The added benefit of using such a short GOP is that you don't get frame pulsing (I have noticed that x264's quality tends to degrade as the GOP progresses) and you eliminate the need to use scene cut.

Maybe x264 is by default geared towards higher compression, but I'm still not sure that's a bad thing, or makes it bad by design. It's why most of us use it.
I can't say I've noticed quality degrade over the duration of a GOP, at least not at higher quality CRF values, but scenecut is my friend. I sometimes split and join encoded video (maybe to re-encode a section of it etc) and being able to split on a scene change 99% of the time, or knowing that's where a keyframe will be is fairly convenient. Trying to split video when keyframes are all over the place in respect to scene changes......
Is there are downside to scenecut in respect to quality?

Anyway, I'm open to trying something different. If you post the x264 settings you normally use in conjunction with bitrate based encodes, I'm willing to give it a spin....

As a side question, does anyone know what h264 encoder Apple uses, assuming all their iTunes video is encoded with the same encoder? I ask, because I find mostly the encoding quality is fairly high without the bitrate being over the top. Just curious.....

[_Al_]

It would be not wise to put --keyint into command line example because it is highly specific variable, it does not mean that it is not used,

bitrate goes up with short GOP, but talking about this, you point it out like something bad, but it is NLE encoders that actually "excel" in that, "pumping" is the main reason to use x264 (besides available CRF choice) NLE could render video as unusable, so most NLE users just encode CBR 25Mbit/s, thinking that that is the only safe way to encode.
What is the best seems never a correct question because question is always, what is the best for me. I could talk forever about NLE encoding but other guy is like what is he talking about? The only thing that matters is when I encode a movie, where long GOP is perfectly legit choice. Anyway there is about 1000 people encoding the same thing today, why it should matter how it is encoded, I mean movie. The other guy says it is my TV recording. The other guy says , I have to encode the other peoples crappy phone videos or mostly Youtube to make it available for streaming for my web news pages and make that video mine with advertisement in it

[Selur]

I typically like to have a GOP length of half frame rate.

I hope you use open gop when using such low gop sizes.

about commercial encoders being better than x264, iirc. the whole 'Friends'-Box set from Warner Brothers was encoded with x264

[poisondeathray]

CRF mode is kind of like shooting darts at a board with a blind fold one, you have no idea of what you're getting, you're choosing an arbitrary number that corresponds to an average quantizer and you're hoping for the best.

With a bit rate based encode you choose an average bit rate that you know will give you uniform quality throughout the encode and you're done.

It's not true or a perfect way either. How are you defining "uniform quality"?

And you don't necessarily know what average bitrate is appropriate for the content. Isn't that like shooting darts also ?

If you pick a random 8Mb/s isn't that going to be too low for some, too high for others ? So some videos will look like crap, others will look ok ?

Neither way is "perfect"

But it's much more than that, the amateur encodes, those done for personal use also tend to use long GOP's as a way of squeezing the most compression out of a file but since this is lossy compression it comes at the expense of quality. There's a reason Blu-Ray, encoded with professional grade encoders like CCeHD use GOP lengths equal to frame rate and those pro caliber encoders do not have a CRF mode. I would think if it were really that good high end encoders like CCeHD and Ateme would have implemented their own versions by now.

Yes, and it doesn't have to do with the reasons you're suggesting.

It's because of the blu-ray spec. You're not allowed more than 1 sec GOP , unless you use 2 sec GOP rules (basically a lower bitrate version). Blu-ray requires VBV constraints and bitrate buffering. Well guess what? You can use Blu-ray compliant settings with x264. Although you can use CRF encoding with VBV, it can lead to problems with buffer handling (you won't violate VBV restrictions, but it can lead to quality issues when constrained)

For personal use, computer playback - unrestricted (non VBV controlled) is always going to give you better quality and compression efficiency than constrained, whether you use 1 pass, multipass, or CRF

Let's put it another way. If you used the same blu-ray bitrates, same settings, and used unrestricted settings - you will have higher quality than the same blu-ray encode. That's the same with retail encoders as well , not just x264 (if they allowed non restricted encoding). Mainconcept has "CRF" encoding now as well, they call it "quality" based control

Professional content providers that offer content for sale via a website typically use double frame rate GOP lengths.

That's not true either - You've sampled what? A few porn sites ? And you think that represents the entire market ?

So I find it odd that people use CRF and long GOP's, I think x264 defaults to 250 frames and the developers. by their own admission mind you, cheat on the MSU tests by specifying GOP lengths of 500 frames.

It's not really "cheating". They ask the developers input on what settings to use. The assumption is that the author of "xyz" codec knows what settings are appropriate for "xyz" codec for that test situation.

They use --tune SSIM for SSIM testing as well - is that "cheating ?". Not really, it's optimizing to score higher "points" . But does that really reflect actual usage situation? Not really.

it's to your benefit to have very quality I frames and it's to your benefit to have as many as possible, I typically like to have a GOP length of half frame rate. The added benefit of using such a short GOP is that you don't get frame pulsing (I have noticed that x264's quality tends to degrade as the GOP progresses) and you eliminate the need to use scene cut.

Like all things, it depends on the situation , and everything is a tradeoff. So why don't you use intra for everything then? Because compression efficiency is impaired.

You can address "frame pulsing" by other means, such as adjusting the I:P , P:B ratios, open GOP's , shorter GOP's (not necessarily as short as you're suggesting) .

[_Al_]

people asking questions like "I always use CRF 18 (or whatever number) for my encodes and have been happy with the results but for some reason movie 'x' is resulting in any of the following - poor quality, bigger size, etc, why is that".

I'd like to point this out, because this is what I try to pinpoint, people do not know what bitrate is all right, I do not know it, you also, so that CRF is most efficient way to get where one stands at. If you want to be thorough you add trim(20000,21000) encode about that half minute and you know, as for 2pass you still do not know. This is the point, extrapolate a theory where unknown value (bitrate that is just fine for me where lowering it becomes not acceptable) is known value for some reason makes little sense.

[poisondeathray]

bitrate goes up with short GOP, but talking about this, you point it out like something bad, but it is NLE encoders that actually "excel" in that, "pumping" is the main reason to use x264 (besides available CRF choice) NLE could render video as unusable, so most NLE users just encode CBR 25Mbit/s, thinking that that is the only safe way to encode.

Yes, the pumping is typically much worse with other encoders. ( Mainconcept is the usual offender)

As a side question, does anyone know what h264 encoder Apple uses, assuming all their iTunes video is encoded with the same encoder? I ask, because I find mostly the encoding quality is fairly high without the bitrate being over the top. Just curious.....

They used to use Quicktime AVC (compression quality isn't that great) , I don' t know if that has changed recently

[_Al_]

CRF mode is kind of like shooting darts at a board with a blind fold one, you have no idea of what you're getting, you're choosing an arbitrary number that corresponds to an average quantizer and you're hoping for the best.

With a bit rate based encode you choose an average bit rate that you know will give you uniform quality throughout the encode and you're done.

It's not true or a perfect way either. How are you defining "uniform quality"?

or I'd say using CQ to stick to a quantizer instead of CRF, but it costs bitrate, choice is camouflage or more bitrate

[Mephesto]

The responses always show that no one has the answer because their is no answer, CRF mode is kind of like shooting darts at a board with a blind fold one, you have no idea of what you're getting, you're choosing an arbitrary number that corresponds to an average quantizer and you're hoping for the best.

Welcome to the whacky world of subjective quality that no two people agree on, bro!

With a bit rate based encode you choose an average bit rate that you know will give you uniform quality throughout the encode and you're done.

I love how careful you were with your choice of words when adding that qualifier "uniform" to quality. Too bad the fact of CRF and 2pass mode being equally uniform destroys your argument altogether. The --qcomp value controls uniformity, sir. The only uniform aspect of 2pass mode not shared by CRF is average bitrate. You don't know what filesize you'll end up with when using CRF.

CRF mode actually produces very slightly better quality than 2pass at the same filesize because the motion vectors don't have to be suppressed to artificially comply with a specific bitrate. But the analytical first pass of 2pass mode greatly mitigates this which is why you won't notice the difference even with close inspection.

This kind of debate is really obsolete, it's been settled almost a decade ago.

[hello_hello]

As a side question, does anyone know what h264 encoder Apple uses, assuming all their iTunes video is encoded with the same encoder? I ask, because I find mostly the encoding quality is fairly high without the bitrate being over the top. Just curious.....

They used to use Quicktime AVC (compression quality isn't that great) , I don' t know if that has changed recently

720p doesn't use CABAC (I've no idea how much that effects compressibility). 1080p does (I think it's generally High Profile, 4.1) but I mainly asked because for TV shows at least, the bitrates aren't too much higher than you'd expect from x264 in the CRF18 region (on average). Around 4000kbps for 720p and 5000kbps for 1080p. For 1080p that might even be a slightly lower average bitrate (I don't do much 1080p encoding), and from the limited amount I've seen, there's not much to complain about quality-wise.

Back to the subject of short GOPs.....

So I find it odd that people use CRF and long GOP's, I think x264 defaults to 250 frames and the developers. by their own admission mind you, cheat on the MSU tests by specifying GOP lengths of 500 frames.

Here's the reality, P and B frames by definition are based on I frames, it's to your benefit to have very quality I frames and it's to your benefit to have as many as possible, I typically like to have a GOP length of half frame rate.

At one stage I ran some comparison encodes between x264's default settings and the settings required for Bluray compatibility. I think the latter increased the bitrate by around 10%. It was a while ago so I can't remember exactly, but I'm sure it was at least 10%, probably more most of the time. When I tested them individually, the only Bluray compatibility setting having any real effect on bitrate was the reduction in GOP size. It was responsible for pretty much all of the increase. A half frame rate GOP would, I imagine, increase the bitrate even more (would 20% be a ridiculous guess?), which has me asking a similar question as before....
Decreasing the CRF value could increase the bitrate by a similar amount, but why wouldn't a lower CRF value distribute those extra bits better in respect to over-all perceived quality?

I've got quite a bit of PAL video de-interlaced with QTGMC to 50fps and for that MeGUI (or maybe it's x264) defaults to --keyint 500. I know the frames are whizzing by twice as fast, but I can't say I've noticed a quality hit compared to --keyint 250 at 25fps, and I assume it's the main reason why the bitrate only increases by something like 15% even though there's twice as many frames being encoded.

[SameSelf]

http://www.extremetech.com/wp-content/uploads/2013/07/HEVC-Encode.jpg

Some great stuff in this thread. I have been busy the last couple of days learning about frameserving out of PP to ffmpeg and Encore even. Truly amazing the things you can do. I feel like I am just scratching the surface, so much to learn. I feel a little sheepish coming back here. Maybe this should be saved for a new thread? But here goes.

If we convince ourselves that frameserving out of PP to x264 is preferred over AME (specifically MainConcept's encoder), what can be said about MPEG-2? Maybe frameserving to mpeg2video or some other encoder is preferred over AME?

I know this may be opening another can of worms but I am curious what you guys use when you need to create MPEG-2 encodes?

[poisondeathray]

If we convince ourselves that frameserving out of PP to x264 is preferred over AME (specifically MainConcept's encoder), what can be said about MPEG-2? Maybe frameserving to mpeg2video or some other encoder is preferred over AME?

I know this may be opening another can of worms but I am curious what you guys use when you need to create MPEG-2 encodes?

Yes, it's really another topic

While the difference is quite noticable for AVC (especially in the lower bitrate ranges), the difference isn't as large with MPEG2. (ie. Mainconcept's MPEG2 encoder isn't that bad). It mainly comes down to personal preference

There is a "x262" variant for MPEG2 (which borrows some x264 code), but development is sparse and very few commits. (And a new one called "y262" which is showing some promise). The fact is MPEG2 is being used less frequently, and is an already very mature format, so there is less incentive to develop encoders for it. The best overall, most mature, high quality free one to test against would be HCEnc. As usual, each mpeg2 encoder has strengths, weaknesses (unlike with x264 in AVC, it's strengths tend to overpower everything and it's clearly the "best" in most situations); but there are some mpeg2 encoders which clearly produce lower quality most of the time eg. ffmpeg's mpeg2 encoder even with massive tweaked settings. There is just no way it can compare to something like hcenc

[SameSelf]

Poisondeathray to the rescue again! I feared that I was opening another can of worms but your reply has already given me about a week's worth of noodling and more learning. Unfortunately I am late to the party but hope to be up to speed soon. As always thanks.

[sophisticles]

It's not really "cheating". They ask the developers input on what settings to use. The assumption is that the author of "xyz" codec knows what settings are appropriate for "xyz" codec for that test situation.

I'm working really long hours this week so I can't address everything you guys have said that I disagree with or upload some encoding samples but I will address this statement.

Would you like to know who considers it cheating? The x264 developer that is now a Pre-OP. I'm assuming you know that back before Fiona became Jason he/she used to have a blog called Diary of an x264 Developer. In it he/she posted a long rant about those that cheat on encoding tests and I'll let you read it for yourself:

http://x264dev.multimedia.cx/archives/472

The article is entitled "How to cheat on video encoder comparisons" and here is a direct quote:

Don’t attempt match compatibility options when it’s reasonable to do so. Keyframe interval is a classic one of these: shorter values reduce compression but improve seeking. An easy way to cheat is to simply not set them to the same value, biasing towards whatever encoder has the longer interval. This is most common as an accidental mistake with comparisons involving ffmpeg, where the default keyframe interval is an insanely low 12 frames.

Use ratecontrol methods unfairly. Constant bitrate is not the same as average bitrate — using one instead of the other is a great way to completely ruin a comparison. Another method is to use 1-pass bitrate mode for one encoder and 2-pass or constant quality for another. A good general approach is that, for any given encoder, one should use 2-pass if available and constant quality if not (it may take a few runs to get the bitrate you want, of course).

As you can see in this rant he excoriates almost all encoder comparisons as poorly done and two of the reasons is that they don't use matching key frames for all encoders in the comparisons and they use different rate control methods for all encoders, 2 things he/she flat out states are wrong and invalidate the comparison.

Yet if you read through the MSU documentations and do a google search for the settings Jason/Fiona tells them to use you will see that he has them use CRF and a gop of 500 with a key frame at the same interval.

He basically admits that he/she has cheated on the tests and that they are invalid.

His/her words, not mine.

As for the claim that "Friends" was encoded with x264, is there any proof of this. I have heard claims over the years that some Blu-Ray here or there was encoded with it, for instance Batman from Warner Bros is rumored to have been encoded with x264 yet I can find no evidence to support such a claim.

I can find evidence that the 2012 Olympics were encoded for broadcast with the MultiCoreWare OCL modified version of x264, but the documentation makes it sound like they used a version that was never committed to the main branch, that it's only available via a special license.

I have to go to work, doing another 16 hours but this weekend I want to post some encoding samples. BTW, does anyone know if any really high quality, as in pristine, never before compressed live action y4m samples I can include?

Thanks for reading.

[badyu17]

Yet if you read through the MSU documentations and do a google search for the settings Jason/Fiona tells them to use you will see that he has them use CRF and a gop of 500 with a key frame at the same interval.

I had the same impression as you regarding their use of these settings for MSU tests but you made one mistake in your sentence. --keyint 500 does not mean that every gop is 500 frames long. It just means that is the maximum distance between two keyframes. That means UP TO 500 frames long gop, not that all gop's will be that long. The other setting --min-keyint is the minimum range between two keyframes.

[poisondeathray]

It's not really "cheating". They ask the developers input on what settings to use. The assumption is that the author of "xyz" codec knows what settings are appropriate for "xyz" codec for that test situation.

I'm working really long hours this week so I can't address everything you guys have said that I disagree with or upload some encoding samples but I will address this statement.

Would you like to know who considers it cheating? The x264 developer that is now a Pre-OP. I'm assuming you know that back before Fiona became Jason he/she used to have a blog called Diary of an x264 Developer. In it he/she posted a long rant about those that cheat on encoding tests and I'll let you read it for yourself:

http://x264dev.multimedia.cx/archives/472

The article is entitled "How to cheat on video encoder comparisons" and here is a direct quote:

Don’t attempt match compatibility options when it’s reasonable to do so. Keyframe interval is a classic one of these: shorter values reduce compression but improve seeking. An easy way to cheat is to simply not set them to the same value, biasing towards whatever encoder has the longer interval. This is most common as an accidental mistake with comparisons involving ffmpeg, where the default keyframe interval is an insanely low 12 frames.

Use ratecontrol methods unfairly. Constant bitrate is not the same as average bitrate — using one instead of the other is a great way to completely ruin a comparison. Another method is to use 1-pass bitrate mode for one encoder and 2-pass or constant quality for another. A good general approach is that, for any given encoder, one should use 2-pass if available and constant quality if not (it may take a few runs to get the bitrate you want, of course).

As you can see in this rant he excoriates almost all encoder comparisons as poorly done and two of the reasons is that they don't use matching key frames for all encoders in the comparisons and they use different rate control methods for all encoders, 2 things he/she flat out states are wrong and invalidate the comparison.

Yet if you read through the MSU documentations and do a google search for the settings Jason/Fiona tells them to use you will see that he has them use CRF and a gop of 500 with a key frame at the same interval.

He basically admits that he/she has cheated on the tests and that they are invalid.

His/her words, not mine.

Yes, it's true, the tests should be controlled. No argument there.

But you're jumping to conclusions, those rambles are in a general context - whereas MSU tests are specific , with a specific set of RULES and guidelines. If one of the rules are guidelines were broken, then yes that would be "cheating"

But you're neglecting the fact that the settings used were submitted by the developers of each encoder themselves. They chose them for those specific encoding scenario. An analogy would be a car race. You have a V8 vs. a V6. Do you deactivate 2 cylinders on the V8 to make it "even" ? Well you might, if the test was for 6 cylinders. And that's the point.

It's not really "cheating" because the test scenario and guidelines are laid out way in advance. And they have been the same for years. For that test you're allowed to submit whatever you think is the "best" for each specific encoding scenario. In case you haven't read it, they set out to test the "best" results regardless of settings, according to that specific test. For example, one test might be speed, one might be SSIM at a given bitrate etc...

If they said, you MUST use XYZ settings, or limit GOP to XYZ value - then yes it would be "cheating". It might be semantics to you, but there is a crystal clear distinction here. It's not a test of SSIM at a set max GOP size. There was no such "rule" or stipulation to those tests. Now you might do some additional tests and test the effect of given max GOP size encodes at various encodes, bitrates, situations. But those are different sets of tests.

And have you tried different very long GOP settings on Mainconcept? The results are terrible, a lot worse. Another way of saying this - Encoder X might get optimized results with a different set of GOP settings, than Encoder Y who might have another optimal range. If you set mainconcept to something like 500 (only the SDK can do this, the "consumer versions" cannot), the keyframe pumping and deterioration is a lot worse

Indirectly, you're raising the point that you could have used "more optimized" settings for some of the other encoders. Perhaps, or maybe you can redo the tests with different settings. But the assumption is the author(s) of each encoder is the most familiar with their own encoder

(And those tests are of limited value anyways, or at least they need to be interpreted properly in context)

[poisondeathray]

As for the claim that "Friends" was encoded with x264, is there any proof of this. I have heard claims over the years that some Blu-Ray here or there was encoded with it, for instance Batman from Warner Bros is rumored to have been encoded with x264 yet I can find no evidence to support such a claim.

You're not going to find very many major titles or big releases on BD that use x264

But the main reason for this is not what you seem to be suggesting . The biggest reason for why it's not used more often is there is no working segment encoding implementation. This is required for studio level encoders. The compressionist will go over the encode multiple times and change parameters over problems sections. Do you think a retail BD is a generic 2 pass encode? No way - at least not for major studio BD releases (some "Indy" releases might not care, or have poor QC). Segment encoding is no problem in BD solutions like CCE-HD, Cinevision, Blu-code etc. You can't do that with x264, and especially not without braking VBV constraints. You can sort of do that with --zones, but it's a painful process and takes much longer, and there is no quick feedback. They apply filters in multiple sections, right in the GUI. Can you do that with x264? Not really, there are some GUI's , some avisynth filters, but no realtime feedback and encoding feedback. Ask any compressionist if they would like to use x264. 9/10 will say yes, if there was a better GUI implentation with segment encoding. Time is money for the studios.

[SameSelf]

You're not going to find very many major titles or big releases on BD that use x264

But the main reason for this is not what you seem to be suggesting . The biggest reason for why it's not used more often is there is no working segment encoding implementation. This is required for studio level encoders. The compressionist will go over the encode multiple times and change parameters over problems sections. Do you think a retail BD is a generic 2 pass encode? No way - at least not for major studio BD releases (some "Indy" releases might not care, or have poor QC). Segment encoding is no problem in BD solutions like CCE-HD, Cinevision, Blu-code etc. You can't do that with x264, and especially not without braking VBV constraints. You can sort of do that with --zones, but it's a painful process and takes much longer, and there is no quick feedback. They apply filters in multiple sections, right in the GUI. Can you do that with x264? Not really, there are some GUI's , some avisynth filters, but no realtime feedback and encoding feedback. Ask any compressionist if they would like to use x264. 9/10 will say yes, if there was a better GUI implentation with segment encoding. Time is money for the studios.

I am glad to see someone confirm what I suspected about how professional studios go about encoding theatrical releases to DVD or BD. Also, I think you meant "breaking VBV constraints". Not trying to nitpick just want to be sure I understand completely the implications in your statement.

As an trained engineer my mind is geared for this sort of minutiae. I have often pined for some sort of tool that allows me to plot various metrics as a function of frame like bit rate. I wonder if there is any effort to build this into x265. Otherwise it is just another black box like every other encoder out there that I have encountered.

[poisondeathray]

You're not going to find very many major titles or big releases on BD that use x264

But the main reason for this is not what you seem to be suggesting . The biggest reason for why it's not used more often is there is no working segment encoding implementation. This is required for studio level encoders. The compressionist will go over the encode multiple times and change parameters over problems sections. Do you think a retail BD is a generic 2 pass encode? No way - at least not for major studio BD releases (some "Indy" releases might not care, or have poor QC). Segment encoding is no problem in BD solutions like CCE-HD, Cinevision, Blu-code etc. You can't do that with x264, and especially not without braking VBV constraints. You can sort of do that with --zones, but it's a painful process and takes much longer, and there is no quick feedback. They apply filters in multiple sections, right in the GUI. Can you do that with x264? Not really, there are some GUI's , some avisynth filters, but no realtime feedback and encoding feedback. Ask any compressionist if they would like to use x264. 9/10 will say yes, if there was a better GUI implentation with segment encoding. Time is money for the studios.

I am glad to see someone confirm what I suspected about how professional studios go about encoding theatrical releases to DVD or BD. Also, I think you meant "breaking VBV constraints". Not trying to nitpick just want to be sure I understand completely the implications in your statement.

As an trained engineer my mind is geared for this sort of minutiae. I have often pined for some sort of tool that allows me to plot various metrics as a function of frame like bit rate. I wonder if there is any effort to build this into x265. Otherwise it is just another black box like every other encoder out there that I have encountered.

That's exactly what it is. Essentially a plot of bitrate, that deals with input vs. output and current buffer level. The tools that analyze this are called "buffer analyzers"

Devices have dedicated silicon hardware chips that have limitations in terms of decoding power, what they can do . Also there are limitations to the physical transfer rate of optical discs. That's why there are "standards" for things such as DVD, blu-ray, and the next gen HEVC UHD blu-ray format . So they can uniformily play without problems. Retail discs are supposed to "just work" out of the box. (I'm discarding some cheap knock offs, or low QC units maybe from some "places", ahm... not to name any countries ) . And yes, x265 does have VBV settings

VBV (video bitrate verifier) can be thought of a method of restricting encoding, so a buffer level can be maintained (no underuns or overruns). e.g When you get buffer underrun, the picture will stutter . The commonly used analogy is the "leaky bucket" model. Water goes in (the input from the optical disc, limited by the max transfer rate". The water in the bucket is the buffer of data. The water dripping out is the data requested for decoding and playback. Empty bucket means buffer underrun = bad

[hello_hello]

http://x264dev.multimedia.cx/archives/472

The article is entitled "How to cheat on video encoder comparisons" and here is a direct quote:

Don’t attempt match compatibility options when it’s reasonable to do so. Keyframe interval is a classic one of these: shorter values reduce compression but improve seeking. An easy way to cheat is to simply not set them to the same value, biasing towards whatever encoder has the longer interval. This is most common as an accidental mistake with comparisons involving ffmpeg, where the default keyframe interval is an insanely low 12 frames.

I'm not sure I agree with your interpretation.
Image you're comparing x264 with x264. The only difference is the first x264 uses a small GOP by default while the second uses a much larger one. Other than that, they're the same encoder, and if the same keyint value is set, they'd produce the same output. So "cheating" would be to use the default keyint values. It'd bias towards the second x264, and obviously produce a misleading result, but that seems to be the extent of the point he's making.

But if someone's comparing encoders and asks, "what settings are recommended for maximum quality at "x" bitrate" and the reply from encoder A developer is -keyint 500, how's that cheating because encoder B takes a quality dive with large GOPS? Wouldn't handicapping encoder A with a small GOP be cheating too?

Your "large GOPS is cheating" argument implies larger GOPS improve the quality for a given bitrate, unless the definition of "cheating" in this context is one with which I'm not familiar, so I'm trying to understand how it doesn't contradict your own use of smaller GOPS for higher quality.

[Cornucopia]

That "bias" is only valid for certain kinds of material.

Another reason why a well-rounded, scientific comparative analysis would use a variety of sample types.

Those tests are valid enough for people who can read between the lines and take it for what it is and no more.

Scott

[Mephesto]

Guys, I think sophisticles is deadrats. Same tired routine of dancing around semantics that made most of us learn to ignore him.

If you're testing purely for quality, all encoders should be configured at settings they work best at. If long GOPs make MainConcept suck, don't use long GOPs. If short GOPs make x264 suck, don't use short GOPs. It's that fücking simple. Handicapping a good encoder so pathetic ones can better compete is not a rule anyone in the business of credibility recognizes.

[sophisticles]

I'm not sure I agree with your interpretation.
Image you're comparing x264 with x264. The only difference is the first x264 uses a small GOP by default while the second uses a much larger one. Other than that, they're the same encoder, and if the same keyint value is set, they'd produce the same output. So "cheating" would be to use the default keyint values. It'd bias towards the second x264, and obviously produce a misleading result, but that seems to be the extent of the point he's making.

But if someone's comparing encoders and asks, "what settings are recommended for maximum quality at "x" bitrate" and the reply from encoder A developer is -keyint 500, how's that cheating because encoder B takes a quality dive with large GOPS? Wouldn't handicapping encoder A with a small GOP be cheating too?

Your "large GOPS is cheating" argument implies larger GOPS improve the quality for a given bitrate, unless the definition of "cheating" in this context is one with which I'm not familiar, so I'm trying to understand how it doesn't contradict your own use of smaller GOPS for higher quality.

You guys seem to be missing the point, in the article DS explicitly says that not using the same settings for all encoders, at least as close as possible, is a form of cheating and thus invalidates the test. Thus it follows that MSU asking the various encoder developers what settings to use in their tests places the start of the test on faulty ground destined to invalidate the results. By the x264 developers complying with the request they are contributing to cheating as defined by DS.

Second, and more importantly, in the article DS explicitly states that for a valid comparison the first choice should be 2 pass vbr:

A good general approach is that, for any given encoder, one should use 2-pass if available and constant quality if not (it may take a few runs to get the bitrate you want, of course).

It doesn't get much clearer than that, by DS' own words if 2 pass is available then it should be used and if it's not available constant quality is used. You will notice nowhere does DS state that constant quantizer should be used and since the article is from 2010 and CRF was introduced back in 2006 I think that means by DS' own standards CRF should not be used for any encoder comparison.

Yet when the MSU people asked what settings to use DS told them CRF 18 and keyint 500, which odd considering that the default is keyint 250. The only reason to specify a larger keyint, per DS' article, is to cheat.

For the test to be valid and fair, the testers should have tested with 2 pass at varying bit rates and also tried different GOP lengths, for instance they could have said a GOP of 50 frames, 100 frames, 200 frames and 300 frames for all encoders. Yes it would have taken longer to test but it would also make the testing and results more rigorous and trust worthy.

As it stands now the MSU tests are worthless, they use a limited sample size of encoders, they use incompatible settings and I'm not too crazy about the test sequences they used either. Instead of samples supplied by 3rd parties I would have liked to if they commissioned their film department (they are a university after all) to go out and shoot some high quality footage and used their own original uncompressed content as masters.

Taking clips that have been around forever and that encoder developers have had time to tune their encoders for and using those as source and then drawing conclusions based on those results smacks of amateur hour and frankly like a thinly veiled attempt at making some easy money by selling their "professional" report and "professional" codec analysis to developers.

[sophisticles]

Guys, I think sophisticles is deadrats. Same tired routine of dancing around semantics that made most of us learn to ignore him.

You are the second person to accuse me of being an expired rodent. Why is that? Is this some kind of inside forum joke that I am not privy to?

[Cornucopia]

We're not missing any point.

But just as there are encoders that do 2pass VBR and encoders that do CQ/CRF, there are tests that use process/methodology constraints and there are tests that use outcome constraints.

The article has many good points to it, but the author can't see the forest through the trees (so it has some rubbish, too).

First rule: there is NO perfect test.

Scott

[hello_hello]

deadrats is a banned poster. He had an extreme bias against x264. A couple of posts into a discussion deadrats is usually resorting so derogatory comments and name calling if anyone disagrees with his viewpoint, so in that respect you don't appear to be him.

You guys seem to be missing the point, in the article DS explicitly says that not using the same settings for all encoders, at least as close as possible, is a form of cheating and thus invalidates the test. Thus it follows that MSU asking the various encoder developers what settings to use in their tests places the start of the test on faulty ground destined to invalidate the results. By the x264 developers complying with the request they are contributing to cheating as defined by DS.

You logic isn't logical to me. Your original contention was the x264 developer cheated by answering a "best settings" question honestly, yet he/she is not involved in the tests. If MSU's methology was flawed, that's completely unrelated.

Second, and more importantly, in the article DS explicitly states that for a valid comparison the first choice should be 2 pass vbr:

A good general approach is that, for any given encoder, one should use 2-pass if available and constant quality if not (it may take a few runs to get the bitrate you want, of course).

It doesn't get much clearer than that, by DS' own words if 2 pass is available then it should be used and if it's not available constant quality is used. You will notice nowhere does DS state that constant quantizer should be used and since the article is from 2010 and CRF was introduced back in 2006 I think that means by DS' own standards CRF should not be used for any encoder comparison.

Once again, cheating implies an unfair advantage. That implies it'd improve the results for the cheater. You're claiming the use of CRF is cheating, but you also say CRF is a bad encoding method so you don't use it yourself. You seem to trying to have it both ways.

I know nothing about the comparisons. I haven't seen them. The methology may be flawed, but that doesn't mean a particular developer cheated by answering a question. In the case of CRF, as far as I'm aware x264 encodes exactly the same way in CRF and 2 pass mode if the resulting bitrate is the same, so in that respect it's not really cheating if bitrate identical encodes were being compared. It'd just mean you could run a CRF encode for x264 and use the resulting bitrate for a 2 pass encode with a different encoder, then compare the quality. That sort of thing. I've not seen the test so I don't know what they did.

I've compared CRF and 2 pass at the same bitrate and as best as I can tell they're virtually the same. There's small differences in bitrate, I assume as 2 pass mode still makes slight adjustments as the encode progresses to hit the target bitrate, but the difference is so small it's not a factor. I've literally watched 2 encodes running together side by side with ffdshow's on screen display reporting the bitrates and for a few minutes 2 pass might be consistently 10kbps higher than CRF, then for a few minutes it'll be around 10kbps lower, then higher again, but that's about it. I've not been able to see quality differences, at least not at the CRF values I use.

[Mephesto]

Does it even matter what DS says? DS is full of shit.

As far as I recall, he did make one TL;DR post about certain exotic coders that have technically beaten x264 and all other H.264 codecs but they cheated because they used exhaustive search techniques which took a few hours to encode a single frame so it took weeks to produce their superior output and the authors were not honest about this before selling their product to illiterate investors.
And I much agreed with his point there.

But whatever he said about MSU cheating for not handicapping certain settings... if he said that, he is full of shit. 'Nuff said.

EDIT: Although I don't like the guy, deadrats shouldn't have been banned. Why validate all his stupidity?

[poisondeathray]

You guys seem to be missing the point, in the article DS explicitly says that not using the same settings for all encoders, at least as close as possible, is a form of cheating and thus invalidates the test. Thus it follows that MSU asking the various encoder developers what settings to use in their tests places the start of the test on faulty ground destined to invalidate the results. By the x264 developers complying with the request they are contributing to cheating as defined by DS.

Second, and more importantly, in the article DS explicitly states that for a valid comparison the first choice should be 2 pass vbr:

A good general approach is that, for any given encoder, one should use 2-pass if available and constant quality if not (it may take a few runs to get the bitrate you want, of course).

It doesn't get much clearer than that, by DS' own words if 2 pass is available then it should be used and if it's not available constant quality is used. You will notice nowhere does DS state that constant quantizer should be used and since the article is from 2010 and CRF was introduced back in 2006 I think that means by DS' own standards CRF should not be used for any encoder comparison.

Yet when the MSU people asked what settings to use DS told them CRF 18 and keyint 500, which odd considering that the default is keyint 250. The only reason to specify a larger keyint, per DS' article, is to cheat.

For the test to be valid and fair, the testers should have tested with 2 pass at varying bit rates and also tried different GOP lengths, for instance they could have said a GOP of 50 frames, 100 frames, 200 frames and 300 frames for all encoders. Yes it would have taken longer to test but it would also make the testing and results more rigorous and trust worthy.

As it stands now the MSU tests are worthless, they use a limited sample size of encoders, they use incompatible settings and I'm not too crazy about the test sequences they used either. Instead of samples supplied by 3rd parties I would have liked to if they commissioned their film department (they are a university after all) to go out and shoot some high quality footage and used their own original uncompressed content as masters.

Taking clips that have been around forever and that encoder developers have had time to tune their encoders for and using those as source and then drawing conclusions based on those results smacks of amateur hour and frankly like a thinly veiled attempt at making some easy money by selling their "professional" report and "professional" codec analysis to developers.

I think you're missing the point

The point of this test is the "best" you can do with any settings. The whole point is to show the encoder at the "best" . That's why they solicited settings from the authors. Non restricted except by things like bitrate, and testing "fast" or "slow" scenarios or whatever condition they sought to investigate.

The other entries could have used any settings, any max keyframe interval. For example , mainconcept could have used 300. But that would negatively impacs their results. Is that "cheating" to set a lower value so their encoder doesn't look as "bad" ?



Of course they could have included other tests, and other stringent testing scenarios with defined parmaters. I mentioned that earlier. It would provide more useful information . But if they did that, they wouldn't need to ask what settings to use from the authors That was the whole point of the test

Of course they should have used other varied sources.

You have to interpet the tests in context. This is what I've been trying to get you to understand all along. Even at the best of times, and specific testing conditions, SSIM is a problematic measure and of limited value. And in actual usage scenarios, who uses --tune ssim ? Almost nobody. So how useful and applicable are these test results to actual usage ? Not very.

[_Al_]

sophisticles,
when you got a time, encode video in this order, not the other way: using CRF first, then calculate average bitrate, or check it in bitrate viewer, then encode same original using 2pass to that average using same settings, look for differences ... did you do test like that? If not you are cheating somehow.

[Steve(MS)]

Guys, I think sophisticles is deadrats. Same tired routine of dancing around semantics that made most of us learn to ignore him.

You are the second person to accuse me of being an expired rodent. Why is that? Is this some kind of inside forum joke that I am not privy to?

He had a mega axe to grind with x264 guy, time will tell if you turn out to be just like him.

You like him seem to bring up personal things about x264 guy, which we don't give a shit about.

I see one person accuse you of deadrats, where is the second one?

Yes, x264 is the best unless one goes to payware.

[sophisticles]

Guys, I think sophisticles is deadrats. Same tired routine of dancing around semantics that made most of us learn to ignore him.

You are the second person to accuse me of being an expired rodent. Why is that? Is this some kind of inside forum joke that I am not privy to?

He had a mega axe to grind with x264 guy, time will tell if you turn out to be just like him.

You like him seem to bring up personal things about x264 guy, which we don't give a shit about.

I see one person accuse you of deadrats, where is the second one?

Yes, x264 is the best unless one goes to payware.

What "personal" thing did I bring up against one of the developers? The fact that he now goes by a girl's name, changed all the copyright's on his commits and has dressed up like a girl for a pic to use as an avatar? Someone else asked if he was undergoing a sex-change and I pointed out that DS decided to live out his/her life in a very public way. How is that my bad? He/she decided to make it public knowledge and beat people over the head with his lifestyle changes. I personally didn't care but when he/she decides to be so open about it then it's fair game for commentating as far as I'm concerned.

The first person accused me in PM, at first I thought it was some kind of death threat, an odd one. Now that I have been informed I did a forum search and can understand the confusion. This deadrats fellow did seem to have a posting style eerily similar to mine and we seem to share a number of viewpoints, though I can't find any reason for him being banned.

Is this a forum where no one is allowed to criticize open source projects or point out any weaknesses? Does that somehow offend some open source advocates. I seem to be sensing a significant amount of hostility for pointing out things concerning x264 that are well establish and obvious to anyone that has done any extensive encoding.

Lastly, I was under the impression that cursing was not allowed on this forum yet thus far I have seen the "F" word allowed as well as the "S" word, just to be clear is cursing allowed but analysis or non-flattering comments about x264 disallowed.

There seems to be a very odd dynamic at play in this forum, reminds me of the way Doom9 used to be, maybe it still is I stopped going there years ago. That forum had a very similar dynamic, anything that was viewed as anything but glowing praise about various open source projects was instantly greeted with hostility and accusations of "trolling".

If you asked a question about AviDemux, x264, xvid, any open source project you immediately had to add something along the lines of "this is the best software or thank you for the great quality" as a qualifier or else posters would descend on you to flame you mercilessly.

I'll leave this discussion, it seems that anything other than glowing praise for x264 is negatively received in this forum.