I'm starting this thread because I don't want to hijack any other thread and to share some thoughts on doing a proper encoder test.
Rule 1: No short clips! Video encoders have all sorts of analysis algorithms that look for redundancies, in the case of lossless encoders they look for redundancies that can be reversibly discarded, i.e. you can reconstruct the data that was thrown out; in the case of lossy encoders they look for redundancies that can be discarded with the least amount of quality/detail loss as possible. A short clip doesn't allow the algorithms to be used to their fullest potential and thus it can have an unfair impact on the test.
Rule 2: No lossy previously compressed clips! Every encoder is free to decide on it's own what details to discard and what details to keep, within the parameters establish by the settings used; the details one encoder decides to discard may not be the details a different encoder may decide to discard. As such it's not fair to use a source compressed with encoder "A" and then run an encoder comparison where that same encoder is one of the participants because that encoder is likely to make the same or at the least similar decisions as when it compressed the source in the first place. This means that Encoder "A" will likely discard less details than the other competitors thus making it look like encoder "A" is the superior encoder.
So, with these 2 basic rules in mind, what do I consider a proper source for testing? A DVD? No! A Blu-ray? Nope! Something downloaded from insert-site-here? No way!
The sad truth is that there are very few sources that I consider the be of sufficient quality for a proper test, here are some of my favorites:
A few words about Tears of Steel, this is an excellent movie to have on your hard drive, it has close ups, special effects, great lighting, more importantly you can download the whole movie in a y4m format or DCP format. The live action footage was filmed using a Sony F65 CineAlta, the source footage is 4TB's of OpenEXR half float file encoding, ToS was edited and produced with Blender, the master was rendered in y4m DCI 4k CinemaScope cropped, this version is available and linked to above, it's 66GB in a .xz archive and 185GB when extracted.
On my system, 16GB DDR3, a Xeon equivalent to the i7 4790 and a Radeon R7 265, it's impossible to play this version smoothly or edit it smoothly if it's on a traditional rotating disk hard drive; only if I put it on my 960GB SanDisk SSD connected via a PCI-E card can I play it smoothly or edit it in a NLE.
Also, thanks to the fact that it's uncompressed, there is no decoding bottleneck, it does aid in showing where possible bottlenecks exist in the video processing pipeline, for instance with x264 + medium with no resizing, this 4 core 8 thread cpu will not hit 100% utilization, though it will come close. If a resize down to 720p is performed, the cpu utilization is less than 50%. Likewise with x265.
There is one other source that ranks right up there with ToS and that's Valkaama. What makes this a must have is that like ToS they too released all their sources, they shot the movie using HDV cameras and they have a torrent file that downloads all the scenes in an unprocessed condition. What they did was film each scene using multiple cameras for multiple angles, each scene has multiple angles. Not all scenes or angles are included in the movie meaning you could take all the source footage and produce more than a few new different movies with the footage they shot.
Though not as high quality footage as ToS, the fact that it's unprocessed and untainted, no post production filtering, nothing, just high quality footage used in a movie before it's been tampered with.
If anyone else knows of sources like these, that are at mastering quality or above, please share in this thread.
+ Reply to Thread
Results 1 to 9 of 9
This is true but not practical for a regular user. Back in the day I would use platformer game recordings as 'pure' sources since they were cost effective but they are so far removed from what typical content looks like that they're useless. Other options are 1080p short movies encoded at high bitrates. Once you halve the resolution, the tiniest compression artifacts that might've existed are pretty much gone.
Testing very high quality sources is only a subset of scenarios. Yes for sure it's important to do, but if that's all you do, you will only come to a rudimentary understanding of how an encoder performs. You need to test a variety of scenarios and sources.
You can construct a test however you like. As long as details are provided, and you can justify your choices or rationale. You just have to be transparent with the testing methodology and state your assumptions, and what you're actually trying to test, and to be careful on how to extrapolate from those tests. For example, you wouldn't be able to apply your observations and interpretations from a super high quality source professionally shot and lit source, necessarily to , say a noisy, shaky consumer handy cam video. If I'm a VOD company with broadcast sources (these are roughly worse than retail BD quality) asking about an encoder comparison, your test observations might not be completely applicable ,etc...
I don't think either of your rules are valid for real world testing.
Rule 1: You're saying that if I encode 10,000 frames of a clip using x264 2 pass encoding, split off the first 1000 frames and determine the average bitrate used for those, then encode the first 1000 frames again while specifying that average bitrate, the second time they'll be encoded quite differently? If that was the case, would it be unreasonable to say something's wrong?
What if I use CRF encoding? If I encode 10,000 frames at CRF18, will the first 1000 frames be encoded differently than if I'd only encoded them on their own? Rule 1 possibly applies to using average bitrate encoding, but that's about it, and if you're going to use average bitrate encoding, which I never have, you'd want to be throwing enough bits at the video to ensure a high quality, even if a large proportion of them are effectively wasted.
Rule 2: Why would you only want to test encoders using only the cleanest sources, and why wouldn't you test them using the same sources you're going to re-encode in the real world. Most of us re-encode lossy sources. I don't care too much how an encoder might re-encode a lossless source direct from a camera, whereas I care a lot more about how an encoder will encode my TV captures, DVD and Bluray video etc.
A source is a source. The encoder is simply fed uncompressed video. It shouldn't need to know it was uncompressed from a lossless source to encode it well.
Content selection is important - mentioned sources are CG and as such they may give biased results - various sequences should be used - preferably lossless 4:4:4 and sufficiently oversampled.
The only fair way to test is if the source is either uncompressed or encoded with a non-competing encoder.
According to what I've read, it makes very little difference anyway.
On one end of the scale I might have a noisy old VHS capture I need to encode, while on the other end there's a pristine, never before compressed source. The average video I encode probably sits somewhere in the middle, but if you want some lossless sources I can capture a few VHS tapes for you.