high quality mpeg-2 encoding

[jagabo]

I wonder if the problem with my x264 encode had more to do with the old version of x264 -- 0.65.1061 according to the --version report. I tried changing the settings to more closely match PDR's. At CRF 20 the file was a few percent smaller and looked pretty much the same. At 2-pass VBR 3000 the file was a little bigger (slightly higher bitrate than the CRF encode) but still looked about the same.

[creamyhorror]

jagabo: Maybe you could screenshot the same frame for a comparison? Even if it isn't exact, we'll be able to see the level of grain/detail preserved. My current Windows installation is completely messed in terms of codecs so I can't do it myself.

So I guess we're waiting for deadrats to post encodes of his Pioneer demo clip? Are we going to go ahead and compare the Mainconcept, Sorenson, and Apple encodes?

[poisondeathray]

jagabo: Maybe you could screenshot the same frame for a comparison? Even if it isn't exact, we'll be able to see the level of grain/detail preserved. My current Windows installation is completely messed in terms of codecs so I can't do it myself.

So I guess we're waiting for deadrats to post encodes of his Pioneer demo clip? Are we going to go ahead and compare the Mainconcept, Sorenson, and Apple encodes?

Be careful. It's important to look at the whole thing, not just single frame comparisons. When using with different settings or patches , some frames get better, but some get worse - that's the nature of the beast. Some builds break things, some fix things. I can show you examples where using mbtree does significantly worse on fades and dark scenes. As I said before, I cherry picked that frame as an illustration. x264 is non deterministic when using threads >1, so even using the same settings on the same computer and setup you might get slightly different results on 10 different encodes each time. It's also important to look at different sources, different bitrate ranges, different settings before even attempting to draw some preliminary conclusions. Remember this is just 1 observation at 1 bitrate on 1 specific clip. Be careful not to generalize anything from that.

But I can tell you after 100's of tests on different sources, x264 performs better than any of those encoders you've mentioned at any bitrate range if you use proper settings. I've posted several comparison threads if you search, and many others have as well. Search the Doom9 forums as well. I have yet to find and example where it performs consistently worse, I'm still looking. Remember I used to hate x264 a few years ago, largely because it oversmoothed; I was skeptical at first, but it really has developed into a very nice, configurable encoder.

But please don't believe me or what others say - You really have to do these comparisons for yourself to see it for your own eyes, on your own learning journey - it's simply that much better. And if you do find that scenario where x264 performs significantly worse as some people say (but never provide any proof), post the source, settings used, and your workflow. It might be something as simple as user error, and if not, at the very least we can help improve the encoder

[creamyhorror]

Be careful. It's important to look at the whole thing, not just single frame comparisons. When using with different settings or patches , some frames get better, but some get worse - that's the nature of the beast...

I understand all of that, but since you've already done a comparison of two particular frames, why not finish the job with the other encodes that deadrats did? In the first place, this was meant to be a comparison of various encoders to put to rest (inasmuch as is possible) questions of encoder capability.

But I can tell you after 100's of tests on different sources, x264 performs better than any of those encoders you've mentioned at any bitrate range if you use proper settings. I've posted several comparison threads if you search, and many others have as well. Search the Doom9 forums as well. I have yet to find and example where it performs consistently worse, I'm still looking. Remember I used to hate x264 a few years ago, largely because it oversmoothed; I was skeptical at first, but it really has developed into a very nice, configurable encoder.

Really, I'm not in need of persuasion myself (x264 is the only H.264 encoder I've ever used), but I'll do the researching you recommend. If it matters, I've read the various MSU codec shootout reports and am relatively certain of x264's superiority; I'm just interested in seeing this particular comparison carried out, to put to rest (or otherwise) claims of x264's tendency to smear or oversmooth that have been made in this forum.

[poisondeathray]

I understand all of that, but since you've already done a comparison of two particular frames, why not finish the job with the other encodes that deadrats did? In the first place, this was meant to be a comparison of various encoders to put to rest (inasmuch as is possible) questions of encoder capability.

If you're still talking about the 1st test clip, if you look above, I've posted comparisons shots along with my x264 and xvid encodes in the zip file. These were full uncropped pngs (I only cropped in the 2nd post because the browser was resizing.) But since it was determined fairly early on that deadrats didn't use decent settings for any of the encoders - and thus not representative of the each encoder - i.e. It's not really a useful comparison when you handicapp all the encoders. So I didn't bother looking at the 2nd test where he resized it to 720x480 16:9.

I already did the full 1280x720 with Divx h.264, and posted a screenshot - that was back on page 2 (before it was decided to resize to 960x544). If you want, I can post the same encode using proper settings with Mainconcept Reference, Apple h.264 ? I can tell you ahead of time (having used these so often), what the results will be...

I did an r1061 encode. Here is the same frame , cropped, same colormatrix as the cropped examples above. I honestly don't know why jagabo is getting those results with his revised settings. Maybe the GUI he mentioned he was using is outdated (I've never heard of it before) and not passing things through correctly? Do you get the same results with command line jagabo? But looking at the other frames, you can tell the more recent version is slightly better on most of frames - again the differences are tiny on most frames, and not observable at realtime

x264 r1061

[jagabo]

My original bad result for that particular frame seems to be mostly from using CRF mode and the old encoder. Simply changing to 2-pass VBR cleaned up a lot of the posterization and restored a lot of the grain.

x264gui generated this command line for the CRF 20 encode:

@x264.exe --progress --no-psnr --no-ssim --cqm flat --mixed-refs --scenecut 40 -I 250 -i 25 --sar 1:1 --threads auto --nr 0 -f -2:-1 -b 5 --direct "none" --b-adapt 1 --b-bias 0 --b-pyramid -w --deadzone-inter 21 --deadzone-intra 11 --aq-mode 1 --aq-strength 1 --chroma-qp-offset 0 --direct-8x8 1 --level 4.1 --psy-rd 1.0:0.0 --crf 20 --8x8dct -A p8x8,b8x8,i8x8,i4x4 -m 7 --me umh --merange 16 --qpmin 10 --qpmax 51 --qpstep 4 --qcomp 0.600000 --ipratio 1.400000 --pbratio 1.300000 --vbv-init 0.9 --vbv-maxrate 24000 -r 8 -t 1 -o "G:\lagarith.mp4" "G:\lagarith.avs" 2>&1|lputs.exe "G:\Program Files\x264gui\log\lagarith.avs.txt"

Note that was with 8 reference frames and 5 b frames but the particular frame in the output file was almost as bad as my earlier post.

[deadrats]

ok, after a couple of more encodes i have come to the following conclusions:

1) i really hate tmpg express with a passion. i don't know what the hell is wrong with this app but i set it up to do a 2 pass encode with a specific target bit rate and often the resulting bit rate is way off.

2) tmpg express is slow beyond belief, doing a 2 pass 2500 kbps vc-1 encode of a 9 min 16 sec clip took over 2 hours and 40 minutes, to put that performance into perspective if performance scaled with cores my e7400 i would need to 17 dual core processors or a 34 core single cpu to do roughly real time vc-1 encoding.

3) i hate badaboom, i never realized how crappy the encodes it produces really were.

4) it's an absolute waste of time to transcode a file from one format to another. when you consider the amount of time it takes to transcode and the fact that under ideal circumstances the best you can hope for is to produce a file almost as good as the original, you are better off buying another hard drive. really what's the point of taking a pristine 1080i file and transcoding it to a smaller size when for $100 you can just pick up a 1.5tb hdd and simply keep the original?

regardless, here's the 2 test clips i did using tmpg express, one is vc-1 and one is main concept's h264:

http://www.sendspace.com/file/46f2vw

http://www.sendspace.com/file/e0gi63

[poisondeathray]

My original bad result for that particular frame seems to be mostly from using CRF mode and the old encoder. Simply changing to 2-pass VBR cleaned up a lot of the posterization and restored a lot of the grain.

x264gui generated this command line for the CRF 20 encode:

@x264.exe --progress --no-psnr --no-ssim --cqm flat --mixed-refs --scenecut 40 -I 250 -i 25 --sar 1:1 --threads auto --nr 0 -f -2:-1 -b 5 --direct "none" --b-adapt 1 --b-bias 0 --b-pyramid -w --deadzone-inter 21 --deadzone-intra 11 --aq-mode 1 --aq-strength 1 --chroma-qp-offset 0 --direct-8x8 1 --level 4.1 --psy-rd 1.0:0.0 --crf 20 --8x8dct -A p8x8,b8x8,i8x8,i4x4 -m 7 --me umh --merange 16 --qpmin 10 --qpmax 51 --qpstep 4 --qcomp 0.600000 --ipratio 1.400000 --pbratio 1.300000 --vbv-init 0.9 --vbv-maxrate 24000 -r 8 -t 1 -o "G:\lagarith.mp4" "G:\lagarith.avs" 2>&1|lputs.exe "G:\Program Files\x264gui\log\lagarith.avs.txt"

Note that was with 8 reference frames and 5 b frames but the particular frame in the output file was almost as bad as my earlier post.

So that was from the 1st encode? I don't see --b-adapt 2 being passed . --b-adapt 2 is the new b-frame decision algorithm and it's way more accurate, but not multithreaded yet and thus much much slower.

Another difference is you used a DXVA mode with VBV, which potentially hurts quality (it never helps, it can only hurt - if you are not encoding for device compliance or a specific target, it shouldn't be used). vbv and crf never quite worked well together properly (you'd get buffer over or underruns), the only 100% sure method was to use 2pass for DXVA compliant encodes. There have been several fixes to VBV since r1061, and the commits this week supposedly even make it work in crf mode properly which is an absolutely amazing feat. This means things like realtime streaming with proper vbv are possible. This is because one of the developers works for Avail who actually uses x264 for their broadcast technology

[poisondeathray]

4) it's an absolute waste of time to transcode a file from one format to another. when you consider the amount of time it takes to transcode and the fact that under ideal circumstances the best you can hope for is to produce a file almost as good as the original, you are better off buying another hard drive. really what's the point of taking a pristine 1080i file and transcoding it to a smaller size when for $100 you can just pick up a 1.5tb hdd and simply keep the original?

100% agree! I've been saying this for quite a while now

There are some scenarios where you might encode: e.g. encoding for flash, websites, making camcorder home movies like blu-rays or BD5/9. All of these have a fixed capacity scenario, so finding efficient encoders is important

[jagabo]

tmpg express is slow beyond belief

It always has been one of the slowest encoders.

doing a 2 pass 2500 kbps vc-1 encode of a 9 min 16 sec clip took over 2 hours and 40 minutes

That's more likely Microsofts fault.

3) i hate badaboom, i never realized how crappy the encodes it produces really were.

That's what I've heard.

4) it's an absolute waste of time to transcode a file from one format to another. when you consider the amount of time it takes to transcode and the fact that under ideal circumstances the best you can hope for is to produce a file almost as good as the original, you are better off buying another hard drive.

I pretty much agree.

[creamyhorror]

x264gui generated this command line for the CRF 20 encode:

@x264.exe --progress --no-psnr --no-ssim --cqm flat --mixed-refs --scenecut 40 -I 250 -i 25 --sar 1:1 --threads auto --nr 0 -f -2:-1 -b 5 --direct "none" --b-adapt 1 --b-bias 0 --b-pyramid -w --deadzone-inter 21 --deadzone-intra 11 --aq-mode 1 --aq-strength 1 --chroma-qp-offset 0 --direct-8x8 1 --level 4.1 --psy-rd 1.0:0.0 --crf 20 --8x8dct -A p8x8,b8x8,i8x8,i4x4 -m 7 --me umh --merange 16 --qpmin 10 --qpmax 51 --qpstep 4 --qcomp 0.600000 --ipratio 1.400000 --pbratio 1.300000 --vbv-init 0.9 --vbv-maxrate 24000 -r 8 -t 1 -o "G:\lagarith.mp4" "G:\lagarith.avs" 2>&1|lputs.exe "G:\Program Files\x264gui\log\lagarith.avs.txt"

Note that was with 8 reference frames and 5 b frames but the particular frame in the output file was almost as bad as my earlier post.

What PDR said about VBV messing up CRF seems to be the most likely explanation for the difference. It appears that MeGUI's DXVA profiles all use VBV options, so they're not suitable for CRF encoding (until you update to the latest x264 revision). To be sure, I'll try to do a CRF 20 encode without VBV - probably using the profile "Unrestricted 1pass Const. Quality Extra Q.".

[creamyhorror]

I did a bunch of test encodes at various settings and came to the conclusion that the difference between jagabo's and PDR's screenshots comes from x264 allocating bits to that frame quite differently.

I made two encodes with identical settings, except that one was 2-pass and the other was CRF. The final bitrates were the same. Take a look at this frame:




Commandlines

You'll see that the mean quantizations and coded sizes are quite different. x264 allocated bits quite differently in these two cases, just like PDR said in an earlier message. I checked the surrounding frames in both encodes, and both had an identical sequence of frametypes - however, the coded size for the CRF was smaller (and quantization higher) for many of the frames.

When I compared random other shots in the encodes, they didn't appear to be differ much.

I'm not sure what to make of this...is 2-pass making a better decision than CRF in this case? What's being traded off? Did CRF mess up somehow?


edit: all I could get from Dark Shikari was that CRF and 2-pass use different metrics in distributing quality, though with mbtree on there shouldn't be as much of a difference. Guess it's just one of those things.

[poisondeathray]

Which version of jagabo's encode, and which x264 version are you referring to?

Do your screenshots and commandlines refer to the same x264 version? ie. you're just comparing 2pass vs. crf on the same x264 revision?

If so, what was you final bitrate or filesize for the crf encode? If it doesn't match the 2pass version, they are not comparable (you might have to do several runs eg. maybe crf 19.1 , 19.5, etc... until they match)

[creamyhorror]

Commandlines, screenshots were all done by the same revision (1817), by me. As I said, I matched the bitrates (by doing several runs):

2-pass
Overall bit rate : 2995 Kbps
Writing library : x264 core 68 r1183M f21daff
Encoding settings : cabac=1 / ref=8 / deblock=1:-2:-1 / analyse=0x3:0x133 / me=umh / subme=9 / psy_rd=1.0:0.0 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / chroma_qp_offset=-2 / threads=3 / nr=0 / decimate=0 / mbaff=0 / bframes=6 / b_pyramid=1 / b_adapt=2 / b_bias=0 / direct=1 / wpredb=1 / keyint=250 / keyint_min=25 / scenecut=40 / rc=2pass / bitrate=3000 / ratetol=1.0 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / cplxblur=20.0 / qblur=0.5 / ip_ratio=1.40 / pb_ratio=1.30 / aq=1:1.00

Filesize: 11,360 KB
SSIM: ~0.98776

CRF
Overall bit rate : 3019 Kbps
Writing library : x264 core 68 r1183M f21daff
Encoding settings : cabac=1 / ref=8 / deblock=1:-2:-1 / analyse=0x3:0x133 / me=umh / subme=9 / psy_rd=1.0:0.0 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / chroma_qp_offset=-2 / threads=3 / nr=0 / decimate=0 / mbaff=0 / bframes=6 / b_pyramid=1 / b_adapt=2 / b_bias=0 / direct=1 / wpredb=1 / keyint=250 / keyint_min=25 / scenecut=40 / rc=crf / crf=18.0 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / ip_ratio=1.40 / pb_ratio=1.30 / aq=1:1.00

Filesize: 11,453 KB
SSIM: ~0.986, need to check

[poisondeathray]

OK, DS is the boss , he would know best

[jagabo]

I'm not sure what to make of this...is 2-pass making a better decision than CRF in this case? What's being traded off? Did CRF mess up somehow?

Well, I can see how that frame would be different. It's the last frame of a quick fade over four frames and the next frame is a different shot. It doesn't surprise me that it would be considered unimportant to assign it much bitrate.

[creamyhorror]

regardless, here's the 2 test clips i did using tmpg express, one is vc-1 and one is main concept's h264:

http://www.sendspace.com/file/46f2vw

http://www.sendspace.com/file/e0gi63

As PDR said, it would have been better for someone to do a deinterlace for the source first and then letting everyone encode the processed clip. Different deinterlacing methods are going to make a lot of difference for image quality.

i know there are some in here that will consider what i am about to say heresy but i personally don't find x264 all that impressive. over the years i have seen hundreds of encodes from blu-ray rips done with x264 and for the most part i find the encodes to be of mediocre quality at best, the only encodes i have ever seen that looked good were from a "scene group" that used a custom matrix, all other encodes the color saturation seemed off, the encode wasn't crisp.

my own encodes, using either avidemux with the built in x264 or the x264 vfw with tmpg express, with all the options for quality maxed out (perhaps my understanding of the options is lacking) resulted in worse results than when i did the same encodes using xvid and choosing the "hi def" profile or with divx and choosing the "insane quality" setting.

my personal feeling is that if x264 was anywhere near as good as it's proponents make it out to be then the studios would be using it for there own blu-ray encodes rather then spending $1000 per seat on apple's h264, $40000 per seat on panasonic's h264 or $70000 per seat on CCE's h264 encoders.

We've tested XviD and it looks like x264 wins. But we've not done Apple's H.264.

I dare you to use decent settings, with the same source, comparing x264, Apple compressor, and Divx h.264 encoder, then come back and we can discuss it.

We haven't tested DivX 7 either.

I'll try to get the Pioneer clip and deinterlace it, unless you want to, jagabo. If I deinterlace it should I use nnedi2 or a bobber?


edit: That demo is really long at 9min. I suggest cutting a segment or two of it.

[poisondeathray]

We've tested XviD and it looks like x264 wins. But we've not done Apple's H.264.

I don't think there was ever any question about XviD vs. x264 in terms of quality (at any equivalent bitrate). There might be 1 or 2 people that think otherwise. Or maybe 1 now...

We haven't tested DivX 7 either.

I've tested it extensively. It's based on the Mainconcept SDK, but a dumbed down version. Very few settings or control. On this testclip I already posted a sample screen vs. x264 back on page 2, but it was done from the original source, not the resized lagarith encode. I can repeats the tests on the same resized clip, or you can test it yourself on the other clip below.

Apple h.264 is limited to AVC Main, (no I8x8 or CQMs), and forces Apple's own custom color matrix. The color coefficients and equations do not match any known standard and you can't get around this, this has been discussed in several forums before. It's also difficult to use on Windows, because you can't frameserve and it only accepts .mov wrapped videos. So often what I've ended up doing in the past is converting it to a lossless .mov just for the Apple test.

I'll try to get the Pioneer clip and deinterlace it, unless you want to, jagabo. If I deinterlace it should I use nnedi2 or a bobber?
edit: That demo is really long at 9min. I suggest cutting a segment or two of it.

I found links to that clip (not my links), and the original can be found here
http://www.whereweget.com/2007/12/05/pyipioneer-hd-1.4gb-demo.html

I prepared a test clip, 1001frames, 960x544, encoded with FFV1 (It's a lossless codec, comes with ffdshow, but slightly more compressed for YV12 clips than lagarith). 275MB, no audio. It was deinterlaced with yadif, resized with lanczos. The frame numbers correspond to 2100-3100 in the original clip. The original clip is pretty average quality, what you would typically get from a consumer level camcorder (19Mbps MPEG2), and there is pixellation in some of the scenes, e.g. with moving water, in the pool, etc... For this test clip, I tried not to use those scenes with alot of pixellation (but there still is a tiny bit), but I used no processing (like deblocking filters, etc.). So if you want, you can use the same parameters as deadrats suggested: 2500kbps, no filters (to be consistent), or anything else you decide. If you want another section cut , or other specifications let me know and I can prepare a different clip.

(And yes, I included PG scenes with deadrats' purty girlfriend)

http://www.megaupload.com/?d=Q743FWBC

[creamyhorror]

I've tested it extensively. It's based on the Mainconcept SDK, but a dumbed down version. Very few settings or control. On this testclip I already posted a sample screen vs. x264 back on page 2, but it was done from the original source, not the resized lagarith encode. I can repeats the tests on the same resized clip, or you can test it yourself on the other clip below.

To be honest I'd rather let someone who's actually familiar with DivX 7/Mainconcept to do the encode. I'm too lazy to fiddle with it

Apple h.264 is limited to AVC Main, (no I8x8 or CQMs), and forces Apple's own custom color matrix. The color coefficients and equations do not match any known standard and you can't get around this, this has been discussed in several forums before.

I've heard about this...it sounds like a mess.

It's also difficult to use on Windows, because you can't frameserve and it only accepts .mov wrapped videos. So often what I've ended up doing in the past is converting it to a lossless .mov just for the Apple test.

Ditto.

Thanks for preparing the test clip, I also downloaded the source but I guess you've handled it.

deadrats/puzzler/anyone, think you want to have a go at the clip?

[jagabo]

I just got around to viewing deadrats' MainConcept h.264 encoded "bikini" video. Is it just my system or is this video completely screwed up? About the time you get to the bikini shot there is a strobing of low extra low quality about once a second.

PDR's FFV1 source (VirtualDub):


Same frame from DR's version (AviDemux):


Next frame from DR's version (AviDemux):


Levels differences may be because of the different decoders.

[creamyhorror]

I just got around to viewing deadrats' MainConcept h.264 encoded "bikini" video. Is it just my system or is this video completely screwed up? About the time you get to the bikini shot there is a strobing of low extra low quality about once a second.

Confirming this on my system. I encountered it but I didn't say anything because I thought it might be VLC messing up. The pulsing continues through most of the video and there's no sound(?).

[poisondeathray]

OK I looked at his MC video. He used poor settings, so it's not representative of MC's potential

1) No b frames. Major loss of efficiency

2) 1 reference frame. Major loss of efficiency

3) Short GOP. Major loss of efficiency, especially on this type of material. He used a max GOP size of 33 (which is the MC's default). This is different material than typical trailers with lots of transitions. There's slow pans, and few scene changes. You can gain lots of efficiency by upping the interval and filling them with B and P's

The strobing occurs exactly on I-P transitions. The "worse" frames are actually low quality I frames, this progressively worsens, as the low quality I frames serve as reference for following P's. The "weight" of I frames are typically higher than others (they should look like peaks), but since his other settings were so bad, they were cut down to fit the bitrate constraints.

Anyways, I suggest maybe doing some tests at a range, maybe 2.5, 3, 3.5 Mbps. He originally suggested 2.5Mbps, but his encode was done at 2Mbps. I did some tests and 2.5Mbps is fairly low for this sequence, and shows the differences quite easily. The key concept here is, I repeat, test using decent settings. If you handicapp the encoder, you can't draw any proper conclusions from that

[jagabo]

OK I looked at his MC video.

What tool are you using to do this?

[poisondeathray]

OK I looked at his MC video.

What tool are you using to do this?

I usually look at it just normally through a media player, or through avsp if parsing frames. But right away you know something is wrong when you open his video...so you have to look for reasons why

You can use ffdshow's osd , like creamyhorror showed in a post above, it shows frametype and allocation for each frame

I use Elecard Streameye. It's not free, but if you're serious into this type of stuff, it's a great tool, and provides more information than the ffdshow osd.


deadrat's distribution. Not there is hardly any fluctation. It's a CBR video.



a more "typical" 33 GOP size distribution (done on the FFV1 video with MCR)

[jagabo]

OK I looked at his MC video.

What tool are you using to do this?

I usually look at it just normally through a media player, or through avsp if parsing frames. But right away you know something is wrong when you open his video...so you have to look for reasons why

You can use ffdshow's osd

Thanks, AvsP and ffdshow get me enough information.

[creamyhorror]

PDR, you have Streameye? You're hardcore. I'll have to be satisfied with ffdshow I guess. Does AvsP actually have some frame analysis capability? I never knew, despite using it often.

Thanks for checking out deadrats' file for us. It looks more and more like this comparison isn't going to happen

[jagabo]

Does AvsP actually have some frame analysis capability?

I think he was just using it for scrubbing through the video.

[creamyhorror]

Oh okay.

[jagabo]

But I'm not sure!

[poisondeathray]

Yes, thx jagabo , I meant just for scrubbing & visual comparison of frames

For anyone else reading who doesn't use AvsP , it's a great tool because you can quickly compare between encodes by tabs, and switching tabs by hitting the number keys. The only "quirk" is you have to set it up correctly so each video is decoded with frame accuracy and have the same #of frames. So this means DirectShowSource() is usually out of the question.

Another useful method is to Interleave() in avisynth, and you can use a tiny transparent overlay using Overlay() to label each video. Temporal changes are easier to spot with this method

On very similar encodes, you can use difference masks visualize the differences. So 100% perfect lossless qualilty would look black, but differences show up as non black. This is essentially what the objective PSNR testing does, when averaging the results and spitting out a number. But it's sometimes nicer to visualize where the compression losses /artifacts are occurring

It looks more and more like this comparison isn't going to happen

What do you mean? There is no reason why other people can't post encodes using decent settings. I can post proper encodes for the other encoders; I'm quite familiar with most of them. We are just doing this to make some observations and learn things right?