X265 "Lossless" not really lossless and I can prove it

[sophisticles]

So here's the deal, currently have a GTX960 and decided to buy mini1050 for $115 when payday comes around. In preparation of this change I have been doing a bunch of tests of the GTX960's nvenc as a baseline to compare how the Pascal powered 1050 performs quality wise, namely I wanted to see if the 1050 offered better quality because it executes the same settings better, is it because it offers more settings than the 960 or both.

So using a static build of ffmpeg and Ubuntu 16.04 LTS and the latest NVIDIA drivers I transcoded a slew of files and used ffmpeg to calculate baseline PSNR and SSIM values. As a reference point I decided to include some x264 ultrafast and medium encodes and some x265 ultrafast encodes, it was when I decided to do a bunch of lossless encodes that I realized something was wrong.

Using a 2.3gb 1080p source file I transcoded it to nvenc_h264 lossless (the 960 doesn't support H265 lossless) and the resulting files size was 30.9gb.

The x264 ultra fast lossless was 28.6gb, with the medium preset it drops down to about 22gb. Then I ran into something really odd, the x265 ultra fast encode was only 14gb, less than half the size of the other 2.

Using ffmpeg to caluclate PSNR and SSIM for all the files i got the following values:

NVENC H264 Lossless
[Parsed_ssim_0 @ 0x27956a0] SSIM Y:1.000000 (inf) U:1.000000 (inf) V:1.000000 (inf) All:1.000000 (inf)

[Parsed_psnr_1 @ 0x2795d40] PSNR y:inf u:inf v:inf average:inf min:inf max:inf

X264 Lossless Ultra Fast
[Parsed_ssim_0 @ 0x37958a0] SSIM Y:1.000000 (97.759916) U:1.000000 (inf) V:1.000000 (inf) All:1.000000 (99.656556)

[Parsed_psnr_1 @ 0x3796040] PSNR y:inf u:inf v:inf average:inf min:inf max:inf

X265 Lossless Ultra Fast
[Parsed_ssim_0 @ 0x4133a40] SSIM Y:0.996310 (24.329467) U:0.997240 (25.590147) V:0.997184 (25.503329) All:0.996610 (24.698549)
[Parsed_psnr_1 @ 0x41340e0] PSNR y:53.354600 u:55.613567 v:55.443195 average:53.965527 min:51.749530 max:72.329853

The only values that make sense are the NVENC Lossless values, SSIM has a range between -1 and 1, with -1 being completely different and 1 being exactly the same and the decibel value for both PSNR and SSIM should be infinite if the encode is truly lossless.

Even x264 isn't really lossless but it's close enough that we can let it slide but the x265 values really surprise me.

Anyone want to run similar test with x265 on test samples they have and see if they see similar results.

For the record, lossless encoding for x264 and x265 where done with CRF 0.

[LigH.de]

For the record, lossless encoding for x264 and x265 where done with CRF 0.

And that's your first mistake. "--crf 0" does not mean "lossless encoding", for neither encoder.

For x264, use:

      Lossless:
            x264 --qp 0 -o <output> <input>

For x265, use:

   --[no-]lossless               Enable lossless: bypass transform, quant and loop filters globally. Default disabled

[jagabo]

Every time I've used --crf=0 x264 has automatically encoded as --qp=0. In fact, any CRF value below 1.0 is translated to qp=0.

[SameSelf]

decided to buy mini1050 for $115 when payday comes around

lol, what? are you 16 with an after school job?

[LigH.de]

Every time I've used --crf=0 x264 has automatically encoded as --qp=0.

That could just be "luck". I bet there are "academic examples" of videos where this is not the case...

In fact, any CRF value below 1.0 is translated to qp=0.

Not certainly. The calculation of a rate factor requires transformations which could be skipped a priori when a specific lossless mode is enabled.

Additionally, GPU encoders will not work exactly like x264/x265.

P.S.: AFAIK, SSIM has a range of 1.0 (identical) to 0.0 (completely different).

[pandy]

decided to buy mini1050 for $115 when payday comes around

lol, what? are you 16 with an after school job?

FYI for approx half of Europe monthly incomes are bellow 600$.

[sneaker]

--crf 0 is lossless for 8 bit x264. (--qp 0 is lossless for all bitdepths)

For x265 use "-x265-params lossless=1" (without quotes) for ffmpeg or --lossless for the x265cli.

[vhelp]

@ sophisticles, you can also verify the parameters via mediainfo and search for the value for { crf or qp } but not sure if mediainfo is supported in linux though you can copy the video file to a windows pc and then run mediainfo on it to review x264 / x265 encoding parameters. I'm not sure if ffmpeg can put the SEI parameters from x264/x265 though I did search for it too, and same with ffprobe. I could not any info of it doing do. So just use mediainfo to review all your encoding parameters from x264/x265 to be sure of the encoding parameters you used. Its to your benefit.

[jagabo]

Every time I've used --crf=0 x264 has automatically encoded as --qp=0.

That could just be "luck". I bet there are "academic examples" of videos where this is not the case...

In fact, any CRF value below 1.0 is translated to qp=0.

Not certainly. The calculation of a rate factor requires transformations which could be skipped a priori when a specific lossless mode is enabled.

If you look at the settings x264 uses you'll see that --crf=0 is translated to --qp=0 before encoding starts. And it generates exactly the same results.


[Attachment 40441 - Click to enlarge]

As you can see, I specified --crf=0 but x264 used rc=cqp and qp=0. Any crf value below 1.0 will give the same qp=0 encoding.

[poisondeathray]

Likely some user error. Even your x264 results - there isn't such a thing as "close enough". Lossless is either/or . I can't reproduce your results. Both are truly lossless when encoded / tested properly. (This means at the same chroma subsampling, same bit depth etc...)

Post more information, including your settings , test sample large enough to reproduce your findings, x264 and x265 versions etc....

[KarMa]

decided to buy mini1050 for $115 when payday comes around

lol, what? are you 16 with an after school job?

Yes, I hate poor people too. Nice addiction to this thread.

[JVRaines]

decided to buy mini1050 for $115 when payday comes around

lol, what? are you 16 with an after school job?

Yes, I hate poor people too. Nice addiction to this thread.

Psst! "ignore list"

[sophisticles]

Every time I've used --crf=0 x264 has automatically encoded as --qp=0. In fact, any CRF value below 1.0 is translated to qp=0.

I reran the x264 and x265 tests with the settings recommended and then reran the ffmpeg PSNR and SSIM tests, this time the x264 test came out to the same 28.6gb but the calculated PSNR and SSIM were:

[Parsed_ssim_0 @ 0x2bd7a00] SSIM Y:1.000000 (inf) U:1.000000 (inf) V:1.000000 (inf) All:1.000000 (inf)
[Parsed_psnr_1 @ 0x2bd81a0] PSNR y:inf u:inf v:inf average:inf min:inf max:inf

The x265 test proved very interesting, this time the encode was 37.1gb and here are the calculated PSNR and SSIM values:

[Parsed_ssim_0 @ 0x34ac340] SSIM Y:1.000000 (nan) U:1.000000 (inf) V:1.000000 (inf) All:1.000000 (nan)
[Parsed_psnr_1 @ 0x34aca60] PSNR y:inf u:inf v:inf average:inf min:inf max:inf

So it would seem that if one wants a true lossless encode with x264 one has to use the -qp 0 switch; the x265 lossless results are surprising from a file size point of view, I expected that file size to be smaller than the 2 H264 lossless encodes not bigger.

[sophisticles]

decided to buy mini1050 for $115 when payday comes around

lol, what? are you 16 with an after school job?

No dunce; I get paid by the hour and routinely make between 15 and 20 hours of OT every week, when I get paid I set aside all the bill money first, set aside all the OT money into a savings account and what ever is left over is my spending money. So as you can see by my own choice I do not leave myself with a lot of disposable income for thrift spending.

It's called being a responsible adult, you should try it some time.

[sophisticles]

Likely some user error. Even your x264 results - there isn't such a thing as "close enough". Lossless is either/or . I can't reproduce your results. Both are truly lossless when encoded / tested properly. (This means at the same chroma subsampling, same bit depth etc...)

Post more information, including your settings , test sample large enough to reproduce your findings, x264 and x265 versions etc....

No user error, the encodes were done with

time ffmpeg -i input.wmv/mp4/mkv -vcodec libx264/5 -preset ultrafast -crf 0 -an output_x264_ultrafast_lossless.mp4

time ffmpeg -i input.wmv/mp4/mkv -vcodec h264_nvenc -preset lossless -an output_nvenc_h264_lossless.mp4

PSNR and SSIM were calculated following the instructions here:

https://ffmpeg.org/ffmpeg-all.html#ssim

ffmpeg -i main.mpg -i ref.mpg -lavfi "ssim;[0:v][1:v]psnr" -f null -

The test environment was Ubuntu 16.04 LTS, ffmpeg 3.2.2 was built statically with ./configure && make && make install

As for "close enough", this refers to the calculated SSIM values, a true lossless encode will have a value of 1 and a decibel measurement of infinite, the tests I did with x264 crf 0 resulted in an SSIM of 1 and a decibel measurement of 99.656556 and the PSNR calculations were infinite as one would expect, thus that's "close enough" to true lossless for most purposes and the resulting file size was the same for crf 0 and -qp 0 just the calculations for one were slightly off.

Perhaps the reason you can't reproduce my results is that you're using Windows and/or a dynamically compiled ffmpeg version and/or using x264/x65 directly.

[SameSelf]

decided to buy mini1050 for $115 when payday comes around

lol, what? are you 16 with an after school job?

Yes, I hate poor people too. Nice addiction to this thread.

Poor/rich/middle class/refugee, makes no difference to me. I am an equal opportunity hater. Rather, I didn't realize he was from Europe, and it sounded like an adolescent idiom, or someone who blows their paycheck as soon as it is cashed versus saving up like a responsible adult. But, who am I to tell someone what to do with their hard earned euros/chf (did I leave anyone out)? I will be the first to admit that this American boor doesn't understand the European mindset. But I do find it humorous at times

[jagabo]

the encodes were done with

time ffmpeg -i input.wmv/mp4/mkv -vcodec libx264/5 -preset ultrafast -crf 0 -an output_x264_ultrafast_lossless.mp4

time ffmpeg -i input.wmv/mp4/mkv -vcodec h264_nvenc -preset lossless -an output_nvenc_h264_lossless.mp4

PSNR and SSIM were calculated following the instructions here:

https://ffmpeg.org/ffmpeg-all.html#ssim

ffmpeg -i main.mpg -i ref.mpg -lavfi "ssim;[0:v][1:v]psnr" -f null -

I see no controls over chroma subsampling.

[ezcapper]

Every time I've used --crf=0 x264 has automatically encoded as --qp=0. In fact, any CRF value below 1.0 is translated to qp=0.

I reran the x264 and x265 tests with the settings recommended and then reran the ffmpeg PSNR and SSIM tests, this time the x264 test came out to the same 28.6gb but the calculated PSNR and SSIM were:

[Parsed_ssim_0 @ 0x2bd7a00] SSIM Y:1.000000 (inf) U:1.000000 (inf) V:1.000000 (inf) All:1.000000 (inf)
[Parsed_psnr_1 @ 0x2bd81a0] PSNR y:inf u:inf v:inf average:inf min:inf max:inf

The x265 test proved very interesting, this time the encode was 37.1gb and here are the calculated PSNR and SSIM values:

[Parsed_ssim_0 @ 0x34ac340] SSIM Y:1.000000 (nan) U:1.000000 (inf) V:1.000000 (inf) All:1.000000 (nan)
[Parsed_psnr_1 @ 0x34aca60] PSNR y:inf u:inf v:inf average:inf min:inf max:inf

So it would seem that if one wants a true lossless encode with x264 one has to use the -qp 0 switch; the x265 lossless results are surprising from a file size point of view, I expected that file size to be smaller than the 2 H264 lossless encodes not bigger.

You will be pleasantly surprised when you get the gtx 1050. It's lossless hevc encodes are very compressed.

[poisondeathray]

As for "close enough", this refers to the calculated SSIM values, a true lossless encode will have a value of 1 and a decibel measurement of infinite, the tests I did with x264 crf 0 resulted in an SSIM of 1 and a decibel measurement of 99.656556 and the PSNR calculations were infinite as one would expect, thus that's "close enough" to true lossless for most purposes and the resulting file size was the same for crf 0 and -qp 0 just the calculations for one were slightly off.

Perhaps the reason you can't reproduce my results is that you're using Windows and/or a dynamically compiled ffmpeg version and/or using x264/x65 directly.

"slightly off" is wrong. Both crf and qp 0 should produce the same thing when used properly - they did for me and always have. They do for everyone else too.

Yes, I am using windows ffmpeg static version, also x264/x265 directly. They are all lossless, not slightly off or "close enough" . I've never seen "slightly off" because that would mean it's not lossless

It looks like your observation was user error or a spurious observation until you can provide something that says otherwise. Then we can try to debug or find out what is going on . Maybe you have some buggy build. For your first ffmpeg libx265 , it looks like you forgot the lossless switch . The ssim/psnr on your second set of tests look correct

So it would seem that if one wants a true lossless encode with x264 one has to use the -qp 0 switch; the x265 lossless results are surprising from a file size point of view, I expected that file size to be smaller than the 2 H264 lossless encodes not bigger.

But you should be getting smaller filesizes for x265 when both use ultrafast - so some more red flags there . At ultrafast x265 has 3 b-frames , x264 does not. That alone should put x265 over the top. Something else is going on . I would look at your encode logs or at least mediainfo and double check

[poisondeathray]

Another issue you can have with that testing method is if there is jitter in the timestamps , slightly off timebase, those sorts of things, it will be "off". Even MP4 vs. MKV container can have differences. That can result in skewed results. Also, some videos have non zero start times. Also you're calculating a single average PSNR or SSIM with that method, which is of limited value.

So what people do it reset the presentation timestamps using setpts . (Other ways include indexing through avisynth , vapoursynth, so you check for frame accuracy), and do a per-frame SSIM or PSNR written to log file, so you can see problem sections, identify frames that don't match etc.. It gives more information, you can compare in spreadsheets, plot graphs etc...

This is for psnr, and it's the same for ssim (just change psnr to ssim)

ffmpeg -i TEST.ext -c:v rawvideo -vf "movie=REF.ext,setpts=PTS-STARTPTS[main];[main][ref]psnr="stats_file=psnr.log" [out]" -f rawvideo -y /NUL

[hello_hello]

decided to buy mini1050 for $115 when payday comes around

lol, what? are you 16 with an after school job?

Yes, I hate poor people too. Nice addiction to this thread.

Poor/rich/middle class/refugee, makes no difference to me. I am an equal opportunity hater. Rather, I didn't realize he was from Europe, and it sounded like an adolescent idiom, or someone who blows their paycheck as soon as it is cashed versus saving up like a responsible adult.

Yes, when I read it I assumed SameSelf had once again jumped to one of his overly opinionated, rude, and generally incorrect conclusions.

Imagine the SameSelf rant we would have had to endure if sophisticles had said he was using a credit card, which would more resemble not saving like a responsible adult, instead of spending the spare cash he has after payday.
.
.
.
.
.
.
And for an added bonus..... sophisticles is proving himself wrong in the same thread.

[raffriff42]

It's possible PSNR & SSIM are being fooled in some way.
It would be interesting to see the same tests repeated with a known lossless codec.
Also, a visual comparison (taking the difference) might tell us something.