Is Intel's QSV encoder better than X264 and X265?

[sophisticles]

Hello folks.

Those of you that have been around this forum for years know that I was never a big fan of x264; I always thought the so-called psy optimizations were overrated and that the encoder as a whole was over-engineered.

X265, which is based on the x264 code base, is also way over-engineered in my opinion.

Having said that, I always said that quality was x265, qsv hevc, x264, qsv h264, in that order.

What if I told you i was about to show you indisputable visual evidence that the quality order is qsv h264, qsv hevc, x264, x265 so long as we define quality to mean that which is the closest to the source?

The tests were carried out using my Ice Lake laptop, Manjaro with the latest Intel software stack and the following workflow.

I downloaded all the 1080p 4:2:2 files found here:

https://media.xiph.org/video/derf/

I fed them into Shotcut and exported a UT encoded lossless video.

This was the source I used.

I encoded this source using the following encoders:

Selur's Hybrid
x264 - preset medium, tune film, crf 23
x265 - preset medium, crf 28, deblock and psy modified to x264's tune film, SAO disabled

FastFlix+QSVEncC
qsv - icq-la for h264, icq for hevc, best settings available, don't remember the exact icq values but they were chosen to result in a bit rate close to their software counterparts.

After this was done, I used the ffmpeg blend filter to remove the similar parts of each frame from the source video and each test video and leave behind a lossless video with just the differences.

If the two videos match, then there should be nothing to see.

Here are those command line arguments:

ffmpeg -i Source.mkv -i x264.mp4 -filter_complex "blend=all_mode=difference" -c:v utvideo x264-diff.mkv

ffmpeg -i Source.mkv -i x265.mp4 -filter_complex "blend=all_mode=difference" -c:v utvideo x265-diff.mkv

ffmpeg -i Source.mkv -i qsv-h264.mkv -filter_complex "blend=all_mode=difference" -c:v utvideo qsv-h264-diff.mkv

ffmpeg -i Source.mkv -i qsv-hevc.mkv -filter_complex "blend=all_mode=difference" -c:v utvideo qsv-hevc-diff.mkv

I have attached svt-av1 encoded versions of these files.

In order to really highlight the difference I create a mosaic of the 4 videos and exported it as a svt-av1 so I could attach the file here.

ffmpeg -i x264-diff.mkv -i x265-diff.mkv -i qsv-h264-diff.mkv -i qsv-hevc-diff.mkv -filter_complex "nullsrc=size=3840x2160 [base]; [0:v] setpts=PTS-STARTPTS, scale=1920x1080 [upperleft]; [1:v] setpts=PTS-STARTPTS, scale=1920x1080 [upperright]; [2:v] setpts=PTS-STARTPTS, scale=1920x1080 [lowerleft]; [3:v] setpts=PTS-STARTPTS, scale=1920x1080 [lowerright]; [base][upperleft] overlay=shortest=1 [tmp1]; [tmp1][upperright] overlay=shortest=1=1920 [tmp2]; [tmp2][lowerleft] overlay=shortest=1:y=1080 [tmp3]; [tmp3][lowerright] overlay=shortest=1=1920:y=1080" -c:v libsvtav1 -pix_fmt yuv420p10le -r 29.97 -b:v 25M -preset 10 "x264 vs x265 vs qsv-avc vs qsv-hevc.mkv"

ffmpeg -i x264-diff.mkv -i x264-psnr-diff.mkv -filter_complex "nullsrc=size=3840x1080 [base]; [0:v] setpts=PTS-STARTPTS, scale=1920x1080 [left]; [1:v] setpts=PTS-STARTPTS, scale=1920x1080 [right]; [base][left] overlay=shortest=1 [tmp1]; [tmp1][right] overlay=shortest=1=1920:y=0" -c:v libsvtav1 -pix_fmt yuv420p10le -r 29.97 -b:v 15M -preset 10 "crf vs qp.mkv"

At first I was really surprised by the results, but then I started thinking about what I always said about the psy optimizations basically being a scam, a way to play tricks on your mind and the results made sense.

DS used to say that he felt that viewers did not want to see a video that looked the same, they wanted a video with the same complexity, whatever that mishmash means.

It looks like the psy "optimizations" are doing their job, they are producing a scene that looks similar but qsv is producing a video that is closer to the original.

So which one is "better"?

Depends on what you definition of "better" is.

Are you trying to create a video closest to the original? Use qsv.

Are you trying to create a video that looks the best?

Well, I guess it's up to each of us to decide which looks the most appealing.

For the sake of comparison, I have also uploaded an encode done with x264+qp27+medium+tune-psnr and a comparison against x264+crf.

[poisondeathray]

Most people would use tune psnr instead of tune film (or some "psy modified to x264's tune film" whatever that means...) if you wanted it mathematically more similar to source . But perceptually tuning it to PSNR will be less similar to source by most subjective tests - e.g. VMAF

[Sharc]

What's the purpose of "comparing" lossy encodes of totally different (factor of about 2) compression ratios (bitrate, filesize)? If filesize doesn't matter include the lossless variant in the list for "comparison". Maybe I missed the point

[poisondeathray]

The main issue with this comparison is actually different container timebases . MP4 30K tbn vs. MKV 1K tbn . The will change the result for difference overlay and other things like metrics in ffmpeg if you don't normalize the timebase. The timestamps for each frame will not be exactly the same - so to ffmpeg that will result in a larger difference . People often think of "frames" , but that's not exactly how ffmpeg works . It works with PTS for each frame . A valid comparison must use the same PTS or the results will be skewed

You can take any video, say in MKV, remux to MP4 (or vice versa) , then do -filter_complex "blend=all_mode=difference" and it will show large flashing differences, or not infinity PSNR. So bit identical video shows massive differences. Why ? wrong timestamps - for ffmpeg they are different videos, because the time at which each frame is presented is different

[Sharc]

The main issue with this comparison is actually different container timebases . MP4 30K tbn vs. MKV 1K tbn

Ah, interesting. I never thought of that. Thanks for pointing this out.

[JN-]

I never did my quality comparison testing as complicated as this. I just used ssim psnr and vmaf.

Although qsv has improved I found that cpu encoding was best and with hw encoding the order was nvenc, qsv and amd.

[sophisticles]

Most people would use tune psnr instead of tune film (or some "psy modified to x264's tune film" whatever that means...) if you wanted it mathematically more similar to source . But perceptually tuning it to PSNR will be less similar to source by most subjective tests - e.g. VMAF

Maybe I wasn't clear, I created a base encode using x264+medium+tune film as a point of comparison because that is close to what most people would use for this type of content.

"psy modified to x264's tune film" means that I check what tune film does to the base x264 configuration, namely --deblock -1:-1 --psy-rd <unset>:0.15 and I modified x265's base medium settings to match these settings as close as possible.

You also ignored that I did in fact test tune film, I did a x264+crf 23+tune film vs x264+qp 27+ tune psnr and posted a side by side comparison video of the differences.

[sophisticles]

What's the purpose of "comparing" lossy encodes of totally different (factor of about 2) compression ratios (bitrate, filesize)? If filesize doesn't matter include the lossless variant in the list for "comparison". Maybe I missed the point

The lossless version is way to large to post here. Each video's lossless version is at least 6gb and the mosaic is over 25gb.

i also posted the command lines so you can do the comparison yourselves, if you are so inclined, and verify my results.

[poisondeathray]

The main issue with this comparison is actually different container timebases . MP4 30K tbn vs. MKV 1K tbn

Ah, interesting. I never thought of that. Thanks for pointing this out.

Yes, it's a common mistake, there are many tests on various websites using ffmpeg with similar problems, unfortunately

The timestamp difference results in skewed results. It's like comparing time shifted frames.

qsv-hevc.mkv

# timecode format v2

0.00
33.00
67.00
100.00
.
.
.

x265.mp4

# timecode format v2
0
33.3666666666667
66.7333333333333
100.1
.
.
.

If the timestamps are not the same - the same "frames" are not being compared with ffmpeg using overlay , or any ffmpeg metrics like psnr/ssim/vmaf

***That's the main difference in the posted videos, not from encoder differences. ie. It's procedural error



It's easy to demonstrate this

Take x265.mp4 and stream copy it into a mkv container, then do the overlay difference on itself. Big difference. How can that be for an identical stream ? It's the timestamp difference



mp4 container (can and usually does) has higher precision for the timebase than mkv, but it can use different timebases. The mkv specs specify 1ms timebase - always - so going mp4 to mkv usually reduces the precision

The reverse is not necessarily true. If you start with truncated precision, e.g. starting with a mkv, it is possible to express those rounded values with a higher precision timebase in a different container


=> Bottom line is the timestamps have to match if you're using ffmpeg to compare - or you get skewed results

[Sharc]

The main issue with this comparison is actually different container timebases . MP4 30K tbn vs. MKV 1K tbn

Ah, interesting. I never thought of that. Thanks for pointing this out.

Yes, it's a common mistake, there are many tests on various websites using ffmpeg with similar problems, unfortunately

The timestamp difference results in skewed results. It's like comparing time shifted frames.

qsv-hevc.mkv

# timecode format v2

0.00
33.00
67.00
100.00
.
.
.

x265.mp4

# timecode format v2
0
33.3666666666667
66.7333333333333
100.1
.
.
.

If the timestamps are not the same - the same "frames" are not being compared with ffmpeg using overlay , or any ffmpeg metrics like psnr/ssim/vmaf

***That's the main difference in the posted videos, not from encoder differences. ie. It's procedural error



It's easy to demonstrate this

Take x265.mp4 and stream copy it into a mkv container, then do the overlay difference on itself. Big difference. How can that be for an identical stream ? It's the timestamp difference



mp4 container (can and usually does) has higher precision for the timebase than mkv, but it can use different timebases. The mkv specs specify 1ms timebase - always - so going mp4 to mkv usually reduces the precision

The reverse is not necessarily true. If you start with truncated precision, e.g. starting with a mkv, it is possible to express those rounded values with a higher precision timebase in a different container


=> Bottom line is the timestamps have to match if you're using ffmpeg to compare - or you get skewed results

Would it help in such case to align (define) the timebase using '-video_track_timescale xx' (or something similar) in the ffmpeg commandline?

[JN-]

The main issue with this comparison is actually different container timebases . MP4 30K tbn vs. MKV 1K tbn . The will change the result for difference overlay and other things like metrics in ffmpeg if you don't normalize the timebase. The timestamps for each frame will not be exactly the same - so to ffmpeg that will result in a larger difference . People often think of "frames" , but that's not exactly how ffmpeg works . It works with PTS for each frame . A valid comparison must use the same PTS or the results will be skewed

You can take any video, say in MKV, remux to MP4 (or vice versa) , then do -filter_complex "blend=all_mode=difference" and it will show large flashing differences, or not infinity PSNR. So bit identical video shows massive differences. Why ? wrong timestamps - for ffmpeg they are different videos, because the time at which each frame is presented is different

I never knew that either PDR.

In my ffmpeg testing most output was to h264/hevc mp4. Some to h264/hevc in a MOV container. Do your above remarks apply as an issue ? Or is it only an issue when comparing mp4 with mkv. I didn't use MKv as container. My source was a h264 mp4 generated initially from within Vegas Pro which uses the MainConcept encoder. Should I have used ffmpeg throughout ? Is there a way to test if the source and output have different pts type ? Thanks.


Is there a way to identify if there is a pts issue between 2 different files ?
Is there a way to "normalize" the pts between different h264/hevc mp4 files ?

[poisondeathray]

Would it help in such case to align (define) the timebase using '-video_track_timescale xx' (or something similar) in the ffmpeg commandline?

Unfortunately it doesn't always work...

In my ffmpeg testing most output was to h264/hevc mp4. Does your above remarks apply as an issue ? Or is it only an issue when comparing mp4 with mkv. My source was a h264 mp4 generated initially from within Vegas Pro which uses the MainConcept encoder. Should I have used ffmpeg throughout ? Is there a way to test if the source and output have different pts type ? Thanks.

Is there a way to identify if there is a pts issue between 2 different files ?
Is there a way to "normalize" the pts between different h264/hevc mp4 files ?

Mp4 can use different timebases depending on the application. Program "A" might assign a certain timebase based on the FPS, but program "B" might assign a different value - this will lead to "non aligned frames" (or more accurately the frames are presented at slightly different times)

Because containers can use different timebases, it's important to verify the timestamps. The exception is MKV which always specifics 1k tbn (1ms timebase) in the MKV specs. So if you assume the program writes spec compliant MKVs....

A quick check is ffmpeg -i input.ext and look at tbn .

However, a procedural error could have rounded timestamps off earlier - so to be 100% certain, you still need to check the timestamps

Another "gotcha" is some programs round off the precision. e.g. "29.97" might mean 29.97/1 in one program , but 30000/1001 in another program. ie. Upstream issues could cause PTS discrepancies.

If you extract timestamps, the "non alignment" should be visible within a few entries for CFR sources. VFR sources can add additional complexity




For ffmpeg metrics - PSNR/SSIM/VMAF use settb and setpts

e.g for PSNR

ffmpeg -i input.ext -i source.ext -lavfi  "[0:v]settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]psnr" -f null -

Note in the documentation , there is "an example with different containers" . It's currently INCORRECT . It should be 1/AVTB
https://ffmpeg.org/ffmpeg-filters.html#psnr




Another way is to decompress to rawvideo. Because there is no container, there is no timebase to confuse the calculations

Other ways are to use avisynth and AssumeFPS() as the inputs. Avs is CFR only and always has the same timebase in ffmpeg for each FPS. You can also visually verify frames are aligned in avspmod tabs

[Sharc]

Thank you pdr for these explanations. Something to digest .....

[JN-]

Thanks for that PDR. Great info.

I think I am ok with my syntax re: SSIM and PSNR ... ?
ffmpeg -i %1 -i %_REFERENCE_FILE% -lavfi "[0:v]setpts=PTS-STARTPTS[main];[1:v]setpts=PTS-STARTPTS[ref];[main][ref]ssim="eof_action=endall";[0:v]setpts=PTS-STARTPTS[main];[1:v]setpts=PTS-STARTPTS[ref];[main][ref]psnr=eof_action=endall" 2>"ffmpeg SSIM and PSNR.txt" -f null -

I am curious as to how to insert a code piece into a white box as you and others do ?

 ffmpeg -i %1 -i %_REFERENCE_FILE% -lavfi "[0:v]settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]ssim="eof_action=endall";[0:v]settb=1/AVTB,setpts=PTS-STARTPTS[main];[1:v]settb=1/AVTB,setpts=PTS-STARTPTS[ref];[main][ref]psnr=eof_action=endall"  2>"ffmpeg SSIM and PSNR.txt" -f null -

Got it, thanks. That's with the "settb=1/AVTB" item added in.

[JN-]

OK, should I add in the ... settb=1/AVTB,

What does it do ?

[Selur]

it sets the time base to 1/(the default time base)
https://ffmpeg.org/ffmpeg-filters.html#settb_002c-asettb

I am curious as to how to insert a code piece into a white box as you and others do ?

use 'code'-tags see https://en.wikipedia.org/wiki/BBCode

[poisondeathray]

I think I am ok with my syntax re: SSIM and PSNR ... ?

Possibly, if container used the same timebase and the timestamps were equivalent

I am curious as to how to insert a code piece into a white box as you and others do ?

Use the code boxes

code and /code  (each enclosed by square brackets) . The brackets doesn't show up on this board

OK, should I add in the ... settb=1/AVTB,

What does it do ?

It normalizes the timebase

I would include it, unless you're certain that your files have the same timestamps

[JN-]

Truly excellent, thanks.

[sophisticles]

You can take any video, say in MKV, remux to MP4 (or vice versa) , then do -filter_complex "blend=all_mode=difference" and it will show large flashing differences, or not infinity PSNR. So bit identical video shows massive differences. Why ? wrong timestamps - for ffmpeg they are different videos, because the time at which each frame is presented is different

Good call.

I went back and redid the tests, and the results were much closer.

I will be posting updated samples shortly.

[JN-]

Post 14 ...
I added in the item settb=1/AVTB, got same results as previous but delighted to have this improvement as it may improve things dependent on files used.

Thanks PDR and SELUR for heads up on adding white code box. So type ... [cod e] Enter your wonderful code snippet here [/cod e]

Code word "cod e" intentially misspelt so that it isn't presented as code box.

[sophisticles]

I never did my quality comparison testing as complicated as this. I just used ssim psnr and vmaf.

Although qsv has improved I found that cpu encoding was best and with hw encoding the order was nvenc, qsv and amd.

I tend to agree that NVENC is the best current solution.

Regarding PSNR, SSIM, VMAF and similar metrics, they have gotten a bad rap over the years because people misuse them.

PSNR is normally calculated across YUV channels for I, P and B frames but when people do quality comparisons they only look at the maximum value which is typically the Y channel of an I frame, but if the UV channels and/or the P or B frames of the other video have lower scores, then people will conclude that PSNR can't be trusted because they are not looking at the complete picture.

What I do is calculate PSNR across all channels and all frames, then I do a frame by frame and channel by channel comparison of the videos and arrive at a statistical view of quality.

For instance, I use Excel or Calc to compare the first frame of each comparison video for what the PSNR if the Y channel is, do the same for the U and V channels, then go to the second frame and repeat.

For each frame I pick the highest value of each channel and when it's all done I can say that video 1 had the highest PSNR on the Y channel for this many frames, so and and so forth and you see which video scored the highest most often.

There are times one video will dominate and will be the highest across the board, there are times when they trade punches.

Proper comparison is time consuming,

[JN-]

"Proper comparison is time consuming,"

I have no doubt that you drill down real deep into all of this. I didn't mean to appear dismissive. Kudos to your dedication. If you hadn't brought this up I wouldn't have got the PSNR/SSIM improvement from PDR and learnt how to post a code snippet.

My use case curiosity and requirement was initially started by wondering what the quality difference was in the different Vegas Pro codecs.

So having a perfect result for each codec wasn't as important as getting a pecking order result. However I think they are ok.

This is typical output to my log file. Oh, meant to say that while I was still not banned from VP forum that fifonik assisted with my above syntax at around the time he was starting to develop his ffmetrics app.

Start Date and Time .. 24/03/2024 .. 17:41:51.05 ------------------------------------------------------------------------------------

----------------------------------------------------- 09 MC CPU, [ High Data Rate ].mp4 metrics below

SSIM All........ 0.988288 (19.313573)
PSNR Average ... 43.409878
VMAF ........... 99.052764


----------------------------------------------------- 27 MC Legacy ~ 20 Mbps.mp4 metrics below

SSIM All........ 0.966509 (14.750672)
PSNR Average ... 37.847128
VMAF ........... 96.335270

Note that there was a duration discrepancy between the Source and this Target file.
Check out the errors log file for details.

End Date and Time .... 24/03/2024 .. 17:42:17.97 ... Duration [ 00:00:27.52 ]. Number of input files [ 2 ].
--------------------------------------------------------------------------------------------------------------------------------------
Above file(s) processed by ....... [ RQM.bat ] ..... Reference file ... [ Source.mp4 ]
__________________________________________________ __________________________________________________ __________________________________
ffmpeg version date .. 27-11-2023
ffmpeg version ....... ffmpeg version 2023-11-27-git-0ea9e26636-full_build-www.gyan.dev Copyright (c) 2000-2023 the FFmpeg developers
__________________________________________________ __________________________________________________ __________________________________


Errors log file ...
-------------------------------------------------------------------------------------------------------

The Source.mp4 and Target input files "Duration" times EXIF data doesn't match ?

27.000000 Duration ... Source file.
27.040000 Duration ... Target, input file to process

Target file to process is ... [ "27 MC Legacy ~ 20 Mbps.mp4" ]
Date and Time .... 24/03/2024 .. 17:42:05.21
-------------------------------------------------------------------------------------------------------

[Sharc]

I am curious as to how to insert a code piece into a white box as you and others do ?

As easy as marking the text and then pressing the # button

[Attachment 77905 - Click to enlarge]

[JN-]

Wow. Thank you Sharc. Just came back from a walk and while out I was thinking why can't it be as easy as say marking text and setting to italic. I had looked at those items earlier but without any pop up balloons?

Brilliant

[sophisticles]

Continuing where I left off.

I created 2 different test samples, one is a combination of a bunch of 1080p samples from Xiph Derf and the other is a combination of 720p samples into one 1440p test file.

I encoded all of them with x264, x265, and qsv using Fast Flix on Manjaro.

The 1080p sample was also put through the ffmpeg difference filter, for some reason the 1440 test file would fail to prodice meaningful results when put through the filter.

The x264 encodes were done using the same crf, the x265 test encodes were done using the same crf and the qsv test encodes were done using the same cqp.

The diff files were done using 10mb/s 3-pass svt-av1 preset 10.

Note, the 1440p video is not my idea, when Intel first released qsv with Sandy Bridge, they released a pdf on how to get qsv working with ffmpeg and how to test it.

The recommend to create a video like the one I used claiming it was an industry standard way of testing encoders.

It is a very tough test for any encoder.

If you enjoy this type content please hit like and subscribe, please remember to check out our sponsors, this specific test was brought to you by beer, and if you don't like it please tell an enemy.

Please remember to tip your waitress.