QuickSync H.264 Settings

[hanaluluelle]

Can I use QuickSync to get an output with quality indistinguishable from the source? I’m gonna use it for my recording settings in OBS. I was trying to ask earlier in their community chat but my connection was bad. I was hoping I could get insight here.

[Bernix]

Hi,
i can't tell you because I am using GPU nvidia, but check in settings-here in output tab recording. If you can set here in encoder QuickSync so choose it. And in rate control on same page choose Lossless, if you have it there. If not just reply what possibilities do you have in rate control.

Bernix

[hanaluluelle]

quicksync has cbr, vbr, vcm, la, la icq, cqp, icq, and avbr.

[Bernix]

O.K. then try cqp with low value as possible. Test it. If is it similar to h264 so about 16 should be good result, but huge filesize. If not satisfied, go lower, if satisfied try higher. It should be 0 for lossless, but it is nonsense settings. If you test it you will see what I mean. Super huge file with best quality but visually similar for example settings 10.

Edit: I noticed they suggest QuickSync preset balanced.

Bernix

[hanaluluelle]

i actually used cqp 13 before. the size was huge and i dont think the resulting bitrate needed to be that high up to be of near lossless quality.

right now i'm experimenting with la, which is a vbr option. bitrate is set to 50000. trying to figure out what is lookahead depth and async.

[Bernix]

Lookahead depth should be how far encoder looks to optimally assign type of frame. Higher better, but usually should be about Keyframe interval if high it cost lot of sources.

[Bernix]

Maybe you can achieve better quality using some fast preset using cpu x264. Just try it. x264 is better than Nvenc is better than QuickSync.
Bernix

[pandy]

I would disagree - NVEnc quality is worse than QSV... Options related to quality should be documented by OBS eventually by Intel official documentation - something like https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/cloud-comput...hite-paper.pdf and similar documents. Not sure how OBS use Intel QSV API.

https://obsproject.com/forum/threads/custom-parameters-of-quicksync.18973/

[Bernix]

Hi Pandy,
Nvenc has at least lossless option here for first. I used to use QuickSync not too much but before I got 1050ti. And I achieve very very good quality for full HD at 3500kbps with Nvenc 264. And OP is not satisfied with 25.000 kbps at say same framerate. So something is wrong. And I think higher framerate not 2x bitrate, but less because of smallest difference between frames.
Edit: O.K. I read about benchmarking and results and it seems quicksync is better, but according this is better then x264, which is I think nonsense.
And doesn't it depend on HW and application that are running when capturing?
Bernix

[pandy]

Well... perceived quality is highly subjective - not sure what OP perceive but perhaps some other problems occurs at the same time (like aliasing quite common for CG aka game capture).

As i'm not using OBS at all then i can only provide info from ffmpeg where QSV works OK-ish (there is some internal timeout so too quickly restarting encoding may trig some problems - this is my interpretation for observed behaviour and it may be incorrect).

From ffmpeg perspective QSV H.264 looks like this:

Encoder h264_qsv [H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (Intel Quick Sync Video acceleration)]:
    General capabilities: delay 
    Threading capabilities: none
    Supported pixel formats: nv12 p010le qsv
h264_qsv encoder AVOptions:
  -async_depth       <int>        E..V.... Maximum processing parallelism (from 0 to INT_MAX) (default 4)
  -avbr_accuracy     <int>        E..V.... Accuracy of the AVBR ratecontrol (from 0 to INT_MAX) (default 0)
  -avbr_convergence  <int>        E..V.... Convergence of the AVBR ratecontrol (from 0 to INT_MAX) (default 0)
  -preset            <int>        E..V.... (from 1 to 7) (default medium)
     veryfast                     E..V....
     faster                       E..V....
     fast                         E..V....
     medium                       E..V....
     slow                         E..V....
     slower                       E..V....
     veryslow                     E..V....
  -vcm               <int>        E..V.... Use the video conferencing mode ratecontrol (from 0 to 1) (default 0)
  -rdo               <int>        E..V.... Enable rate distortion optimization (from -1 to 1) (default -1)
  -max_frame_size    <int>        E..V.... Maximum encoded frame size in bytes (from -1 to 65535) (default -1)
  -max_slice_size    <int>        E..V.... Maximum encoded slice size in bytes (from -1 to 65535) (default -1)
  -bitrate_limit     <int>        E..V.... Toggle bitrate limitations (from -1 to 1) (default -1)
  -mbbrc             <int>        E..V.... MB level bitrate control (from -1 to 1) (default -1)
  -extbrc            <int>        E..V.... Extended bitrate control (from -1 to 1) (default -1)
  -adaptive_i        <int>        E..V.... Adaptive I-frame placement (from -1 to 1) (default -1)
  -adaptive_b        <int>        E..V.... Adaptive B-frame placement (from -1 to 1) (default -1)
  -b_strategy        <int>        E..V.... Strategy to choose between I/P/B-frames (from -1 to 1) (default -1)
  -cavlc             <int>        E..V.... Enable CAVLC (from 0 to 1) (default 0)
  -idr_interval      <int>        E..V.... Distance (in I-frames) between IDR frames (from 0 to INT_MAX) (default 0)
  -pic_timing_sei    <int>        E..V.... Insert picture timing SEI with pic_struct_syntax element (from 0 to 1) (default 1)
  -single_sei_nal_unit <int>        E..V.... Put all the SEI messages into one NALU (from -1 to 1) (default -1)
  -max_dec_frame_buffering <int>        E..V.... Maximum number of frames buffered in the DPB (from 0 to 65535) (default 0)
  -look_ahead        <int>        E..V.... Use VBR algorithm with look ahead (from 0 to 1) (default 1)
  -look_ahead_depth  <int>        E..V.... Depth of look ahead in number frames (from 0 to 100) (default 0)
  -look_ahead_downsampling <int>        E..V.... (from 0 to 2) (default unknown)
     unknown                      E..V....
     off                          E..V....
     2x                           E..V....
  -int_ref_type      <int>        E..V.... Intra refresh type (from -1 to 65535) (default -1)
     none                         E..V....
     vertical                     E..V....
  -int_ref_cycle_size <int>        E..V.... Number of frames in the intra refresh cycle (from -1 to 65535) (default -1)
  -int_ref_qp_delta  <int>        E..V.... QP difference for the refresh MBs (from -32768 to 32767) (default -32768)
  -recovery_point_sei <int>        E..V.... Insert recovery point SEI messages (from -1 to 1) (default -1)
  -trellis           <flags>      E..V.... Trellis quantization (default 0)
     off                          E..V....
     I                            E..V....
     P                            E..V....
     B                            E..V....
  -profile           <int>        E..V.... (from 0 to INT_MAX) (default unknown)
     unknown                      E..V....
     baseline                     E..V....
     main                         E..V....
     high                         E..V....
  -a53cc             <int>        E..V.... Use A53 Closed Captions (if available) (from 0 to 1) (default 1)

[Bernix]

From ffmpeg perspective QSV H.264 looks like this:

Hi,
have you same for NVENC? If it exist. But most probably not.
But you provide that there is lot of option for QuickSync, that Nvenc doesn't provide. Not sure about quicksync h265, it could be advantage of Nvenc. AFAIK, it provides just six presets and bitrate and max bitrate. Maybe there will be something more on nvidia site. I will check it.
Not latest release but some option available to 7.1 PDF
Some information about 8.0
Plenty of useful links here one of them And performance vs quality nvenc vs x264

OP question max lookahead provided by Pandy is 100 if you are planing 60fps. Default is 0 and for recording (capturing) I would keep it at 0.

[pandy]

As i have no NVidia in home then i have no NVEnc options to query however you can ask for encoder capabilities ffmpeg itself - bellow examples:

ffmpeg -h encoder=dnxhd >dnxhd.txt
ffmpeg -h encoder=prores_ks >prores_ks.txt
ffmpeg -h encoder=libxvid >libxvid.txt
ffmpeg -h encoder=mpeg2video >mpeg2video.txt
ffmpeg -h encoder=libx264 >libx264.txt
ffmpeg -h encoder=libx265 >libx265.txt
ffmpeg -h encoder=libvpx-vp9 >libvpx-vp9.txt
ffmpeg -h encoder=libmp3lame >libmp3lame.txt

ffmpeg -h encoder=h264_qsv >h264_qsv.txt

ffmpeg -h filter=vmafmotion > vmafmotion.txt

[Bernix]

Here is the Text file from ffmpeg info about h264_nvenc capabilities from hybrid released at september.

See here
Edit: Here is Hevc_nvenc
Bernix

[pandy]

Here is the Text file from ffmpeg info about h264_nvenc capabilities from hybrid released at september.

See here

Bernix

Thx, definitely NVEnc looks like more versatile but QSV is praised for it's quality sometimes comparable to x264. I like NVEnc for its repeatability and stability - maybe not great HW but reliable.

[hanaluluelle]

Thanks for the replies. I find it hard to get good results at 50mbps bitrate using hw. The colors always look washed whether it’s nvenc or qs. I would try capturing in yuv444 but i dont think youtube displays them correctly. I am uploading them to youtube btw

[Bernix]

Hi,
what bitrate do you get when choose Nvenc lossless(BTW it using cabac at least it should)? I think you are capturing in at least Full HD at 60fps?
Thank you
Edit: Once you have captured in yuv 444 you can then transcode it with x264 and I believe 4:2:0 will look good.
Bernix

[hanaluluelle]

900p 60fps. If i do lossless, i get 200mbps-300mbps when in yuv420. The size is too big for me.

[Bernix]

Hi,
MegaByte or Megabits? It is of course huge file, 60fps with nearly fullHD. It is less than 2MP but x60 it is around 100MP/s In this POV it doesn't look such big.
Thank you!

Maybe better keep standard 720p, and you spare some bits. You can get about 160 Msomething And because you are european try Pal 50 fps. Another 12% off.

Bernix

[Bernix]

An idea. In recording, you can choose in nvenc h264 2pass settings. I think it shouldn't work for capturing, but if it works (nvenc is fast enough), it can improve your quality of video significantly but I suggest not lossless of course.
Just try it. Also remember, you are most probably downscaling video. Try bicubic filter instead of Lanczos. It is I think less demanding (not sure). And you can save some cpu gpu time.
And try 1280x720x50fps. You can get in 1pass lossless mode bitrate about 140mb.
Bernix

[hanaluluelle]

bits

with my current source, i am not downscaling the resolution, but if ever, my gpu can handle lanczos just fine.

i feel like i found my sweet spot. i just realized i probably can't get near lossless quality with 50mbps in yuv420. i tried to increase it to 75mbps, set format to yuv444, nvenc 2-pass cbr.

the quality is great! i'll probably still use this setting even when i'm capturing full hd.

[pandy]

Thanks for the replies. I find it hard to get good results at 50mbps bitrate using hw. The colors always look washed whether it’s nvenc or qs. I would try capturing in yuv444 but i dont think youtube displays them correctly. I am uploading them to youtube btw

Ok, first washed colors are usually sign of that your capture is with limited quantization range used in TV (as opposite to PC world) - you need to properly signal that your source use limited quantization range. and proper signal level will be restored on decoder sit or you need to convert limited quantization range to full (and also signal this properly).
Another topic is YT - add some motion blur filter, add antialiasing, peroform some small spatial blur, all this should improve quality, don't forget that YT will recode your video to relatively low 2.5 - 3Mbps bitrate and you will be unable to do anything with this (unless you will go for 4k 60p where slightly higher bitrate will be allocated).
You can't do with YT anything.

bits

with my current source, i am not downscaling the resolution, but if ever, my gpu can handle lanczos just fine.

i feel like i found my sweet spot. i just realized i probably can't get near lossless quality with 50mbps in yuv420. i tried to increase it to 75mbps, set format to yuv444, nvenc 2-pass cbr.

the quality is great! i'll probably still use this setting even when i'm capturing full hd.

CBR is waste of space, 444 have no chance to survive YT reencoding (420 color space is used by YT), Lanczos is bad for rescaling as it will introduce ringing around edges (Lanczos is practically same approach to resize as window limited SinC kernel) - Spline based resizer is preffered.

[Bernix]

According to Pandy advice, just try vbr, it will not cost you much to try it. It also should have benefit more from 2pass encoding.
I think when you lately reencode 444 to 420 with x264 it should be good. Just if you are satisfied with capture settings why not.
Try change your dynamic range in nvidia settings. Switch to setting Nvidia in video - color setting - with Nvidia setting instead of player and in last tab here (extension)<- or something like this change it. There are two option full 0-255 and limited 16- 235. Then try OBS again.
Bernix

[hanaluluelle]

i just wanted to get the best quality i can get before i upload it to youtube. hence i was trying to get an output that's indistinguishable from my source.
i can't achieve this with a lower setting than what i just said. yuv444 looks so much better which is why uploading it to youtube is really problematic.

someone from a forum told me that the best filter is a subjective opinion. fortunately atm, i am not rescaling. but if i must in the future, what do you think is best between bilinear, bicubic, and lanczos?

[Bernix]

Bilinear fastest Lanczos declined by Pandy, try bicubic then
Edit: there is no more option I think
You mentioned 900p, which is not mod8 friendly. Is there any disadvantage of not using mod8 or even mod16?
Bernix

[jagabo]

yuv444 looks so much better which is why uploading it to youtube is really problematic.

Youtube is going to downsample to 4:2:0 so it doesn't really matter if you upload 4:4:4 or 4:2:0. Small colored objects/text will lose saturation and sharpness either way.

what do you think is best between bilinear, bicubic, and lanczos?

It depends on how sharp the video is and the look you want. Bilinear is the most natural looking and will decrease the buzzing of aliased edges. Lanczos sharpens but will create oversharpening halos if your source is very sharp. And it will increase the buzzing of aliased edges. Bicubic is in between.

As Pandy pointed out, your washed out colors is probably due to a TV vs computer levels issue, not an encoding issue.

[pandy]

i just wanted to get the best quality i can get before i upload it to youtube. hence i was trying to get an output that's indistinguishable from my source.
i can't achieve this with a lower setting than what i just said. yuv444 looks so much better which is why uploading it to youtube is really problematic.

someone from a forum told me that the best filter is a subjective opinion. fortunately atm, i am not rescaling. but if i must in the future, what do you think is best between bilinear, bicubic, and lanczos?

Once again - read about YT and how video are re-compressed by YT engine - pushing highest quality when uploading has no sense - if you search there is at least few threads on this (game to YT) - people complain but you can only accept YT policy or upload video somewhere else - sorry. There are some tricks to force YT higher quality, one of common tricks is upsizing video to higher resolution as YT will allocate for such video higher bitrate and overall quality loss will be lower.
444 will be immediately converted to 420 so it is better to upscale your 1920x1080 (my assumption) twice to 4k.
I would advise you to upscale with some xbrz family filters or spline - one of easiest tricks id upscale xbrz twice and downsize spline by half - this works quite well as crude antialiasing filter.
Lanczos may increase amount of ringing around edges when encoder is forced to use high qp factor.

Concept of best is purely subjective and on some very important video forum it is forbidden to use "best" to describe quality at all - however behind filter there is math and math lead to some objective conclusions - splines are very well balanced between objective and subjective aspects of video resizing.

If you have quick upstream capabilities don't waste your time for searching close to perfect settings - YT will destroy your video and you will be very frustrated after all.

[Bernix]

So why YT doesnt do conversion Chroma subsampling during compression, it seems they do it prior then.
https://en.wikipedia.org/wiki/Chroma_subsampling#Effectiveness
I think you can recompress it by yourself to x264 and then on YT. I know, twice compression, but if YT doesnt handle it properly...
Possibly some magic trick can exist in ffmpeg... don't know.

[pandy]

So why YT doesnt do conversion Chroma subsampling during compression, it seems they do it prior then.
https://en.wikipedia.org/wiki/Chroma_subsampling#Effectiveness
I think you can recompress it by yourself to x264 and then on YT. I know, twice compression, but if YT doesnt handle it properly...
Possibly some magic trick can exist in ffmpeg... don't know.

IMHO YT always perform conversion to 420 - never saw any video on YT with different than 420 sampling...
What im trying to express with my poor English is fact that there is no sense to pursue perfect quality on local computer as YT will perform recompression anyway.

[Bernix]

I forget to add do conversion 444 to 420 yourself properly in first compression. If there is some proper way how to do it. Sorry for misunderstanding. Your english is perfect, my causes many misunderstanding. Sorry again.
Edit: for example 8 bit 444 to 10bit 420 Can't this help a bit?

[Bernix]

Here is difference of two jpeg 100% quality slowest but subsampling 2x2 and 1x1 what I believe is equivalent to 4:2:0 and 4:4:4. The original image has 3 colors and is bellow the difference picture.

[Attachment 43814 - Click to enlarge]

[Attachment 43815 - Click to enlarge]
If you use any resize algorithm then difference will be much bigger. Even with neural network resize difference will be at least similar.
I hope, I did it properly. Any suggestion or what I did wrong is highly appreciated.

Bernix