Is there a reason my NVENC encodes look better than my x264 ones?

[CursedLemon]

I was doing some test encodes with MediaCoder just for the sake of doing the good old "eye test" in comparing various AVC and HEVC presets utilizing x264, x265, and NVENC. You can find the test encodes here.

I've got an i7-6700K and a GTX 960 in my rig presently, if that matters.

My source clip was made using a screen recording of Turok 2 utilizing Bandicam's highest HEVC recording setting which utilizes NVENC. Bandicam's setting specifies 100Mbps, but the file ended up being around only 20Mbps according to MediaInfo. The keyframe interval was set to 1. I used Turok 2 because it seemed to me that the differences in various settings were most easily identifiable with simple, low-resolution textures.

Inside MediaCoder, all the settings are left to default (so 2 B-Frames, 250 GOP on everything for example) except for the x264/x265 quality presets, which were Ultrafast, Medium, and Very Slow respectively. All of my re-encodes were done at 1kbps, since to me it was easiest to see the differences in encoding quality as well as clue in on how the encoders function (motion vs. still images, etc.) at a very low bitrate. The encoder "source" is left to auto.

And...according to this test, while NVENC suffers heavily against x265, it actually does better than x264's "very slow" preset in this case. The way it preserves the look of the portal in the video is especially noteworthy.

That kind of sounds like heresy.

Is MediaCoder doing something funky and causing this to happen? Because everywhere I read, NVENC shouldn't be competing with x264 at anything over "very fast".

[Atak_Snajpera]

All of my re-encodes were done at 1kbps, since to me it was easiest to see the differences in encoding quality as well as clue in on how the encoders function (motion vs. still images, etc.) at a very low bitrate. The encoder "source" is left to auto.

1kbps you say? 1000 bits is like 125 bytes/s.

[poisondeathray]

It's mainly because mediacoder is using "CBR" rate control for x264

Normally you would use CRF rate control with x264, but the bitrate are unpredictable. But CRF is usually used in actual practice

So for testing, you'd encode x264 using a range of different CRF values, and NVEnc to match the bitrates . e.g you 'd test 1000kbps vs. 1000kbps, 2000kbps vs. 2000kbps, etc...

Here is a random B to B frame comparison , and the "proper" x264 encode at medium , and the NVenc encode you provided at about the same bitrates

[CursedLemon]

1kbps you say? 1000 bits is like 125 bytes/s.

Now I feel really stupid that I went through how many iterations of typing that without realizing how wrong it was. lol But, also I feel very at home in the newbie section, now!

It's mainly because mediacoder is using "CBR" rate control for x264

Normally you would use CRF rate control with x264, but the bitrate are unpredictable. But CRF is usually used in actual practice

So for testing, you'd encode x264 using a range of different CRF values, and NVEnc to match the bitrates . e.g you 'd test 1000kbps vs. 1000kbps, 2000kbps vs. 2000kbps, etc...

Here is a random B to B frame comparison , and the "proper" x264 encode at medium , and the NVenc encode you provided at about the same bitrates

What exactly is the difference between CRF and CBR in that regard? The bitrates (or the average bitrates, I suppose) for all of the re-encoded videos are virtually identical, basic intuition would suggest that the quality presets/encode time would be the determining factor from that point.

[jagabo]

CRF is variable bitrate. It gives each shot the bitrate it needs to maintain the specified quality. CBR is constant bitrate. Every shot gets the same bitrate, regardless of the resulting quality.

[CursedLemon]

CRF is variable bitrate. It gives each shot the bitrate it needs to maintain the specified quality. CBR is constant bitrate. Every shot gets the same bitrate, regardless of the resulting quality.

Does that imply that x264 is not designed to use CBR?

Also, MediaCoder is hinting at me that NVENC doesn't support variable bitrate, is this correct? If so, is that basically NVENC's entire drawback in a nutshell?

[poisondeathray]

Does that imply that x264 is not designed to use CBR?

Yes

In fact x264's "CBR" isn't true CBR . (it's why I put the parentheses around "CBR") But the point is, when using that mode, there are limitations placed on the encoding that make it always worse, efficiency wise .

Every encoder will yield worse quality when using CBR compared to VBR, even NVEnc

There are appropriate situations where you might need to use CBR, but that's for another discussion

Also, MediaCoder is hinting at me that NVENC doesn't support variable bitrate, is this correct? If so, is that basically NVENC's entire drawback in a nutshell?

Nope, NVEnc definitely supports VBR

[poisondeathray]

If mediacoder can't or doesn't allow you to use VBR for NVEnc, then that' s not a valid test for NVEnc and it should be repeated using better software if you can't figure it out how to do it with mediacoder. There is no indication of what mode is used in mediainfo for your NVEnc encode, so I assumed it was the default mode, which is VBR. But the metadata tells what settings were used for x264, x265 when using mediainfo

[CursedLemon]

All of my encodes in this test were done at 1000kbps CBR in MediaCoder, since that seemed the most "fair" to me in my layman intuition, which obviously appears to be quite wrong.

In MediaCoder, when I try to select "Variable Bitrate" for NVENC in the rate mode drop-down option, starting the encoding process gives me an error, "Rate mode not supported, incorrect parameters for NVENC". The available rate mode options are:

• Average Bitrate
• Variable Bitrate
• Constant Bitrate
• Constant QP
• 2-Pass
• 3-Pass

[poisondeathray]

I don't like or use mediacoder for several reasons, so someone else will have to help you with that

But for NVEnc, the actual encoder, typically 2pass High Quality VBR is used (it's an internal 2pass , but still much faster than x264) . That's generally regarded as the "best" method for NVEnc. There are different settings and presets (you can encode ~2x faster if you wanted using the High Performance preset with 1pass) . I don't know if mediacoder gives you access to this or maybe you have to pay more to unlock some features that are free in other software - mediacoder has a bad reputation for that sort of thing

[CursedLemon]

Damn, and MediaCoder has been so reliable up until now, hah.

Welp, looks like I'm going to have to learn to use FFmpeg the right way if I want to pursue it. What's the status of FFmpeg in Linux these days, are there libraries for NVENC? I need an excuse to boot up Xubuntu.

[CursedLemon]

Also, referencing this:

If mediacoder can't or doesn't allow you to use VBR for NVEnc, then that' s not a valid test for NVEnc and it should be repeated using better software if you can't figure it out how to do it with mediacoder. There is no indication of what mode is used in mediainfo for your NVEnc encode, so I assumed it was the default mode, which is VBR. But the metadata tells what settings were used for x264, x265 when using mediainfo

What would be the strategy for getting an NVENC encode to match the bitrate of an x264 export? For example, I ran a CRF30 Medium test x264 encode and the bitrate according to MediaInfo is 870kbps. If I wanted to do an equivalent NVENC encode to compare them side by side, what parameters would I be adjusting? The parameter list is as follows:

Encoder h264_nvenc [NVIDIA NVENC H.264 encoder]:
    General capabilities: delay 
    Threading capabilities: none
    Supported pixel formats: yuv420p nv12 p010le yuv444p yuv444p16le bgr0 rgb0 cuda
h264_nvenc AVOptions:
  -preset            <int>        E..V.... Set the encoding preset (from 0 to 11) (default medium)
     default                      E..V.... 
     slow                         E..V.... hq 2 passes
     medium                       E..V.... hq 1 pass
     fast                         E..V.... hp 1 pass
     hp                           E..V.... 
     hq                           E..V.... 
     bd                           E..V.... 
     ll                           E..V.... low latency
     llhq                         E..V.... low latency hq
     llhp                         E..V.... low latency hp
     lossless                     E..V.... 
     losslesshp                   E..V.... 
  -profile           <int>        E..V.... Set the encoding profile (from 0 to 3) (default main)
     baseline                     E..V.... 
     main                         E..V.... 
     high                         E..V.... 
     high444p                     E..V.... 
  -level             <int>        E..V.... Set the encoding level restriction (from 0 to 51) (default auto)
     auto                         E..V.... 
     1                            E..V.... 
     1.0                          E..V.... 
     1b                           E..V.... 
     1.0b                         E..V.... 
     1.1                          E..V.... 
     1.2                          E..V.... 
     1.3                          E..V.... 
     2                            E..V.... 
     2.0                          E..V.... 
     2.1                          E..V.... 
     2.2                          E..V.... 
     3                            E..V.... 
     3.0                          E..V.... 
     3.1                          E..V.... 
     3.2                          E..V.... 
     4                            E..V.... 
     4.0                          E..V.... 
     4.1                          E..V.... 
     4.2                          E..V.... 
     5                            E..V.... 
     5.0                          E..V.... 
     5.1                          E..V.... 
  -rc                <int>        E..V.... Override the preset rate-control (from -1 to INT_MAX) (default -1)
     constqp                      E..V.... Constant QP mode
     vbr                          E..V.... Variable bitrate mode
     cbr                          E..V.... Constant bitrate mode
     vbr_minqp                    E..V.... Variable bitrate mode with MinQP
     ll_2pass_quality              E..V.... Multi-pass optimized for image quality (only for low-latency presets)
     ll_2pass_size                E..V.... Multi-pass optimized for constant frame size (only for low-latency presets)
     vbr_2pass                    E..V.... Multi-pass variable bitrate mode
  -rc-lookahead      <int>        E..V.... Number of frames to look ahead for rate-control (from -1 to INT_MAX) (default -1)
  -surfaces          <int>        E..V.... Number of concurrent surfaces (from 0 to 64) (default 32)
  -cbr               <boolean>    E..V.... Use cbr encoding mode (default false)
  -2pass             <boolean>    E..V.... Use 2pass encoding mode (default auto)
  -gpu               <int>        E..V.... Selects which NVENC capable GPU to use. First GPU is 0, second is 1, and so on. (from -2 to INT_MAX) (default any)
     any                          E..V.... Pick the first device available
     list                         E..V.... List the available devices
  -delay             <int>        E..V.... Delay frame output by the given amount of frames (from 0 to INT_MAX) (default INT_MAX)
  -no-scenecut       <boolean>    E..V.... When lookahead is enabled, set this to 1 to disable adaptive I-frame insertion at scene cuts (default false)
  -forced-idr        <boolean>    E..V.... If forcing keyframes, force them as IDR frames. (default auto)
  -b_adapt           <boolean>    E..V.... When lookahead is enabled, set this to 0 to disable adaptive B-frame decision (default true)
  -spatial-aq        <boolean>    E..V.... set to 1 to enable Spatial AQ (default false)
  -temporal-aq       <boolean>    E..V.... set to 1 to enable Temporal AQ (default false)
  -zerolatency       <boolean>    E..V.... Set 1 to indicate zero latency operation (no reordering delay) (default false)
  -nonref_p          <boolean>    E..V.... Set this to 1 to enable automatic insertion of non-reference P-frames (default false)
  -strict_gop        <boolean>    E..V.... Set 1 to minimize GOP-to-GOP rate fluctuations (default false)
  -aq-strength       <int>        E..V.... When Spatial AQ is enabled, this field is used to specify AQ strength. AQ strength scale is from 1 (low) - 15 (aggressive) (from 1 to 15) (default 8)
  -cq                <int>        E..V.... Set target quality level (0 to 51, 0 means automatic) for constant quality mode in VBR rate control (from 0 to 51) (default 0)
  -aud               <boolean>    E..V.... Use access unit delimiters (default false)
  -bluray-compat     <boolean>    E..V.... Bluray compatibility workarounds (default false)

[poisondeathray]

You would use something like

ffmpeg -i "INPUT.ext" -pix_fmt nv12 -preset:v slow -c:v h264_nvenc -b:v ()k -bf 3 -refs 3 -an "OUTPUT.ext"

Where the ()k is in kbps, so -b:v 870kbps for 870kbps

nv12 is required for NVEnc, otherwise you get chroma errors. Slow preset automatically enables 2pass, high quality for NVEnc. But you still need to specify GOP size , bframes , reference frames . In ffmpeg , it's -g , -bf, -refs , respectively. Default GOP size is 250, same as x264, but that is not necessarily "optimal" for NVEnc. -an is no audio (no use encoding audio , if you're testing video) .

Play with the other options and settings, but the only "rule" is you just need to make sure the bitrates are the same (or very close) for a proper comparison. For example x264 has AQ enabled by default, you could play with the nvenc AQ options as well (which are disabled by default) . Sometimes you might be slightly off the bitrate targets, then you just tweak the bitrate (e.g. you might need to enter 900 to get actual 870 or soemthing like that) .

[CursedLemon]

Awesome, I appreciate the help immensely.