Artifacts after HEVC 10-bit encoding using NVENC

[Cryman]

Hello.

Recently I purchased a brand new GPU - AORUS GeForce GTX 1080 Ti. I found out that it supports HEVC 10-bit encoding, so I wanted to give that a try. Unfortunately, after encoding I noticed some artifacts, which occur in dark scenes and last one frame of the video. You can see them on these screenshots:




I was wondering if someone could help me figure out what might be the cause of these artifacts and how I can get rid of them.

Here is the MI of the source video:

ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4.1
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 4 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 2 h 2 min
Bit rate mode                            : Variable
Bit rate                                 : 29.5 Mb/s
Maximum bit rate                         : 37.0 Mb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.593
Stream size                              : 25.2 GiB (66%)
Language                                 : English
Default                                  : Yes
Forced                                   : No

And here is the MI of the encoded video:

ID                                       : 1
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main 10@L4@Main
Codec ID                                 : V_MPEGH/ISO/HEVC
Duration                                 : 2 h 2 min
Bit rate                                 : 3 689 kb/s
Width                                    : 1 920 pixels
Height                                   : 800 pixels
Display aspect ratio                     : 2.40:1
Frame rate mode                          : Constant
Frame rate                               : 23.976 (24000/1001) FPS
Standard                                 : Component
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 10 bits
Bits/(Pixel*Frame)                       : 0.100
Stream size                              : 3.15 GiB (95%)
Default                                  : Yes
Forced                                   : No
Color range                              : Limited

The command I'm using for encoding:

ffmpeg -hide_banner -i "<input_file>" -map 0:v:0 -map_chapters -1 -map_metadata -1 -vf "crop=1920:800:0:140" -vcodec hevc_nvenc -pix_fmt p010le -preset hq -profile:v main10 -rc constqp -global_quality 21 -rc-lookahead 32 -g 240 -f matroska Video_CQP21_LAF32_GOP240.mkv

[poisondeathray]

Does error occur always in the same spot (is error exactly repeatable on the encoded file if you open and close, or is it random ? )

Check if it's a display or decoding problem of the encoded file. e.g. try with different decoder/player MPCHC, VLC, potplayer, smplayer etc...

Check if it's a display or decoding problem of the source file - if input has the same error, then it's not NVEnc issue / or if you encode with other encoder using same decoder without errors, it's not NVEnc issue

Is card overclocked ? If so, check temps & voltage / repeat at stock speeds

[Cryman]

Does error occur always in the same spot (is error exactly repeatable on the encoded file if you open and close, or is it random ? )

Yeah, these artifacts occur in the same frames every time I open the video. However, if I do another encode of the video, they appear in different frames with dark colors, but still persist after I close the file and open it again.

Check if it's a display or decoding problem of the encoded file. e.g. try with different decoder/player MPCHC, VLC, potplayer, smplayer etc...

Tried playing the video in MPCHC, VLC, PotPlayer - artifacts occur in every player.

Check if it's a display or decoding problem of the source file - if input has the same error, then it's not NVEnc issue / or if you encode with other encoder using same decoder without errors, it's not NVEnc issue

Source file definitely doesn't contain these artifacts. I tried encoding to 8-bit AVC instead of 10-bit HEVC - the artifacts were still there.

Is card overclocked ? If so, check temps & voltage / repeat at stock speeds

Yeah, it has out-of-the-box overclocking, but the temps are all good.

I was wondering if it could be a color conversion issue.

[poisondeathray]

Check if it's a display or decoding problem of the source file - if input has the same error, then it's not NVEnc issue / or if you encode with other encoder using same decoder without errors, it's not NVEnc issue

Source file definitely doesn't contain these artifacts. I tried encoding to 8-bit AVC instead of 10-bit HEVC - the artifacts were still there.

8bit AVC using what ? ffmpeg nvenc through ffmpeg ?

If you encode using software x264 8bit AVC with ffmpeg libx264, and the artifacts are still there, then that suggests ffmpeg decoding issue . Then the next step would be test swapping decoders

[Cryman]

8bit AVC using what ? ffmpeg nvenc through ffmpeg ?

In case of 8bit AVC, I did a full hardware transcoding through FFmpeg using CUVID to decode and NVENC to encode.

If you encode using software x264 8bit AVC with ffmpeg libx264, and the artifacts are still there, then that suggests ffmpeg decoding issue . Then the next step would be test swapping decoders

Okay, I'll try encoding using libx264 and libx265.

[Cryman]

Alright, so I encoded the video using libx265, watched the whole video and didn't notice any artifacts. After examining the MI of the encoded file, I noticed some differences compared to the MI of the video encoded via hevc_nvenc.
Here is the MI of video encoded via libx265:

ID                                       : 1
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main 10@L4@Main
Codec ID                                 : V_MPEGH/ISO/HEVC
Duration                                 : 2 h 2 min
Bit rate                                 : 1 535 kb/s
Width                                    : 1 920 pixels
Height                                   : 800 pixels
Display aspect ratio                     : 2.40:1
Frame rate mode                          : Constant
Frame rate                               : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 10 bits
Bits/(Pixel*Frame)                       : 0.042
Stream size                              : 1.31 GiB (88%)
Writing library                          : x265 2.5+2-18fa144d453e:[Windows][GCC 7.1.0][64 bit] 10bit
Encoding settings                        : cpuid=1173503 / frame-threads=3 / numa-pools=8 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=1 / input-res=1920x800 / interlace=0 / total-frames=0 / level-idc=0 / high-tier=1 / uhd-bd=0 / ref=1 / no-allow-non-conformance / no-repeat-headers / annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / open-gop / min-keyint=23 / keyint=250 / bframes=3 / b-adapt=0 / b-pyramid / bframe-bias=0 / rc-lookahead=5 / lookahead-slices=5 / scenecut=0 / no-intra-refresh / ctu=32 / min-cu-size=16 / no-rect / no-amp / max-tu-size=32 / tu-inter-depth=1 / tu-intra-depth=1 / limit-tu=0 / rdoq-level=0 / dynamic-rd=0.00 / no-ssim-rd / no-signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / strong-intra-smoothing / max-merge=2 / limit-refs=0 / no-limit-modes / me=0 / subme=0 / merange=57 / temporal-mvp / no-weightp / no-weightb / no-analyze-src-pics / deblock=0:0 / no-sao / no-sao-non-deblock / rd=2 / early-skip / rskip / fast-intra / no-tskip-fast / no-cu-lossless / no-b-intra / rdpenalty=0 / psy-rd=2.00 / psy-rdoq=0.00 / no-rd-refine / analysis-reuse-mode=0 / no-lossless / cbqpoffs=0 / crqpoffs=0 / rc=crf / crf=20.0 / qcomp=0.60 / qpstep=4 / stats-write=0 / stats-read=0 / ipratio=1.40 / pbratio=1.30 / aq-mode=1 / aq-strength=0.00 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / no-const-vbv / sar=1 / overscan=0 / videoformat=5 / range=0 / colorprim=2 / transfer=2 / colormatrix=2 / chromaloc=0 / display-window=0 / max-cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / opt-qp-pps / opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / no-opt-cu-delta-qp / no-aq-motion / no-hdr / no-hdr-opt / no-dhdr10-opt / analysis-reuse-level=5 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=0 / no-limit-sao / ctu-info=0
Default                                  : Yes
Forced                                   : No

Now, this MI does not contain these lines:

Standard                                 : Component
Color range                              : Limited

They were present in the MI of the video encoded via hevc_nvenc.

So, does it mean that something goes wrong during color conversion via NVENC? Or is there a problem with FFmpeg's hevc_nvenc encoder?

[poisondeathray]

So, does it mean that something goes wrong during color conversion via NVENC? Or is there a problem with FFmpeg's hevc_nvenc encoder?

No, because you started with YUV 420, encoded YUV 420 (chroma subsampling is the same) , and the range is the same - it doesn't get altered unless you specify. Those info fields in mediainfo are just metadata tags (they are just "labels", don't affect the actual video). The only ACTUAL conversion done prior to encoding is an 8bit => 10bit depth conversion, but x265 was fine and it underwent everything else the same . So that suggests the problem is your NVEnc setup , card, or local configuration (other parts of your HW) . It might be a global NVEnc issue affecting everyone, or might be your specific card (try underclocking it)

I would try SW decode, NVEnc 8bit AVC encode next , and then rigaya's NVEncC to rule out a FFmpeg NVEnc implementation problem . If NVEncC works fine without those artifacts, that strongly suggests it's a ffmpeg implemntation problem

[poisondeathray]

also were you using -rc constqp -global_quality 21 everytime for the tests ? It's a buggy and you need to specify the min and max quantizers as well . I would run some vbr bitrate tests too in case that was causing it

[Cryman]

No, because you started with YUV 420, encoded YUV 420 (chroma subsampling is the same) , and the range is the same - it doesn't get altered unless you specify. Those info fields in mediainfo are just metadata tags (they are just "labels", don't affect the actual video). The only ACTUAL conversion done prior to encoding is an 8bit => 10bit depth conversion, but x265 was fine and it underwent everything else the same . So that suggests the problem is your NVEnc setup , card, or local configuration (other parts of your HW) . It might be a global NVEnc issue affecting everyone, or might be your specific card (try underclocking it)

Okay, I try underclocking it. I guess, I'll ask my friend, who has GTX 1050 Ti, to encode this video to rule out the problem with my GPU.

I would try SW decode, NVEnc 8bit AVC encode next , and then rigaya's NVEncC to rule out a FFmpeg NVEnc implementation problem . If NVEncC works fine without those artifacts, that strongly suggests it's a ffmpeg implemntation problem

Alright, I'll try to do that.

also were you using -rc constqp -global_quality 21 everytime for the tests ? It's a buggy and you need to specify the min and max quantizers as well . I would run some vbr bitrate tests too in case that was causing it

I did encode my video using -rc cbr_hq and -rc vbr_hq, but to no avail - the artifacts were still there. I also updated my hevc_nvenc encoder, and ran an encoding using -rc constqp -qp 21, but the end result still contained these artifacts.

[Cryman]

After a lot of encoding and testing my friend and I did, we found out that the problem has to do with GPU overclocking. The higher the GPU clock, the more artifacts appear in scenes with dark colors. After underclocking the GPU to the minimum possible clock value and running an encode, everything is fine - no artifacts whatsoever.
So why exactly higher GPU clock produces these artifacts when encoding via NVENC?

I was also wondering which NVENC encoder is more stable and reliable - FFmpeg's hevc_nvenc or NVEncC? You mentioned that there are problems with FFmpeg's CQP mode.

BTW, it's surprising that given the same encoding params libx265 produces files almost three times smaller than NVENC encoder. Say, using CQP 21 I got a 1,31 GB file via libx265 and 3,17 GB file via NVENC encoder.

[poisondeathray]

After a lot of encoding and testing my friend and I did, we found out that the problem has to do with GPU overclocking. The higher the GPU clock, the more artifacts appear in scenes with dark colors. After underclocking the GPU to the minimum possible clock value and running an encode, everything is fine - no artifacts whatsoever.
So why exactly higher GPU clock produces these artifacts when encoding via NVENC?

It's obviously an unstable overclock. This can occur with CPU encoding too. Heat and excessive voltage are the ones that can cause problems and damage, not just to the encode, but hardware. Again, check your temps. If you can , adjust the fan /cooling speeds . (Undervolting can also cause problems too, also crashing, but it's usually overclock with the accompanying voltage & heat that cause problems)

Since it's an aftermarket retail overclock , you should be able to set it at default overclock without errors. If not, I would return the card

I was also wondering which NVENC encoder is more stable and reliable - FFmpeg's hevc_nvenc or NVEncC? You mentioned that there are problems with FFmpeg's CQP mode.

BTW, it's surprising that given the same encoding params libx265 produces files almost three times smaller than NVENC encoder. Say, using CQP 21 I got a 1,31 GB file via libx265 and 3,17 GB file via NVENC encoder.

Stability wise, they are the same for NVEnc. But the default settings are different. NVEncC uses higher quality , slower settings by default. NVEncC also automatically converts to the required NV12 pixel format . ffmpeg does not, and you will get chroma artifacts unless you specify. But FFmpeg is much more versatile, has many more filters, able to take many more types of inputs directly, and is a "swiss army knife" .

The quantizer scaling can mean completely different things between encoders , it's not directly comparable. Filesize is almost meaningless unless you measure something like "quality" with it. But NVEnc HEVC doesn't use b-frames, so that severely negatively impacts compression efficiency.

[Cryman]

It's obviously an unstable overclock. This can occur with CPU encoding too. Heat and excessive voltage are the ones that can cause problems and damage, not just to the encode, but hardware. Again, check your temps. If you can , adjust the fan /cooling speeds . (Undervolting can also cause problems too, also crashing, but it's usually overclock with the accompanying voltage & heat that cause problems)

Since it's an aftermarket retail overclock , you should be able to set it at default overclock without errors. If not, I would return the card

Well, IDK, I'm no OC expert. The GPU has vendor OC and vendor software for easy OC, which I used. Temperature-wise, everything is fine - temps are not going higher than 60 °C while encoding. GPU performance in games is great and the temps are all good.

Stability wise, they are the same for NVEnc. But the default settings are different. NVEncC uses higher quality , slower settings by default. NVEncC also automatically converts to the required NV12 pixel format . ffmpeg does not, and you will get chroma artifacts unless you specify. But FFmpeg is much more versatile, has many more filters, able to take many more types of inputs directly, and is a "swiss army knife" .

The quantizer scaling can mean completely different things between encoders , it's not directly comparable. Filesize is almost meaningless unless you measure something like "quality" with it. But NVEnc HEVC doesn't use b-frames, so that severely negatively impacts compression efficiency.

Okay, good to know. Unfortunately, NVEncC comes with Trojans and malware in archive, so IDK about that.

Anyway, thanks for the help, man!

[poisondeathray]

Well, IDK, I'm no OC expert. The GPU has vendor OC and vendor software for easy OC, which I used. Temperature-wise, everything is fine - temps are not going higher than 60 °C while encoding. GPU performance in games is great and the temps are all good.

The % of people using it for encoding only is tiny compared for using it for games . But shadowplay makes use of it, so it should work as advertised. I personally wouldn't "let it slide", especially when you're paying a premium for a factory OCed card. You're about the only one reporting this, and these pascal cards have been out for quite a while now, so I would have suspected that it's defective card - if your friend didn't get errors too on a different card. You should check the nvidia forum if there are other user reports. Ideally, you should be able to OC the shaders / memory separately with a divider for the NVEnc SIP core, because that's the part doing most of the encoding work, and probably the part causing the errors

Unfortunately, NVEncC comes with Trojans and malware in archive, so IDK about that.

Where did you download it from ?

The original source on github , (and original binaries rigaya's onedrive) doesn't have malware, it's probably a false positive from your security software . But if you downloaded it from somewhere else it could certainly have malware injected

[Cryman]

The % of people using it for encoding only is tiny compared for using it for games . But shadowplay makes use of it, so it should work as advertised. I personally wouldn't "let it slide", especially when you're paying a premium for a factory OCed card. You're about the only one reporting this, and these pascal cards have been out for quite a while now, so I would have suspected that it's defective card - if your friend didn't get errors too on a different card. You should check the nvidia forum if there are other user reports. Ideally, you should be able to OC the shaders / memory separately with a divider for the NVEnc SIP core, because that's the part doing most of the encoding work, and probably the part causing the errors

I'll look into it.

Where did you download it from ?

The original source on github , (and original binaries rigaya's onedrive) doesn't have malware, it's probably a false positive from your security software . But if you downloaded it from somewhere else it could certainly have malware injected

I followed a link provided in this thread. My 360 Total Security reported auo_setup.exe as Win32/Trojan.044 and other stuff in auo folder as malware.

Anyway, I wanted to ask what NVENC encoding params would you recommend for optimal quality/file_size ratio. Should I stick to CQP or try out 2-pass VBR?

[poisondeathray]

I followed a link provided in this thread. My 360 Total Security reported auo_setup.exe as Win32/Trojan.044 and other stuff in auo folder as malware.

It's a false positive . That onedrive link is Rigaya's personal storage. You can build it directly from source at github if you want to be sure



Looks like a known issue with NVEnc HEVC and overclocking on the 1080ti . But other users are reporting h.264/AVC is fine, yet you had problems with it...
https://forums.geforce.com/default/topic/1003100/gamestream/grey-squared-artifacts-aft...-to-1080-ti/1/

They need something like a divider that prevents the NVEnc SIP from being overclocked, only shaders, memory affected separately . Or they need better QC control, both NVidia and partners



CQP requires you to specify the min/max quantizers . When you alter those you get vastly different encodes. It ends up being mostly the effect of setting the min max values, not the cqp level . You can play with the values and see if you find a level that is acceptable. 2pass vbr (using -preset slow in ffmpeg activates internal 2pass hq) is the recommended high quality mode by nvidia. But then you need to specify a bitrate. What you bitrate is required changes depending on the source and what you're trying to encode. Fast action gameplay would need higher bitrate , than say, a turn based game. (or slow drama would require less bitrate than action movie) to look a certain "quality level". How do you know what appropriate bitate to use? good question . You don't.

It's very fast , but not a very good encoder. The lack of b-frames is a severe handicap. You probably need about 1.5-2x the filesize for equivalent quality as x265 . And at least 1 batch of cards has those grey block artifact issues with HEVC . I guess people don't care, or encoding is an afterthought for Nvidia

[Cryman]

It's a false positive . That onedrive link is Rigaya's personal storage. You can build it directly from source at github if you want to be sure

Maybe, maybe not. There are some signature detections on VirusTotal. Have it caused any problems on your computer?

Looks like a known issue with NVEnc HEVC and overclocking on the 1080ti . But other users are reporting h.264/AVC is fine, yet you had problems with it...
https://forums.geforce.com/default/topic/1003100/gamestream/grey-squared-artifacts-aft...-to-1080-ti/1/

They need something like a divider that prevents the NVEnc SIP from being overclocked, only shaders, memory affected separately . Or they need better QC control, both NVidia and partners

Yeah, they say it's an issue with PureVideo SIP on GP102 family cards (NVIDIA TITAN Xp, TITAN X, GeForce GTX 1080 Ti) that causes these problems with grey artifacts. Which is most unfortunate.

CQP requires you to specify the min/max quantizers . When you alter those you get vastly different encodes. It ends up being mostly the effect of setting the min max values, not the cqp level . You can play with the values and see if you find a level that is acceptable. 2pass vbr (using -preset slow in ffmpeg activates internal 2pass hq) is the recommended high quality mode by nvidia. But then you need to specify a bitrate. What you bitrate is required changes depending on the source and what you're trying to encode. Fast action gameplay would need higher bitrate , than say, a turn based game. (or slow drama would require less bitrate than action movie) to look a certain "quality level". How do you know what appropriate bitate to use? good question . You don't.

To speak the truth, I'm a little confused here. Isn't the essence of CQP to set one value - the quantization parameter (or three values for I, P and B frames with the latter missing in case of H.265 encoding done by NVENC)? And isn't setting min/max quantizers for VBR?

2pass vbr (using -preset slow in ffmpeg activates internal 2pass hq) is the recommended high quality mode by nvidia. But then you need to specify a bitrate.

I tested out -preset slow, and it seems like it produces pretty much the same result as -preset medium (hq). I used these params:

-c:v hevc_nvenc -pix_fmt p010le -preset slow -rc constqp -qp 21 -rc-lookahead 32

[poisondeathray]

Maybe, maybe not. There are some signature detections on VirusTotal. Have it caused any problems on your computer?

No problems. I think they are false positives. But I don't blame you for feeling uncomfortable using it

To speak the truth, I'm a little confused here. Isn't the essence of CQP to set one value - the quantization parameter (or three values for I, P and B frames with the latter missing in case of H.265 encoding done by NVENC)? And isn't setting min/max quantizers for VBR?

Constant quantization, yes . But you were using -global_quality earlier to control it , correct ? I only tested AVC, but others had similar observations, that nothing affected it except for adjusting the min/max quantizers. All the other settings were overriden.

People want to approach x264/x265's "constant quality" mode , or crf, not quantization . There, the mb quantizers fluctuation and vary around a level, giving you a rough similar average "quality" level . Constant quantization is different, the quantizer is the same, but the quality fluctuates. Some frames might pixelate, others might be overallocated. It's an inefficient method of rate control and encoding as bits are wasted and could have been allocated in a better fashion

2pass vbr (using -preset slow in ffmpeg activates internal 2pass hq) is the recommended high quality mode by nvidia. But then you need to specify a bitrate.

I tested out -preset slow, and it seems like it produces pretty much the same result as -preset medium (hq). I used these params:

-c:v hevc_nvenc -pix_fmt p010le -preset slow -rc constqp -qp 21 -rc-lookahead 32

For -preset medium vs slow, was this at the same bitrate (filesize) ? . In theory, a constant quantizer is 1 pass anyways, and 2 passes shouldn't affect distribution (preset slow activates the internal 2 pass). It should affect VBR 2pass. Recall the earlier min/max qp comments, you get the same results regardless when using that mode. e.g 21 gave same results 40 unless you specify min/max. Other custom settings were also ignored when you used that mode. Mind you that was for AVC, it might be slightly different for HEVC.

[poisondeathray]

3/62 say it's infected, am I reading that right ? I think it's a false positive . I would think >50% would flag it , if it was truly a virus ?

Also , for true constant quantization, the -rc-lookahead would be useless too, it's not going to impact any decisions because the quantizer is already chosen and set in stone, analgous to how 2pass wouldn't affect anything either. I think the only thing worse than constant quantization (I mean quality/efficiency wise) would be a CBR mode.

[Cryman]

Constant quantization, yes . But you were using -global_quality earlier to control it , correct ? I only tested AVC, but others had similar observations, that nothing affected it except for adjusting the min/max quantizers. All the other settings were overriden.

Yeah, I experienced that problem at the very beginning. Nothing affected it, there was a fixed bitrate at 2000 kbps. After searching on different forums, I found out that global_quality is the way. But with the latest rebuild of my FFmpeg and tools included in it, I started using 'qp' parameter.
It says in hevc_nvenc help:
-qp <int> E..V.... Constant quantization parameter rate control method (from -1 to 51) (default -1)

After doing some encodes and testing, it seems like the latest version of hevc_nvenc produces roughly the same results with -global_quality 21, -qp 21, -qmin 21 -qmax 21.

People want to approach x264/x265's "constant quality" mode , or crf, not quantization . There, the mb quantizers fluctuation and vary around a level, giving you a rough similar average "quality" level . Constant quantization is different, the quantizer is the same, but the quality fluctuates. Some frames might pixelate, others might be overallocated. It's an inefficient method of rate control and encoding as bits are wasted and could have been allocated in a better fashion

Okay, so if I understand it correctly, -rc constqp enables constant quality mode, right? The one which gives a similar average quality level. Because that what I'm after. Mostly.

For -preset medium vs slow, was this at the same bitrate (filesize) ? . In theory, a constant quantizer is 1 pass anyways, and 2 passes shouldn't affect distribution (preset slow activates the internal 2 pass). It should affect VBR 2pass. Recall the earlier min/max qp comments, you get the same results regardless when using that mode. e.g 21 gave same results 40 unless you specify min/max. Other custom settings were also ignored when you used that mode. Mind you that was for AVC, it might be slightly different for HEVC.

In case of -rc constqp, presets medium and slow give roughly the same file size. For VBR the only params that matter are -qmin and -qmax, right?

3/62 say it's infected, am I reading that right ? I think it's a false positive . I would think >50% would flag it , if it was truly a virus ?

Yeah, 3 out of 62. One can only hope that it isn't. Anyway, as I get it, that's the audio encoder. So far, I'm happy with audio encoders FFmpeg offers.

Also , for true constant quantization, the -rc-lookahead would be useless too, it's not going to impact any decisions because the quantizer is already chosen and set in stone, analgous to how 2pass wouldn't affect anything either. I think the only thing worse than constant quantization (I mean quality/efficiency wise) would be a CBR mode.

So, setting params like -rc-lookahead 32 -g 240 along with -rc constqp will not have any effect on the final result?

[poisondeathray]

Okay, so if I understand it correctly, -rc constqp enables constant quality mode, right? The one which gives a similar average quality level. Because that what I'm after. Mostly.

Not what it says - constant qp - which is not constant quality.

  -rc                <int>        E..V.... Override the preset rate-control (from -1 to INT_MAX) (default -1)
     constqp                      E..V.... Constant QP mode

For VBR the only params that matter are -qmin and -qmax, right?

For VBR, potentially all settings can affect the encode . Including GOP size, references, (b-frames if you had any for HEVC, maybe in the future) ; even the additional settings like AQ (temporal and spatial) modes would have effect, but be disabled in constqp mode (because the C means constant, not A - adaptive)

Also , for true constant quantization, the -rc-lookahead would be useless too, it's not going to impact any decisions because the quantizer is already chosen and set in stone, analgous to how 2pass wouldn't affect anything either. I think the only thing worse than constant quantization (I mean quality/efficiency wise) would be a CBR mode.

So, setting params like -rc-lookahead 32 -g 240 along with -rc constqp will not have any effect on the final result?

You wouldn't expect rc-lookahead to do anything, because the quantization decisions have already been made - they are Constant.

But you would expect -g to have effect. It should respect that and place an I-frame every 240

[Vince300]

You're about the only one reporting this, and these pascal cards have been out for quite a while now, so I would have suspected that it's defective card - if your friend didn't get errors too on a different card.

I just found out about this thread, I have the same exact problem on a ZOTAC GeForce GTX 1080 Ti AMP Extreme Core Edition, which comes with factory overclocking (1721MHz by default). Underclocking GPU and memory seems to fix the problem.

[ibbanez]

I was actually looking everywhere about this as I just started testing h265 encoding on the GPU, but noticed these random artifacts. I will try stock speeds. Its crazy because my 1080ti is watercooled and never goes above 50 degrees. It does have a high overclock at 2000 and the GDDR is at 11,800. But I will test fully unclocked and report the results. Sad day

[Mart666]

Exatly same artifacts problems encoding using NVENC. I'm using Staxrip 1.7.0.0. But my 1080TI is a Founder's Edition, not overclocked...

There is a way to increase quality compression with nvenc in staxrip (like forcing preset like "slow" to nvenc) to increase quality ?

[ibbanez]

So one wierd thing that I noticed, this seemed to happen with recordings that were done with OBS as the base recording, then re-encoding with Staxrip, but if I use Shadowplay, then when I use Staxrip, I don't have any issues, while still being fully overclocked to over 2000 Mhz. Its weird. i've re-encoded about 40 files with 265 while overclocked since I switched back to Shadowplay for some of my longer playthroughs. I don't understand it, but it maybe on the recording settings that Shadowplay is using.

[Mart666]

My sources are always bluray remux in mkv that i open in staxtip directly. I've made a lot of tests, whatever the source, it gives me artifact in dark areas that are not in the source video.

[Bernix]

anyone tried two pass nvenc?
EDIT: see https://forum.videohelp.com/threads/385859-QuickSync-H-264-Settings#post2502403
there are two links to txt files what is supported in nvenc h264 and nvenc h265 by ffmpeg.
BTW OP's 2pass capture improved quality of picture significantly.
O.k. I dind't saw you actually talk about it. He was using 2pass at CBR, which seems like nonsense but it worked for him.

[Mart666]

Thanks for info, in staxrip i'll add this parameters on command line :

in VBRHQ mode : -rc vbr_2pass

in CQP mode : -2pass

Edit : if adding "-rc vbr_2pass" i get error "Invalid option : -rc"

If i do like others parameters i see in command line i have to write "--" for option. So if i add "--rc vbr_2pass" i get error "Unknown Option : --rc"

What is the option for two pass encoding in nvenc 3.22 ?

[Bernix]

Found this.
https://ffmpeg.zeranoe.com/forum/viewtopic.php?t=3729

So you have to somehow also add bitrate it seems.

Also -rc-vbr_2pass can maybe work. Don't know.

In OBS, that is capture software there is just check box.

[Mart666]

Got it works with your help ! I added this to command line : --fullrange --vbr2 4500
I encode in VBRHQ at 4500Kbps. Now encoding :

D:\StaxRip-x64\Apps\NVEnc\NVEncC64.exe --vbrhq 4500 --codec h265 --preset quality --lookahead 32 --max-bitrate 0 --fullrange --vbr2 4500 --crop 0,140,0,140 -i "I:\video.mkv" -o "I:\video_out.h265"

NVEncC (x64) 3.22 (r663) by rigaya, Sep 23 2017 09:17:29 (VC 1900/Win/avx2)
OS Version Windows 10 x64 (16299)
CPU Intel Core i7-3770K @ 3.50GHz [TB: 4.10GHz] (4C/8T)
GPU #0: GeForce GTX 1080 Ti (28 EU) @ 1582 MHz (388.31)
NVENC / CUDA NVENC API 8.0, CUDA 9.1, schedule mode: auto
Input Buffers CUDA, 36 frames
Input Info avcuvid: H.264/AVC, 1920x1080, 24000/1001 fps
Vpp Filters copyDtoD
Output Info H.265/HEVC main @ Level auto
1920x800p 1:1 23.976fps (24000/1001fps)
Encoder Preset quality
Rate Control VBRHQ
Bitrate 4500 kbps (Max: 11520 kbps)
Target Quality auto
Initial QP I:20 P:23 B:25

I'll check the video quality and the artifacts this evening. Thanks !!

[Bernix]

Glad you manage it. My help on other hand is doubtful

Bernix