Tweaking best settings for NVENC h.264 with Staxrip

[salvo00786]

Hi... I have tried so many settings for NVENC in Staxrip. Now I will post my screen with all these settings and after that, I will ask two questions...

Now, please can you tell me:

1) Why when I use deband, why it add static noise like green color (I don't know if my eyes are wrong)?
2) Can you suggest me some other better settings?

[Selur]

Not using StaxRip at all, but here are

1) Why when I use deband, why it add static noise like green color (I don't know if my eyes are wrong)?

You configured it to add dithering. Adding noise is what dithering does. Since you are aiming for 8bit encoding dithering can make sense to avoid (re-)introducing banding during the encoding.

2) Can you suggest me some other better settings?

Better to archive, what? Atm. it seems like you simply set everything to the max.
Not knowing what your aim is and what sort of content you are using, what filtering you do to it in 'AviSynth/VapoutSynth' kind of hinders to others from suggesting much.
Personally, I would avoid setting 'B-Frames Ref. Mode' to 'Each' unless I aim for really bit rate starved encodings. Enabling 'Enable adapt. non-reference P frames insertion' also seems like a bad more, since this usually lowers general efficiency and is meant when encoding for streaming content
Since you are using 'AviSynth/VapoutSynth' adding debanding or other filtering through the encoder instead of using 'AviSynth/VapoutSynth' seems a bad move. I would usually recommend to only use encoder side filtering if the NVEncC is also used for the decoding of the source.
=> you might want to read up on what the parameters do, before blindly enabling them.

Cu Selur

[salvo00786]

Thanks Selur. I want to use this for high quality conversion of movies from bluray 1080p to 720p.

I know that the dither is Noise, but why I see that this noise is green on the black scenes?

1)For the first settings, I used Each, because I thought it was better than Middle, please explain to me the difference between Middle and Each, and why you would avoid it, thanks.

2)For the second setting 'Enable adapt. non-reference P frames insertion', is an error , because subsequently, I disabled it, but I hadn't saved the last setting. I hadn't blindly enabled the parameters, I tried to convert the same file about 50 times with all the various settings and tweaked my settings step by step, with all the dark scenes.

3)If I understand correctly, you are suggesting me to use f3kdb_neo or Gradfun instead of the deband included in NVENCC, right? What is your suggestion, what is the best between these two for my case?

[Selur]

I want to use this for high quality conversion of movies from bluray 1080p to 720p.

okay. I would recommend using a lower crf/vbr quality setting then, at least if you plan to play this back at a higher resolution later.

I tried to convert the same file about 50 times with all the various settings and tweaked my settings step by step, with all the dark scenes.

Okay, surprised you ended up with aq strength 0 then.

because I thought it was better than Middle, please explain to me the difference between Middle and Each, and why you would avoid it, thanks.

B-frames are stronger compressed than other frames, thus usually using them too much can destroy small details, which is why one usually would not use them so much unless one compresses animes/cartoons. Since you prefer 'vbr quality' this might not be an issue for you.

3)If I understand correctly, you are suggesting me to use f3kdb_neo or Gradfun instead of the deband included in NVENCC, right?

No. Use whatever works better for you. Both f3kdb_neo and Gradfun will also add dithering. Whether one or the other works better for you depends only on your source.
Using Vapoursynth/Avisynth filters usually has the advantage that it allows one to look at a filter preview to see the effect of the filters. Using NVEncCs own filter has the advantage that decoding, filtering and encoding can be done all on the gpu which speeds things up.

Cu Selur

[salvo00786]

I used Aq Strenght 0, because 0 is Auto if I understood correctly and I haven't tried other value. If I am wrong, I will try tonight. Let me know if I am Wrong, thanks.

[Selur]

No, 0=auto according to the doc, still I would recommend giving it a try.

[salvo00786]

Selur, I have another question. Please can you explain to me two resize?

Dither_resize (what is the effect?)

and how to use
Hardware Encoder in resize section? (I can't find anything on this that explain how it work and how to use it)

[Selur]

Sorry, can't answer that since, like I wrote: 'Not using StaxRip at all'

[salvo00786]

Oh Sorry, thanks, anyway.

For the 8 bit question, I use 8bit, only because the source is 8bit. I have tried to use conversion to 10 bit and effectively without deband, the results are very good.

1) If I use 10 bit instead of 8 bit, what cons can I have? Some players or some old pcs can't play the files or other problems?

2) Are there some pros in using h.264 or now is better to switch to h.265?

[Selur]

10bit H.264:
con: you basically will have no hardware decoding support when using H.264 with 10bit, so players that just supports hardware decoding will likely fail.
pro: better compression ratio, no banding caused by the encoding.
10bit H.265:
con: slower encoding, higher energy requirement during encoding. Since H.265 has 10bit in its main profile, all typical hardware decoders should support it.
pro: better compression ratio (due to H.265), no banding caused by the encoding.

=> unless you have old hardware, I would recommend using H.265 over H.264, especially for HD and higher resolutions.

Cu Selur

Ps.: note that vbr quality 25 with H.264 and vbr quality 25 with H.265 will not aim at the same 'quality'.

[salvo00786]

Selur, the strange facts is that my conversions are faster with h265 than h264. With h264 I have 150/160 fps, with h265 10 bit I have 220/230 fps in conversion (with hardcoded subtitle that require more processing).

If you see my settings, you can see that I disabled deblock. In my test, I have seen that I have converted movies, without any block, for this reason I haven't tried with deblock. What is your suggestion on this?

What is the meaning of: Ps.: note that vbr quality 25 with H.264 and vbr quality 25 with H.265 will not aim at the same 'quality'.

My thoughts are that h.265 at vbr 25 is far better than h.264 because h.265 is more efficient, but I don't know if I understood your sentence.

[Selur]

Selur, the strange facts is that my conversions are faster with h265 than h264. With h264 I have 150/160 fps, with h265 10 bit I have 220/230 fps in conversion (with hardcoded subtitle that require more processing).

Might simply be that the implementation on the vpu is better for h.265.

If you see my settings, you can see that I disabled deblock. In my test, I have seen that I have converted movies, without any block, for this reason I haven't tried with deblock. What is your suggestion on this?

Assuming this is the inloop deblocking of H.264 I would not disable it since it's a essential part of H.264.
For H.265 there is no deblock option, see: https://github.com/rigaya/NVEnc/blob/master/NVEncC_Options.en.md

My thoughts are that h.265 at vbr 25 is far better than h.264 because h.265 is more efficient, but I don't know if I understood your sentence.

That is correct. I just wanted to emphasize that vbr quality isn't an absolute quality measure.

Cu Selur

[salvo00786]

Selur, Probably there is an error in this page or this page is old: https://github.com/rigaya/NVEnc/blob/master/NVEncC_Options.en.md,
but if I want, I can check the Deblock option in H265. I tried some movies with h.265 10bit and the results are wonderful. Now I want to switch to h265 10bit. For the deblocking part is valid the same suggestion for the h.264, and is better to active it too for h265 or not?

[Selur]

More likely, Stax simply doesn't disable the option based on the NVEncC codec choice.
In general, both H.264 and H.265 should be used with deblocking on, depending on the encoder lowering the strength of the deblocking might help to limit too strong deblocking.

[salvo00786]

I have seen In an old thread that someone used edgelevel, to sharpen a little. It's good to use or is better to stay away from that option?

In slice decision there is a slices option that I have set on 1. What is the meaning of that option?

[Selur]

You have to test with your sources, personally I prefer Vapoursynth/Avisynth filters since this way I can see what is happening without having to encode beforehand.

[salvo00786]

Ok, thanks.
In slice decision there is a slices option that I have set on 1. What is the meaning of that option?

[Selur]

The number of slices tells the encoder to split the frame into X parts (=slices) and encode these separately, which potentially improves threading support, but also potentially lowers coding efficiency and can cause issues with filtering.

[salvo00786]

It is better to use 0 that is Auto or 1 that force the encoder to use 1?

What is a good vbr value for h265 10 bit 720p with the best ratio compression/quality? 23?

[Selur]

To both: no clue, try it.

[salvo00786]

Ok, I will try.

For quality purpose, It is better to check Strict GOP or not?

[Selur]

Open gop can potentially create better compression ratios at the same quality than closed gop.

[poisondeathray]

--strict-gop for nvencc refers to avoid GOP length variability, not open vs closed GOP

But the answer is the same - non strict (adaptive) is potentially better for compression

[salvo00786]

Hi Selur. Probably you are right about deblock. The file is identical with or without deblock, and during the conversion, if you read all the settings, the option deblock is not existent.