Transcodes in H265 SW/HW keeping more details/grain/quality

[einstein1969]

Greetings,

I have some coded video files with a tall bit-rates and I'm trying to codify them (transcode?) To save the space. Obviously the priority is the quality of the video.
To check the quality I am trying to use both PSNR/SSIM/VMAF and the eyes that have precedence.

I started coding using Staxrip and in particular I use NVENCC H265 of Rigaya that uses my RTX3060.
Coding so that you can check the Bitrate Max because the videos will be broadcast via wifi to my TVs.
In particular, I use the "Quality VBR" to control the maximum "bit-rate" and I would like to have more details /grain without having to increase the bit-rate too much.

But the results I am getting are not satisfying me.

So I'm considering using the X265 software coding (CPU and Non GPU).
I did some tests with "tune grain" but I'm not visually satisfied.

My questions are:

GPU
1) What are the optimal parameters to obtain more details/grain for the NVIDIA NVIDIA NVENCC CODEC of Rigaya (with API 12.2) in "Quality VBR" mode?

2) Why does VMAF generally increase if I don't use B-FRAME? (I compared two codes that have the same "bit-rates"/used space)

CPU
3) Since Tune Grain does not serve to keep/remove grain but only to prevent artifact in the event of an irregular grain, as I know if I have to apply it or not.

4) What other parameters can help me to get more details/grain in X265 for the same "bitrate"/used space?

[jagabo]

1) The best way is to use 10 bit encoding. Even if your source is 8 bit. This is true for both x265 and NVENCC. Fortunately, most players and devices that support HEVC also support 10 bit HEVC. The same is not true for AVC where few devices support 10 bit encoding.

2) B-Frames are the lowest quality frames. P-Frames have higher quality. I-Frames the highest. VMAF and all the other metrics are very sensitive to edges that aren't exactly in the same place as the original. You're eyes aren't. x264 and x265 trade positional accuracy for better visual quality.

3) If you are using bitrate based encoding the bitrate that's spent on retaining grain/noise has to come from somewhere else.

4) This might help: https://forum.videohelp.com/threads/390036-Can-the-smoothness-or-blurred-look-of-x265-...ed#post2527784

[sophisticles]

Regarding the the B frame question, by default I frames are encoded using the lowest quantizer, P frames use a higher quantizer and B frames use the highest quantizer.

Using x264 as an example, assume you set qp 20, I seem to recall the x264 developers saying years ago that this referred to the quantizer used by P frames.

There are two setting that you may have noticed, I/P ratio and P/B ratio. The first defaults to 1.4 and the second defaults to 1.3.

So if qp 20 refers to the P frame quantizer, then B frames will use 20x1.3 or qp 26 and the I frames will use 20/1.4 or about 14.

In the event that qp refers to I frames then a qp of 20 would result in P frames using qp of 20x1.4 or 28 and B frames using a qp of 28x1.3 or about 36.

You can see why quality decreases with different types of frames.

Tune grain changes these settings to I/P and P/B = 1.1, which results in more overall bit rate being used.

When encoding using nvenc look for a cqp setting that allows you to manually set quantizer parameters and manually set each frame type to the same number, i.e. if you want 18 for i frames, set them all to 18.

File sizes will get bigger but in the end quality is defined by how much bit rate is being used in any given region. All algorithms attempt to decide how best to use the bit rate allocated so if you want higher quality, use more bit rate.

Better yet, don't transcode at all.

Problem solved.

[einstein1969]

1) The best way is to use 10 bit encoding. Even if your source is 8 bit. This is true for both x265 and NVENCC. Fortunately, most players and devices that support HEVC also support 10 bit HEVC. The same is not true for AVC where few devices support 10 bit encoding.

2) B-Frames are the lowest quality frames. P-Frames have higher quality. I-Frames the highest. VMAF and all the other metrics are very sensitive to edges that aren't exactly in the same place as the original. You're eyes aren't. x264 and x265 trade positional accuracy for better visual quality.

3) If you are using bitrate based encoding the bitrate that's spent on retaining grain/noise has to come from somewhere else.

4) This might help: https://forum.videohelp.com/threads/390036-Can-the-smoothness-or-blurred-look-of-x265-...ed#post2527784

Thanks that you replied.
I read the link that you gave me and I realized that I am very ignorant and I miss many bases about coding in general.

And a doubt came to me by reading the HELP of the X265 where he talks about psychoVisual optimizations. (Among other things, I am not an English native speaker and I can't understand what is written even by automatically drawing it).

When I do the tests generally compare individual frames (use nvidia icat).

But maybe it would be better to see the movie on the move and directly on my TV. How can I do to prepare a audition with the converted video where I can see the different results of the various codes (with different parameters)?

What do you say should I open another thread just for this?

[einstein1969]

Regarding the the B frame question, by default I frames are encoded using the lowest quantizer, P frames use a higher quantizer and B frames use the highest quantizer.

Using x264 as an example, assume you set qp 20, I seem to recall the x264 developers saying years ago that this referred to the quantizer used by P frames.

There are two setting that you may have noticed, I/P ratio and P/B ratio. The first defaults to 1.4 and the second defaults to 1.3.

So if qp 20 refers to the P frame quantizer, then B frames will use 20x1.3 or qp 26 and the I frames will use 20/1.4 or about 14.

In the event that qp refers to I frames then a qp of 20 would result in P frames using qp of 20x1.4 or 28 and B frames using a qp of 28x1.3 or about 36.

You can see why quality decreases with different types of frames.

Tune grain changes these settings to I/P and P/B = 1.1, which results in more overall bit rate being used.

When encoding using nvenc look for a cqp setting that allows you to manually set quantizer parameters and manually set each frame type to the same number, i.e. if you want 18 for i frames, set them all to 18.

File sizes will get bigger but in the end quality is defined by how much bit rate is being used in any given region. All algorithms attempt to decide how best to use the bit rate allocated so if you want higher quality, use more bit rate.

Better yet, don't transcode at all.

Problem solved.

Thanks to you too for the answer.

I start from my consideration that I made in the answer to Jagabo. Maybe I'm wrong everything because static frame comparison.

Also about the B-FRAME I just wanted to say that everywhere I read that the Bframe help to increase the quality of the video. Even in the Nvidia documents of the latest "API 12.2" there is recommended to use B-FRAME to increase quality.

But I understood your example. But I don't understand why the developers have decided to reduce the quality of the B-FRAME. Too complicated to understand for me.

However, I will keep in mind to try the i/p and P/B relationships of 10% in my tests, both directly and with Tune Grain. Thanks a lot for this information!

As for the NVENC codec, unfortunately I cannot do it because I have to use QVBR as CQP does not allow you to limit the maximum bitrate and this is needed to not have the blocks during the broadcast on my TV via wifii.

Do you know if I can also change the parameters with QVBR? I only know the Codec Rigaya and I don't know if it allows it or if there are other GPU codecs that allow it.

My goal is to increase the quality to the maximum for the same bitrate by intervening on the parameters. Evidently every source has a different parameters set. Otherwise all that huge number of parameters would not be explained.
Once the optimal parameters have been found (trying to abstract from the type of video content) then I would intervene on the bitrate by increasing or decreasing it. We hope it can be done.

Unfortunately I have to transcode for space and band problems.

[einstein1969]

@jagabo
I will use the 10bit coding where they can.

About VMAF. I thought VMAF kept in understanding the visual quality of the moving video. Or even VMAF takes into account only the single instant frame?

[einstein1969]

I can't understand these pieces.

VMAF and all the other metrics are very sensitive to edges that aren't exactly in the same place as the original. You're eyes aren't. x264 and x265 trade positional accuracy for better visual quality.

Psycho-visual options
Left to its own devices, the encoder will make mode decisions based on a simple rate distortion formula, trading distortion for bitrate. This is generally effective except for the manner in which this distortion is measured. It tends to favor blurred reconstructed blocks over blocks which have wrong motion. The human eye generally prefers the wrong motion over the blur and thus x265 offers psycho-visual adjustments to the rate distortion algorithm.

in particular

The human eye generally prefers the wrong motion over the blur

[jagabo]

But I don't understand why the developers have decided to reduce the quality of the B-FRAME.

The basic idea is that you won't notice a few B frames of lower quality as a new P frame will come along soon and clean the picture up. All high compression codecs use this.

My goal is to increase the quality to the maximum for the same bitrate by intervening on the parameters.

Keep in mind that different videos require different amounts if bitrate. Just because you have one video that looks good a 5000 kbps doesn't mean all videos will look good at that bitrate. Some will look good a lower bitrates, some will require a lot more. And not all encoders will look equally as good at a particular bitrate. In my experience the hardware encoders look worse than the x264 and x265 software encoders (comparing AVC to AVC, or HEVC to HEVC, at the same bitrate).

[jagabo]

About VMAF. I thought VMAF kept in understanding the visual quality of the moving video. Or even VMAF takes into account only the single instant frame?

All the metrics predominantly compare a pixel at x,y in one frame to the corresponding pixel in the reference video. The more similar the pixels, the higher they rate the quality. When moving edges don't line up exactly in the encoded video it appears to the metric as a huge error. But when watching the video at normal playback speed (or even looking at individual frames without A/B switching) you usually won't notice the difference.

[sophisticles]

Also about the B-FRAME I just wanted to say that everywhere I read that the Bframe help to increase the quality of the video. Even in the Nvidia documents of the latest "API 12.2" there is recommended to use B-FRAME to increase quality.

You need to understand how I/P/B frames work:

https://en.wikipedia.org/wiki/Video_compression_picture_types

This is a good explanation of how frames work in general.

Since P and B frames don't contain as much information as I frames, they can use less bit rate.

In a bit rate constrained use case, where let's say you only want to use 5000kb/s, B frames improve overall quality because by having more frames that contain less information that I or P frames you can ensure that the information they do have has more bit rate to encode it.

In a quality based encoding, like CRF or QP, having more B frames results in smaller file sizes over all.

But like everything in life, you don't want too many B frames, or P frames for that matter, because eventually the scene changes so much that the original I frame is no longer a good reference point.

While i like hardware encoders, i think for your use case, x264 with the very fast preset, I/P and P/B ration manually set to 1 and a 2-pass encode should be a good starting point.

[einstein1969]

I am reading and rereading your advice and I opened a thread on purpose to make conversions and compare videos with the play and frame by frame. (https://forum.videohelp.com/threads/414453-simple-video-compare)

So I'm not stopped but I will take tests with your advice and if needed I will ask you for further help.

I also read the links you gave me and let's say I didn't understand them. But for the moment I don't want to put more meat on the grill.

[einstein1969]

I did a first test without intervening on the parameters that you recommended and on scenes with high movement where the bitrate is not evidently sufficient you can see as X265 CPU is superior of NVENC H265.

This is evident on the still image but I want to see if a difference is noted in the movement/play.

orig file: dolby-vision-amaze-(dolby-vision)-(www.demolandia.net).mp4 4k

resize 2160p / UHD -> 1920 x 1080, plus:

• nvcencC 7.50 --qvbr 28 --codec h265 --output-depth 10 --vbv-bufsize 45000 --max-bitrate 15000

  (9190.40 kbps)

• x265 3.6+2+13-9a3dac6e5-.Mod-by-Patman.-x64-avx2-msvc1937 --crf 22 --output-depth 10 --vbv-bufsize 45000 --vbv-maxrate 15000

  (9151.27 kb/s)

Let me know if the file generated by ICAT works: index.zip


[Attachment 78952 - Click to enlarge]


[Attachment 78953 - Click to enlarge]

nvenc in still image is horrible.

EDIT: I think I made of errors and the VBV parameters on X265 CRF are not valid/ignored, right?

[jagabo]

As I understand it, VBV parameters can be used with CRF encoding in both x264 and x265.

It's not surprising that x265 is better than NVEnc. That's been my experience and, I believe, the general consensus. The 4xxx series may do better than your 3060 though.