Video Encoding leads in strange artefacts when using hybrid

[Platos]

I wonder why i got strange artefacts in hybrid while upscaling with LD-Anime Compact (it happens also with other models). Then i found out, it has sth. to do with the encoding. When i use lossles it has absolutely no artefact. if i higher the bitrate, the artefacts will be less.

Ok, now you think, then use higher bitrate. But the problem is, when i do LD-Anime Compact with lossles and then i use this file, put in handbrake and use Nvenc 10bit h265, slowest, 1500kBit/s, it has absolutely none (of these) artefacts. When i put the same lossles file into hybrid, apply no filters and set the same encoding-settings, i get alot artefacts. I will do a slider picture, that you see it directly. Handbrake has also "normal" compression-artefacts, but i talk about these very strange colored "square" artefacts. I know it can appear block-artefacts on compression, but it does only happen on hybrid and its strange so...

And also: On hybrid it is waaay worse, when i directly use 1500kBits without first doing a lossles-version, and that's what i actually want to do (i will do a screenshot of it too)

Here are the 4 Pictures: https://imgsli.com/MTc1NjQ4

Here the 3 files: https://uploadnow.io/f/HFDPkW5

I have also a debug file from the lossles-> to 1500kbits encoding. Do you want that ? Is that safe to post in public?

Edit: nearly i forgot: I did a complete deinstall of hybrid because of this issue and reinstalled it (with the newer version 02.05.2023 i got). But now, after the reinstall, when i want to start a job, i get the message: "Can't properly parse mkvmerge version 76.0, this might cause problems!".

I installed mkvCleaver and mkvToolNix before i reinstall hybrid. Maybe it has sth to do with it. But i did use hybrid before and after i isntalled these two software.

But at the end this problem with the artefacts were before the reinstall and are now still get produced.

[poisondeathray]

HB encode is actually 8bit and uses ~15% more video bitrate

Try using the same bit depth, same bitrate to make it a valid comparison . +/- 0.5% bitrate might be ok, but +/- 15% definitely not ok for comparison

[Platos]

HB encode is actually 8bit and uses ~15% more video bitrate

Try using the same bit depth, same bitrate to make it a valid comparison . +/- 0.5% bitrate might be ok, but +/- 15% definitely not ok for comparison

Ups. It's late here. Did a new one:

But actually then it shows it even more. If handbrake does still a better job with 8 bit. Because hybrid is doing the artefact, not handbrake.

Have to sleep now. 4.45 a.m here.

last edit: With 1400kbit/s it now has similar bitrate. i have to watch it tomorrow:

[jagabo]

Even better, encode lossless and compare. That separates processing and encoder issues (except maybe uncontrolled colorspace conversions).

[poisondeathray]

What were the encoding settings for Hybrid's NVEnc ? Does hybrid use Rigaya's NVEncC or ffmpeg NVEnc ? You can look at a log file for more details

Or if you are going for lower bitrates with better quality, you can use x265 or AV1

[Selur]

On Windows Hybrid uses Rigaya's NVEncC.

[Platos]

Even better, encode lossless and compare. That separates processing and encoder issues (except maybe uncontrolled colorspace conversions).

Thats not the point. The Problem is, that it leads in strange Artefacts, when i use Hybrid. I knoe its fine, when i use lossles. So i already used lossles and its finr, so i already knoe its a encoding-issue (i think).

On Windows Hybrid uses Rigaya's NVEncC.

Would it be helpfull to post a debug file? Is it save to post it in public or is this meant to send in private message ?

What were the encoding settings for Hybrid's NVEnc ? Does hybrid use Rigaya's NVEncC or ffmpeg NVEnc ? You can look at a log file for more details

Or if you are going for lower bitrates with better quality, you can use x265 or AV1

I thought i was using x265 nvenc. But it was so late, maybe i did some strange things.

I used variable bitrate 1.5mbit/s. Quality-preset, main10, 10 bit, quarter pixel, no 2 pass.

If it's using rigayas encoder i can also use fastflix to compare, if you want.

I will do new compares in half an hour and post it in an hour.

(Edit: if you mean using software x265 then i could do that, but it would not solve the problem, because i would just "evade" from it. And i also want to avood that, because software x265 is slow, av1 is even more slow. At least on my test with fastflox some weaks ago.

[Selur]

Would it be helpfull to post a debug file?

If you want to know what is happening in Hybrid, a debug output is the only way to get inside into that,...
(side note: if you messed with the frequencies of your gpu, issues are bound to happen)

[Platos]

Ok, i did now a new test with the lossless file i produced yesterday. I did use hybrid and used nvenc, h265, 10bit, 2 full passes, quality preset, 1500kbits variable bitrate. Then i used Fastflix, which has also Rigayas encoder. I used the same settings.

Then i also did again an upscale with LD-Anime Comapct (happens on other models too) and do directly use 1500kbit/s with same settings (isntead of doing first a lossles and encode then again).

Here is a slidercompare (look on his fists): https://imgsli.com/MTc1Nzcy

Here are all the videos (including un-upscaled source): https://uploadnow.io/f/M8Yk3m0

Would it be helpfull to post a debug file?

If you want to know what is happening in Hybrid, a debug output is the only way to get inside into that,...
(side note: if you messed with the frequencies of your gpu, issues are bound to happen)

Ok. the _2 file is the debug, when i use upscaling and a 1500kbit/S bitrate. the _1 file is, when i do apply no filter and encode a lossless upscaled file to 1500kbit/s.
Edit: And i deactivate all manual tweeking like gpu undervolting. All default.
Edit2: and i have also the newest gpu driver. But it happened also with the last one.

[Selur]

What I did:

• Start Hybrid
• load your source.mkv
• set resizing to 1426x1080
• made sure I use LWLibavSource like you.
• set 2x_LD-Anime_Compact_330k_net_g.pth as model in VSGAN (preview looked fine)
• disabled 32bit in VSGAN (preview looked fine)
• configured x265,
  got a:
  

  ---------------------------
  Info
  ---------------------------
  Current settings would result in a file size below 1 MB.
  -> changing Filesize to 1MB
  ---------------------------
  OK   
  ---------------------------

  message (and ignored it)
  to match your settings.
  

  NVEnc --y4m -i - --fps 23.976 --codec h265 --profile main --level auto --tier high --sar 1:1 --lookahead 16 --output-depth 10 --multipass "2pass-full" --vbr 1500 --max-bitrate 800000 --gop-len 0 --ref 3 --weightp --bframes 3 --bref-mode disabled --no-b-adapt --mv-precision Q-pel --preset quality --colorrange limited --colormatrix bt709 --cuda-schedule sync

• create a job with the current settings.

Attached the output, I got a file, which looks similar to what Fastflix produced (looking at your screenshots), still has some of those compression artifacts but less.

I additionally did an encode ( reeencode_with_bframes) where I adjusted the settings a bit, to be more what I would use for animes:

NVEnc --y4m -i - --fps 23.976 --codec h265 --profile main --level auto --tier high --sar 1:1 --lookahead 16 --output-depth 10 --multipass "2pass-full" --vbr 1500 --max-bitrate 800000 --aq --aq-strength 5 --aq-temporal --gop-len 0 --ref 7 --weightp --bframes 4 --bref-mode middle --mv-precision Q-pel --preset quality --colorrange limited --colormatrix bt709 --cuda-schedule sync

the resulting file is larger, but doesn't seem to have the compression artifacts.

=> from what I see, there is no bug in Hybrid, but NVEncC is running out of bit rate with that content and resolution.

Cu Selur

[Platos]

With your reencode file i see no of these artefacts, just "normal" compression artefact. There are some "dots", but that's because of the source.

I dont understand, why you dont get these huge squares i get with the same settings. it's still there. Maybe i reeinstall the driver from my gpu or so. You rememmber i had some strange artefacts too when i used sRestore and it "solved" by restart software ? Maybe this is some strange driver problem in nvenc, because it does not appear, if i use x265 cpu with same bitrate. Maybe i just use that. Maybe i also try a complet reinstall of the gpu-driver.

Question: Example: When my model (ld-anime compact) has 10fps on upscaling and my cpu can get 11 fps on encoding with x265 software, it should actually not slow down the upscaling, right? So that means, as long my cpu can encode same speed as my gpu can upscale with the model, it should be the same speed (let's say i dont use any other filter and let's ignore the cpu-usage of the upscaling itselve for simplify).

[Selur]

x265 doesn't use the gpu, also decoding&filtering comes before the encoding.
You are probably fine, with using x265, since your filtering is basically gpu only.

Using x265 with:

x265 --input - --output-depth 10 --y4m --profile main10 --level-idc 5.1 --limit-modes --opt-ref-list-length-pps --bframes 16 --lookahead-slices 0 --pass 1 --no-slow-firstpass --bitrate 1500 --opt-qp-pps --cbqpoffs -2 --crqpoffs -2 --nr-intra 500 --nr-inter 500 --qpfile GENERATED_QP_FILE --limit-refs 0 --ssim-rd --psy-rd 2.50 --rdoq-level 2 --psy-rdoq 10.00 --dynamic-rd 2.00 --aq-mode 4 --aq-strength 1.5 --no-cutree --vbv-maxrate 160000 --vbv-bufsize 160000 --deblock=-1:-1 --limit-sao --no-repeat-headers --range limited --colormatrix bt709 --sar 1:1 --stats "J:\tmp\reeencode_x265_generated.stats" --multi-pass-opt-analysis --multi-pass-opt-distortion --analysis-reuse-file "J:\tmp\reeencode_x265_generated.analysis" --output NUL

x265 --input - --output-depth 10 --y4m --profile main10 --level-idc 5.1 --limit-modes --no-early-skip --opt-ref-list-length-pps --bframes 16 --ref 6 --lookahead-slices 0 --pass 2 --bitrate 1500 --opt-qp-pps --cbqpoffs -2 --crqpoffs -2 --nr-intra 500 --nr-inter 500 --limit-refs 0 --ssim-rd --psy-rd 2.50 --rdoq-level 2 --psy-rdoq 10.00 --dynamic-rd 2.00 --aq-mode 4 --aq-strength 1.5 --no-cutree --vbv-maxrate 160000 --vbv-bufsize 160000 --deblock=-1:-1 --limit-sao --no-repeat-headers --range limited --colormatrix bt709 --sar 1:1 --stats "J:\tmp\reeencode_x265_generated.stats" --multi-pass-opt-analysis --multi-pass-opt-distortion --analysis-reuse-file "J:\tmp\reeencode_x265_generated.analysis" --output "J:\tmp\reeencode_x265.265"

I get: reeencode_x265.mp4, which also has tons of arifacts.

When I aim for 1MB output (reeencode_x265_1MB) the output looks better again.
-> your issue is your bitrate! (1500Mbit/s isn't enough for that content at 1426x1080, with smoother edges&co you can get low bitrates, but if you sharpen the edges bite consumption goes up,...)
using NVEncC with av1 and:

NVEnc --y4m -i - --fps 23.976 --codec av1 --sar 1:1 --output-depth 10 --vbr 0 --vbr-quality 23.00 --aq --aq-strength 5 --aq-temporal --gop-len 0 --ref 7 --nonrefp --weightp --bframes 4 --bref-mode middle --mv-precision Q-pel --preset quality --colorrange limited --colormatrix bt709 --cuda-schedule sync --keyfile GENERATED_KEY_FILE --output "J:\tmp\reencode_av1.av1"

I get reencode_av1.mp4 (which looks fine)

Cu Selur

[Platos]

I tried now CPU x265, but it's to slow. Even on ultra-fast preset it's a bit slower than nvenc, but looks then much uglier. I have btw. ~90-95% CPU Usage with LD-Anime Compact, no other filter and NVEnc H265. Have 12400f CPU (6 Cores 12 Threads). And it does look like its doing the encoding at same time as upscaling, because it uses the Video Processing unit at the same time (while it doesnt, when i use CPU x265)

So i use nvenc now again, but higher bitrate and use H265 and 5000kBit/s. That looks good. AV1 does not look better in my eye at same bitrate.

Isn't it the best to use variable bitrate ? So for the most quality per size, it's the best to go for variable bitrate i guess? Because then each scene get as much, as it needs (in the range of the bitrate).

[Selur]

1pass encoding is only as good as the rate adjustment algorithm that distributes the bit rate by guessing how complex the next scene will get. Usually, 1pass bit rate encoding isn't good.
1pass vbr is better hand 1pass abr or 1pass cbr, but it's usually still not really good.

Because then each scene get as much, as it needs (in the range of the bit rate).

That will only work if your encoder can at least analyze the whole scene, otherwise it's guessing.
The lookahead (= number of frame the encoder looks at before making rate decisions) can be max 32 frames with NVEnc, so unless your scene is really short the encoder will need to guess.
You get the best quality for a fixed bit rate / target size using 2(or more) pass encoding.
When target size isn't the issue, most folks would go for 'fixed quality' or 'fixed rate control' with whatever setting that keeps enough quality while not getting too large. (for most folks this is usually a range of 18-23, if the range goes from 0-52)

Cu Selur

[Platos]

So i mean specific for Nvenc H265. I can use there 2pass for variable bitrate and constant bitrate only. And hybrid says it uses the first pass to determine which part of the frame needs how much bitrate (or so).

So at least with Handbrake i get alot better results per filesize, when i use bitrate instead of quality factor on NVEnc H265.

Anyway, thanks for your help here again

[Selur]

2pass encoding in NVEnc isn't what one usually calls 2pass encoding. What NVEnc does in 2pass, is optimize the encoding of a frame, not optimize the bit rate distribution over a scene.
If you get better results with Handbrake use that. Hybrid totally isn't a tool for everybody, it's not meant to be.

[jagabo]

Even better, encode lossless and compare. That separates processing and encoder issues (except maybe uncontrolled colorspace conversions).

i already used lossles and its finr, so i already knoe its a encoding-issue

Sorry. I missed that in the first post.

[Platos]

2pass encoding in NVEnc isn't what one usually calls 2pass encoding. What NVEnc does in 2pass, is optimize the encoding of a frame, not optimize the bit rate distribution over a scene.
If you get better results with Handbrake use that. Hybrid totally isn't a tool for everybody, it's not meant to be.

No, i use hybrid. It works fine with 5000KBit. I just mean, i testet on Handbrake variable vs Quality (i did not test on hybrid).

It works fine on Hybrid. Great Software

But thanks for 2 pass clarification. I will post soon some more CGAN Compact Model-compare in the other thread if you are interested.