Tiff rgb48le to yuv420p making color banding. How to resolve?

[ForSerious]

My goal is to use Topaz Video AI to upscale my DVDs to FHD and maybe denoise some Blu-rays. I've been using it for years, and have built a script to streamline all the programs needed. I just need the final files to be able to play on my newish Roku player through Plex.

The issue that I have is that TVAI processes everything in rgb48le. I output that to tiff, then I use FFMPEG to convert it to H.265. When I put it to the standard yuv420p pixel format, I get color banding that's pretty easy to see in some scenes. If I set it to yuv420p10le, the banding goes away—but only on my computer since my TV does not support 10 bit. It displays the color banding.

Here's my command for the tiff to H.265 conversion:

ffmpeg.exe -hide_banner -stats_period 2.0 -nostdin -framerate 23.976 -y -i "input\%6d.tif" -c:v libx265 -crf 18 -pix_fmt yuv420p -preset slow -profile:v main -vf "zscale=pin=bt709:min=gbr:tin=bt709:rin=pc:d=3:f=5:p=709:t=709:m=709:r=tv:c=input,format=yuv420p" "output.mkv"

Here's the 8 bit:

Here's the 10 bit:


Everything I can find says to dither, but I'm already doing that with zscale. It doesn't seem to have any effect.
I did have some success with adding the deband filter, I feel like it shouldn't be needed though.

[ForSerious]

I might have found a few more details. rgb48le should be easily converted to a 12 bits per channel pixel format. So really, I just need to know how to get FFMPEG to dither from 12 to 8 bits per channel. Maybe I need to chain the filters like 12 -> 10 -> 8. I don't know. Everything online says dithering should happen automatically just by the 'format=yuv420p' filter.

[ForSerious]

I think chaining the format filter helped a little. It looks better to me than the what I was getting before, but not quite as good as the 10 bit version.

[pandy]

IMHO you should use ordered dither as it less affect overall quality - error diffusion dither is more like noise so it may be or filtered or lead to unnecessary increase of quantization factor.

[ForSerious]

Are you saying in zscale, use d=1?
I tried that and the banding was still there. Like the exact locations and sizes of the bands changed, but visually they were still just as noticeable.

What I feel like I need is a replacement to the format filter that I can specify the dither method or amount or something.

[bettergenes]

Try any of these 3 commands and see what works better for you:

ffmpeg.exe -hide_banner -stats_period 2.0 -nostdin -framerate 23.976 -y -i "input\%6d.tif" -c:v libx265 -crf 18 -pix_fmt yuv420p -preset slow -profile:v main -vf "zscale=pin=bt709:min=gbr:tin=bt709:rin=pc:d=3:f=5:p=709:t=709:m=709:r=tv:c=input,noise=alls=1:allf=t,format=yuv420p" "output.mkv"

ffmpeg.exe -hide_banner -stats_period 2.0 -nostdin -framerate 23.976 -y -i "input\%6d.tif" -c:v libx265 -crf 18 -pix_fmt yuv420p -preset slow -profile:v main -vf "zscale=pin=bt709:min=gbr:tin=bt709:rin=pc:d=3:f=5:p=709:t=709:m=709:r=tv:c=input,noise=alls=1:allf=t,grain=1:2,format=yuv420p" "output.mkv"

ffmpeg.exe -hide_banner -stats_period 2.0 -nostdin -framerate 23.976 -y -i "input\%6d.tif" -c:v libx265 -crf 18 -pix_fmt yuv420p -preset slow -profile:v main -vf "zscale=pin=bt709:min=gbr:tin=bt709:rin=pc:d=3:f=5:p=709:t=709:m=709:r=tv:c=input,gradfun=radius=16:strength=2,format=yuv420p" "output.mkv"

[ForSerious]

The grad function is a good idea, but always adds blur to the image.
I tried a bunch of combinations with the noise filter. Seems a little backwards. I'm only using TVAI in this sample to get rid of noise, adding some back in defeats that though…

So what I came up with, chaining the format filter

-vf "zscale=pin=bt709:min=gbr:tin=1:rin=pc:d=3:f=5:p=709:t=709:m=709:r=tv:c=input,format=yuv420p12le,format=yuv420p10le,format=yuv420p"

[Attachment 84584 - Click to enlarge]
Somehow, I don't have the grain filter, but the noise one was fine. I decided to keep the chained formats in. I put the noise before the last one.

-vf "zscale=pin=bt709:min=gbr:tin=1:rin=pc:d=3:f=5:p=709:t=709:m=709:r=tv:c=input,format=yuv420p12le,format=yuv420p10le,noise=alls=1:allf=t,format=yuv420p"

[Attachment 84585 - Click to enlarge]

Those pictures don't really show any difference. The noise version looks sharper and I can't see any added noise. So it's doing what it needs to. I can still see the banding. It's reduced though.
I'm going to see what chaining the noise does…

[pandy]

You can try to split planes and apply controlled quantization to each and after all combine planes... ffmpeg can be tricky on this.

[bettergenes]

For experimenting with chained noise, you could try:

ffmpeg.exe -hide_banner -stats_period 2.0 -nostdin -framerate 23.976 -y -i "input\%6d.tif" -c:v libx265 -crf 18 -pix_fmt yuv420p -preset slow -profile:v main -vf "zscale=pin=bt709:min=gbr:tin=1:rin=pc:d=3:f=5:p=709:t=709:m=709:r=tv:c=input,format=yuv420p12le,noise=alls=0.5:allf=t,format=yuv420p10le,noise=alls=0.75:allf=t,format=yuv420p" "output.mkv"

You might want to experiment with different noise levels at each stage (the alls parameter). Using lower values earlier in the chain and slightly higher values later might give better results than equal values throughout.

[ForSerious]

You can try to split planes and apply controlled quantization to each and after all combine planes... ffmpeg can be tricky on this.

I don't even know how I would start to do this. Some of the ideas I have had get shot down right away because the filter requires 10 bit or more output.

You might want to experiment with different noise levels at each stage (the alls parameter).

Thanks. That is better than the same value in each stage. It's still coming out with some banding. I'll keep trying variations along these lines.

[pandy]

You can try to split planes and apply controlled quantization to each and after all combine planes... ffmpeg can be tricky on this.

I don't even know how I would start to do this. Some of the ideas I have had get shot down right away because the filter requires 10 bit or more output.

Filter 'extractplanes' https://ffmpeg.org/ffmpeg-filters.html#extractplanes
You can check also 'libplacebo' filter https://ffmpeg.org/ffmpeg-filters.html#libplacebo - it have debanding and dithering subfilters so perhaps it may help in your problem.

[ForSerious]

libplacebo took me full circle. It has all of the options needed to be a replacement for zscale and format. In the end I was able to get the same results as zscale and chaining format—not better. It might have had a slight color change.

After all that, I want to lay this thing to rest. I checked again how the final video looks on my TV setup. The banding is much less noticeable on that, then on the computer.

I'm settling with the chain format command. Zscale with d=1 makes a slight difference.

-vf "zscale=pin=bt709:min=gbr:tin=1:rin=pc:d=1:f=5:p=709:t=709:m=709:r=tv:c=input,format=yuv420p12le,format=yuv420p10le,format=yuv420p"

[rgr]

My goal is to use Topaz Video AI to upscale my DVDs to FHD and maybe denoise some Blu-rays.

Never use Topaz to denoise video. It just does it terribly.

[rgr]

I get color banding that's pretty easy to see in some scenes. If I set it to yuv420p10le, the banding goes away—but only on my computer since my TV does not support 10 bit. It displays the color banding.

In my experience, no matter how hard you try, compression will eventually destroy even the best dithering and artifacts will appear. You might want to consider encoding to 10 bits and streaming to TV using software that converts to 8 bits on the fly. I don't know Plex, but Universal Media Server has that option.

[jagabo]

Add noise in ffmpeg:

-vf "noise=all_strength=1.0,format=yuv420p"

You may have to play around with strength and flags.

[ForSerious]

Never use Topaz to denoise video. It just does it terribly.

There are sooo many ways to denoise with TVAI. I doubt you have found and evaluated them all. Anyway, that's not helpful since I already found what model and settings to use and love the results.

Add noise in ffmpeg:

-vf "noise=all_strength=1.0,format=yuv420p"

You may have to play around with strength and flags.

I tried incrementing the strength. It never started reducing the banding before it started adding noise back in.

[jagabo]

Add noise in ffmpeg:

-vf "noise=all_strength=1.0,format=yuv420p"

You may have to play around with strength and flags.

I tried incrementing the strength. It never started reducing the banding before it started adding noise back in.

Of course. You can't get perfectly smooth gradients with 8 bit YUV. Limited range 8 bit YUV only has about 1/6 as many different colors as 8 bit RGB, full range YUV about 1/4 as many. Even with 8 bit RGB you can see light banding without noise.

Oh, and make sure you add noise before reducing the bit depth.

[rgr]

Never use Topaz to denoise video. It just does it terribly.

There are sooo many ways to denoise with TVAI. I doubt you have found and evaluated them all.

I don't think there's more than one way to denoise in Topaz (not to confuse it with image enhancement models), and it's one of the worst there is.

Anyway, that's not helpful since I already found what model and settings to use and love the results.

I once thought so too
https://imgsli.com/MjkwNDU3

[ForSerious]

I agree that TVAI doesn't really help VHS sources. The best way to denoise DVDs in TVAI is the Proteus 4 model. Manual mode, only increase the Fix Compression. I suppose it probably does a little more than just denoise, but I've usually been able to find values for it that look the same or better than the original. In all cases, I'm able to get better looking results with TVAI than with any of the many denoise filters in ffmpeg. Point being, would anyone have guessed that the best denoiser is the parameter named 'Fix Compression'?
Not sure why I'm defending TVAI. It's a hot mess right now, and they only keep making it worse. I'm not affected by most of the issues because I only use the CLI, but no one should be buying it right now, unless they already have working solutions for it's many shortcomings.

Oh, and make sure you add noise before reducing the bit depth.

I know I was doing it that way at one point, but I might have wondered into doing it after. The fact that I don't remember exactly is a good reason to give it another round of tests.

[pandy]

I think there is a way - perform color requantization (level reduction) in at least Y plane with help pr or ordered dither (such as Bayer) or pseudonoise (Ulichney) - it will be static in temporal and spatial domain - ordered dithering may be performed in ffmpeg indirectly with help of https://ffmpeg.org/ffmpeg-filters.html#paletteuse filter and or static palette (like 64 grayscale) or dynamically generated one thus using https://ffmpeg.org/ffmpeg-filters.html#paletteuse .
This can be tricky - separate your source on Y, Cb, Cr planes, apply requantization ( IMHO 64 colors should be sufficient to prevent banding ) , merge Y, Cb, Cr - encode.

Static dither should survive encoding process (ideally dithering shall be applied at frequency domain in encoder in wise way - without introducing too much entropy - afraid since MPEG-2 no one even think about such things).

[rgr]

I'm able to get better looking results with TVAI than with any of the many denoise filters in ffmpeg.

Well, denoise filters in ffmpeg are not a big challenge