Debanding help

[killerteengohan]

I rarely need to use debanding, but when I do, I simply use the built in debanding from McTemporalDenoise.
mctemporaldenoise(sigma=1, sharp=false, radius=1, ecrad=1, AGstr=0.5, GFthr=1.01) # GFthr controls the Gradfun2db threshold

It usually does a pretty good job and doesn't need to add lots of thick grain over the video, or somehow does it and cleans it to where I would not notice. There's a problem with this though. While it does a nice job, and keeps the image clean, it seems to smear away things in really dark areas or backgrounds.

You can see what I am referring to here. Just look at the background on the right.
https://slow.pics/c/T6TKjVOb

(Same but with stronger GFthr setting)
https://slow.pics/c/stHDn6YP

If I turn GFThr off and make it 1.0, the background details are unaffected and are fine, so I figured I would try messing with the underlying filters it uses that were built in for more customization to protect the details.


I tried using FFT3D like McTemporalDenoise does by going through the readme file and mimicking settings I saw, but that filter just makes it a banded mess and it looks TONS worse and makes more banding than before when I tried it. Here is a screenshot of it.
https://forum.videohelp.com/images/imgfiles/LziFshC.png
I tried using GradFun2db on top of it but I just cant get it to work as good as when it was built in with mctemporaldenoise.


While its usually pretty good at what it does, I don't think its doing well enough here with this source, and I am not liking all of that background detail smear when using GFthr in mctemporaldenoise either. Can someone help me with the banding in this? While mctemporaldenoise GFtr is reducing visiblity, its not quite enough for me. I would love it if I can keep that background detail in tact, and not have to add grain that just sits there in a still pattern over the video and never moves. Since dithering is indeed adding grain, it would be nice if it blended in better or moved around some so it looked natural.

I have very little experience with dithering outside of mctemporaldenoises built in debanding options. Is that degradation of background details a side effect I am going to have to deal with when dithering? Kind of like how denoise removes detail if you use it.
Is this a trade off I will be stuck with, or do I just need better filters and settings?

I am guessing someone with more debanding experience, and competent with masking if needed might know how to do this much better than I do. Maybe even overlays would be the key here, I'm not 100% sure.


Here is a video sample
https://mega.nz/file/osIl2CRQ#H3PczpahKcKBWWWEABWt3iphQvNrbPBv-W2dc_MlESs

Here is one with a much brighter scene
https://mega.nz/file/U1RDiYSD#17JdyJ-ehqAxkT0BqeA6Mzb_VQKvHhgfTdWGJz_An3Q

I don't see it all over the place, and it doesn't look very visible in most scenes, but there is the occasional scene like this one in episodes, where its just awful on the characters skin and faces and you cant miss it. It seems like it just sits there over top of the image, while the characters move around underneath it.

[killerteengohan]

I noticed the FFT3D that made it look awful to me, seems better when I tried to use GradFun2db on top of. FFT3D did give the entire image a smeared look before I tried adding GradFun2db.

If its doing those two things to the video in mctemporaldenoise, then I think mctemporaldenoise is using the FFT3D first and then GradFun2db afterwards, and that FF3TD might be whats smearing the background details out like that.

[jagabo]

You always deband after noise reduction. The reason you have to add noise back to the picture is because 8 bit YUV isn't sufficient for smooth gradients.

Look at the 8 bit RGB cube inside the 8 bit YUV (YCbCr) cube:



All ~16 million RGB colors are reduced to about 2.6 million different YUV colors. So the steps between adjacent colors are much larger. Adding noise helps hide those steps.

[Cornucopia]

To add to that, it's a best practice in terms of "fidelity to naturalness in perception" to ALWAYS dither whenever reducing bit depth (just prior to). This includes post-calculation bitdepth reductions as well. The added noise is minor overall (even when compounded during multiple iterations), and yet it does much to reduce quantization error and the effects of correlated distortion (of which, banding is one kind).
Works for audio, photo & video.

Scott

[killerteengohan]

I know you deband after noise reduction. Denoising can cause banding. I know basic reasons for its cause, I just don't know many ways to fix it and rarely ever needed or wanted to.

There's no better way to deband this? I cant get around the smearing in the image examples I shared?

The debanding I do use from mctemporaldenoise, is not working well enough on this one. Not only is it doing that smearing to the background and removing dark area detail as seen in the comparison images, but the banding looks pretty bad and its not doing a well enough job to fix it like it usually does for me.

I would like to be able to fix that heavy banding on the characters faces as can be seen in the video examples. I don't know what else to use other than than the debanding built in mctemporaldenoise and its not really cutting it for this source.

What filters should I dither with? I tried the fft3d gradfun2db combo, but its not getting the banding on the faces off, and is tearing up that background like in the image comparison examples. I tried using f3kdb(range=5, grainy=32, grainc=32). It doesn't smear or destroy the background nearly as much as the other filters i tried, but it also just barely covers or does any kind of fix to the banding.

Maybe I just don't what settings to use?, maybe there's something better to use?.

What would you do to it? Can I keep the background and dark area lines/details from smearing out or is that a side effect that is unavoidable if I want to fix the banding in this source?

[jagabo]

I don't know any better ways to deband. But if you only need certain colors/intensities debanded you might be able to use a mask generated with MaskHS(), ColorKeyMask(), etc.

[killerteengohan]

I spent hours toying with settings, and while its not perfect, it does about as good of deband as mcdenoise's deband settings, but it also does a LOT less damage to details in darker areas and backgrounds.

f3kdb(range=3, grainy=32, grainc=32, y=56, cb=56, cr=56)
mctemporaldenoise(sigma=1, sharp=false, radius=1, ecrad=1, AGstr=0.5, GFthr=1.00) #Deband is turned off
LimitedSharpenFaster(ss_x=1.0, ss_y=1.00, strength=22, overshoot=0, undershoot=0, soft=0, edgemode=0)

I still cant get rid of it all, but this is better than nothing, better than mcedenoise deband, and I got to keep the majority of the darker area and background details.

I guess I just wont be able to get it all without going overboard on settings and smearing things away so I had to choose a medium area between the two.

Thanks for the idea on things to look into.

[jagabo]

The way I see it: you will always lose some detail when you deband. There's no way for a filter to know for certain when a step from one value to another a gradient or an actual detail.

[killerteengohan]

I fixed the script. I forgot a line in it.

Yeah, I have to lose some detail, but the amount of loss with f3kdb set up like so, is much much less loss than what I was getting with the debanding built in mctemporaldenoise, even at its lowest possible deband strength setting.

(mcdenoise deband vs f3kdb)
https://slow.pics/c/koMU66wx
# The background and dark area details are much less smeared away.
# Not quite as good of banding removal as mcdenoise buit in, but kind of close. The detail retention is worth it to me.

(source vs f3kdb)
https://slow.pics/c/1sflU58Y
# Not a huge amount of deband strength, but certainly better than the source, and I get to keep more detail.


In this much higher detail shot you can see how it took away less from that really dark background where the shelves are. It also blurred away less black lines from the top of the gun being held up.

(mcdenoise deband vs f3kdb)
https://slow.pics/c/zaFgkV92
# Some parts of the image, like the wood for example may look slightly little bit softer, but overall I think it did a better job at keeping things in tact.

(source vs f3kdb)
https://slow.pics/c/GFHKobyl

[jagabo]

The problem with MCTC may be related to its 3d noise reduction, not its debanding.

[killerteengohan]

I'm not sure its the noise reduction. If I disable the debanding but keep on the denoising by setting mctemporaldenoise GFthr to 1.0, all of that detail is unharmed and no debanding is done to it. It only happens when I turn on the use of gradfun2db in it.

GFthr=1.0 (deband off) -vs- GFthr=1.01 (deband on)
https://slow.pics/c/9nm16tpD