Need help with debanding

[Reiden]

Hello, dear Community.

Since I had problems with massive Banding in my video, I was looking for a way to fix this issue. (I'm encoding with Avisnyth/MeGUI)
I found a DDL plugin named "gradfun2db.dll", which should remove the banding from my video, so i downloaded it and moved it to my "megui\tools\avisynth_plugin" folder.
In my script I used the following code in order to use the plugin:

LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\gradfun2db.dll")
gradfun2db()

I saved my script and checked the previews (DirectShowSource) and it worked; the banding was reduced extremly.
So I continued and encoded the video via MeGUI, but now here comes my problem.
The encoded file didn't look like the preview and the banding wasn't reduced at all.

Did I do anything wrong or do I have to use another plugin?

[poisondeathray]

Probably not enough bitrate.

Post your encode settings, and details on the source (dimensions, fps). What kind of content? anime ?

You can try dither's gradfun3 or flash3kyuu_deband

[epsilonalpha]

+1 with not enough bitrate, if you don't mind losing compatibility with anything but pc playback, try using 10 bit x264, it would help preserve debanded video tremendously.

[Reiden]

I'm pretty new to the whole encoding thing, so I hope these are the settings you asked for:

Encoder Settings: x264: *scratchpad*
File Format: MP4
Sound: Nero AAC: NDAAC-LC-MultiChannel-HQ-256Kbps
The only thing I changed at this encoding preset was the "M.E. Alogorithm" (now Multi Hex).

The RAW is an Anime RAW with 23,976 fps, 1280x720 and 5628 kBit/s.


I tried to use GradFun3 and flash3kyuu but I failed...
I already googled and read the dither post several times, but I think I'm missing some files.
If I use this code:

LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\mvtools2.dll")
LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\dfttest.dll")
LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\dither.dll")
LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\DGDecode.dll")
LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\gradfun2db.dll")
LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\mt_masktools-26.dll")
LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\MaskTools.dll")
LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\ColorMatrix.dll" )
LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\TIVTC.dll")
LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\UnDot.dll")
LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\AddGrainC.dll")
LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\RemoveGrain.dll" )
Import("C:\Program Files (x86)\megui\tools\avisynth_plugin\mt_xxpand_multi. avsi")
Import("C:\Program Files (x86)\megui\tools\avisynth_plugin\GradFun2DBmod.av si")
GradFun3 ()

it says "There is no function named "GradFun3" and if I'm trying to use flash3kyuu it says "The script's return was not a video clip".

Edit: Now I added the line "Import("C:\Program Files (x86)\megui\tools\avisynth_plugin\dither.avsi") " and no MeGUI is crashing with the error "RemoveGrain invalid mode: 22"


Edit2: Now I tried to encode with gradfun2db with a higher filesize and bitrate. Now the debanding works but the filesize sucks extremly... I don't want to increase the filesize of the release by more than 700MB.
There is another release from another group and they managed to remove the banding but their file has only >1000 Bitrate while mine has about 2000 but still has banding ._.

[jagabo]

To retain more noise use --qcomp 100 in your x264 command line. The default is 60. Use values in between for... in between results. I recommend this for CRF encoding, not bitrate encoding.

[poisondeathray]

gradfun2dbmod uses noisier dithering, but gradfun3 uses ordered dithering (better compressiblity because less fluctionations between frames), so gradfun3 usually requires smaller filesize

as mentioned above, encoding with 10bit x264 will also give you better quality at smaller filesizes, and less banding

to retain more dither use a grain preset, lower deadzones, lower deblock values, but effective dither requires more bitrate. Period.

You can find dither package and .dll's with documentation here
http://forum.doom9.org/showthread.php?p=1386559#post1386559

flash3kyuu_deband
http://forum.doom9.org/showthread.php?t=161411

You can see a bunch of examples , comparisons, and code in the threads

[Reiden]

Thank you for your answers.
I managed to make it work now. I don't know why MeGUI crashed but I had to remove 1 Line from dither.avs, but it seems to work though.
Encode is now slow as hell, but if debanding is working I've got no reason to complain.

[jagabo]

to retain more dither use a grain preset, lower deadzones, lower deblock values

qcomp=1 will have a much bigger effect than all of those.

I stumbled across this when playing around with different presets on a very noisy video. Ultrafast retained all the grain whereas none of the slower presets did (working at CRF 18). So I went through all the settings one by one until I figured out which was the cause. At CRF 18 the final bitrate of clean videos may go up 50 percent with qcomp=1.0. The bitrate of noisy videos may go up 300 percent or more.

[poisondeathray]

to retain more dither use a grain preset, lower deadzones, lower deblock values

qcomp=1 will have a much bigger effect than all of those.

I stumbled across this when playing around with different presets on a very noisy video. Ultrafast retained all the grain whereas none of the slower presets did (working at CRF 18). So I went through all the settings one by one until I figured out which was the cause. At CRF 18 the final bitrate of clean videos may go up 50 percent with qcomp=1.0. The bitrate of noisy videos may go up 300 percent or more.

Yes, the bottom line is bitrate, but raising qcomp to 1.0 usually produces undesirable effects with 2pass encoding such as the OP is doing . When qcomp approaches 1.0 , this means constant quantizer

The reason why qcomp is raised in the --tune grain preset to 0.8 is to help prevent visible quality fluctuations in I,P,B frame quality with grain retention on heavy coarse grain material (it's the same reason why IP, PB ratios are decreased to 1.1 and 1.1) . Higher qcomp lowers the effectiveness of mb-tree, which is not ideal for anime

But fine grain patterns and dither are slightly different than coarse film grain - especially ordered dithering with gradfun3 . The pattern is small and barely noticeable, fluctuations are less dramatic. This is where lowering deadzones and increasing psy-trellis really helps (--tune grain also lowers deadzones to 6,6 and psy-trellis to 0.25)