Filtering help needed

[x264]

What are the things that needs to be fixed in this source?

Sample :

http://www.mediafire.com/?y9vqkd98gku7hrs

http://www.mediafire.com/?40n08brccnvqvkb

AVS Script

MPEG2Source("C:\Users\x265\Desktop\Death note vol1.d2v", cpu=0)
AnimeIVTC(mode=1, aa=0, ifade=true)
trim(1918, 30688)+trim(32368, 32727)
Crop(2, 2, -2, -0)
SMDegrain(tr=6, pel=4, refinemotion=true, subpixel=3, lsb=true, prefilter=3, mode=6)
LSFmod(defaults="slow", strength=40)

[x264]

how to i join the video in this order? :
OP+Episode+ED+Preview

[Selur]

What are the things that needs to be fixed in this source?

If you don't know what's bothering you with the source, then there's no need to 'fix' anything.
It's like asking: "Is green a nice colour?"

how to i join the video in this order?

a. Why did you split them to begin with?
b. Do you want to join already finished files or do you want to join them inside your Avisynth script?
Joining finished files depends on the container used.
Joining stuff in Avisynth normally happens through AlignedSplice or UnalignedSplice (http://www.afterdawn.com/guides/archive/using_avisynth_page_7.cfm) depending on what you want,...

[x264]

The source looks too soft and contains haloing, also the colours are dull. could you have a look at the sample?

[x264]

I want to add in the creditless OP and ED. Is the code below correct?

ClipTwo=trim(1918,30688)
ClipFour=trim(32368,32727)
ClipThree=MPEG2Source("C:\Users\Abhijith Nair\Desktop\ED.d2v", cpu=0).AnimeIVTC(mode=1, aa=0, ifade=true)
ClipOne=MPEG2Source("C:\Users\Abhijith Nair\Desktop\OP.d2v", cpu=0).AnimeIVTC(mode=1, aa=0, ifade=true)
AlignedSplice(ClipOne, ClipTwo, ClipThree, ClipFour)

[sanlyn]

There is no heavy grain in either of your sample clips. There is visible color banding, most evident in your 01.demuxed.m2v sample. Running degrainers is the very thing you don't want: it exposes more banding, and then LSFMod sharpens its edges. Banding and noise removal have a long thread with the dither() plugin on Doom9: "Color banding and noise removal" (http://forum.doom9.org/showthread.php?p=1386559). The same scene that's in your first sample is discussed in post #8 in that thread with a script and suggestions (running GradFun2DBmod is one suggestion) showing various ways to defeat the banding: http://forum.doom9.org/showpost.php?p=1390594&postcount=8 .

[x264]

What about high bitdepth denoising(SMDegrain(lsb=true)), won't that prevent the banding?. Also LSFmod has a preblur option which will prevent noise sharpening.

MPEG2Source("C:\Users\Abhijith Nair\Desktop\Death note vol1.d2v", cpu=0)
AnimeIVTC(mode=1, ifade=true, aa=0)
ClipTwo=trim(1918,30688)
ClipFour=trim(32368,32727)
ClipThree=MPEG2Source("C:\Users\Abhijith Nair\Desktop\ED.d2v", cpu=0).AnimeIVTC(mode=1, aa=0, ifade=true)
ClipOne=MPEG2Source("C:\Users\Abhijith Nair\Desktop\OP.d2v", cpu=0).AnimeIVTC(mode=1, aa=0, ifade=true)
AlignedSplice(ClipOne, ClipTwo, ClipThree, ClipFour)
Crop(2, 4, -2, -0)
LSFmod(defaults="slow", strength=50 , preblur="SMDegrain(tr=6, thSAD=300, mode=6, pel=4, subpixel=3, lsb=true, search=5)")

[x264]

Should i use the Gradfun3() filter to remove the banding which is already present in the source?

[Selur]

"denoising(SMDegrain(lsb=true)), won't that prevent the banding?"
denoising will cause additional banding

you could:

1. denoise, which will cause additional banding and destroy details, in example the structure on the 'death note' manuscript or the table
2. deband, which will add additional noise due to the dithering

to find a good balance it might be a good idea to use something like AvsPmod and play with the settings,..

Should i use the Gradfun3() filter to remove the banding which is already present in the source?

I would probably recommend something more flexible like flash3kyuu_deband,...

Cu Selur

Ps.: LOL, I just realized how small my samples turned out,... -> added two encodes with 1500kBit/s (still not really happy with the result, but I rarely encode any animes, so I'm not really used to the material)

[x264]

When should i use flash3kyu_dither?

[Selur]

as one of the last filters,...

[x264]

should i use it like this ?
....
.....
Flash3kyuu_deband(range=18, output_depth=16)
f3kdb_dither()

The filesize of the filtered frame is big compared to the original unfiltered frame? why is this happening?

[Selur]

should i use it like this ?

Personally I wouldn't:
a. use output_depth=16
b. use f3kdb_dither
but if you are into 16bit and range=18 is to your liking: yes

The filesize of the filtered frame is big compared to the original unfiltered frame? why is this happening?

dithering = adding noise = adding details -> preserving details requires more data rate
=> read up on the filters you use before using them, simply copying stuff from other scripts, without understanding what you do, isn't going to do you any good (especially if you work with filters that should (need to) be adjusted on a per clip basis)

Cu Selur

[poisondeathray]

The filesize of the filtered frame is big compared to the original unfiltered frame? why is this happening?

Because all dithering filters essentially add "noise" . In order to retain the dither, you need high bitrates and grain retetion settings, or the banding will come back . 10bit helps too

[x264]

how do i downsample 16bit video to 10bit, f3kdb_dither dithers to 8 bit. Should i pipe the 16bit video into x264 using avsyuv?

[poisondeathray]

how do i downsample 16bit video to 10bit, f3kdb_dither dithers to 8 bit. Should i pipe the 16bit video into x264 using avsyuv?

There are some examples in the dither documentation and the dither thread .

Or another method is to use vapoursynth .

[x264]

What's the use of debanding if the noise removed by the denoiser is restored?

[poisondeathray]

What's the use of debanding if the noise removed by the denoiser is restored?

What kind of "debanding" are you referring to specifically ?

If you mean some type of dithering, it's a different pattern of "noise" . It essentially "covers" up the banding . There are different types of dithering as well, ordered, non ordered, many different algorithms

As explained above - denoising removes noise (obviously), but will also remove grain, dithering patterns . This predisposes you to banding along gradients, especially with 8bit YUV, and especially with anime

[x264]

Ok, so what would you do with the sample above?

[poisondeathray]

Personally, I wouldn't do anything and live with it. Not worth the trouble in my opinion on this source. The filesizes you will require to improve the banding in the 1st clip will be enormous , even with 10bit AVC.

This type of animation is supposed to be diffuse, blurry, not oversharpened (it's the style of the animation)

[x264]

Ok, I have decided not to use a denoiser but i want to remove the banding which is already present, filesize doesn't matter.

[poisondeathray]

Then add some grain or dither filter and encode with --crf 0 . It's going to be many times larger than the original

[x264]

can you post the code for loading high bitdepth avs into avs script into x264tmod?

[poisondeathray]

can you post the code for loading high bitdepth avs into avs script into x264tmod?

Just copy & paste from the instructions , change the resolution and fps to match yours


eg.

avs2yuv -raw "script.avs" -o - | x264-10bit --demuxer raw --input-depth 16 --input-res 720x480 --fps 24000/1001 --output "output.mp4" -

[x264]

http://forum.doom9.org/showpost.php?p=1536519&postcount=157

The author of flash3kyuu_deband says its better to dither to 10bit is better than directly feed 16bit to x264. I don't know how to load the high bitdepth avs into x264tmod, could you show me how it's done?

[poisondeathray]

You "load" it with the command line above, plus whatever settings you want to encode with

I've used tmod, but I've never used his hacked version before (link is dead) . http://j.mp/x264tMod

You can do it with dither to 10-bit data aligned on 16-bit words using the dither package ; you still use --input-depth 16 in the commandline


To be clear, instead of
8bit source => fake 16bit + fake 16 bit filters => 16bit => 10bit x264

This does
8bit source => fake 16bit + fake 16 bit filters => dither to 10bit => 10bit x264

.
.
.
Dither_quantize (10, mode=6) 
Dither_out ()

You should really READ the instructions and look at the examples, that's what they are there for


I don't know how to use flash3kyuu for this, I've only used it for 8bit . But look at the instructions, you can probably specify the input/output depth there .

input_mode
Specify source video type.
0: Regular 8 bit video
1: 9 ~ 16 bit high bit-depth video, stacked format
2: 9 ~ 16 bit high bit-depth video, interleaved format

Default: 0 (input_depth = 8 or not specified) / 1 (input_depth > 8)

input_depth
Specify bit-depth of source video.

Range: 8 ~ 16
Default: 8 (input_mode = 0 or not specified) / 16 (input_mode = 1 or 2)

output_mode
Specify output video type. Meaning of values are the same as input_mode.

Only valid when dither_algo = 1 / 2 / 3 .

If dither_algo = 0, output_mode is set to 0 and can't be changed.

When output_mode = 2, frames will be 2x wider and look garbled on preview,
it will return to normal after correctly encoded by high bit-depth x264)

Default: 0 (output_depth = 8 or not specified) / 1 (output_depth > 8)

output_depth
Specify output bit-depth.

Only valid when dither_algo = 1 / 2 / 3 .

If output_depth = 16, dither algorithm specified by dither_algo won't be
applied.

Range: 8 ~ 16
Default: 8 (output_mode = 0 or not specified) / 16 (output_mode = 1 or 2)

[x264]

Ok, i tried the code which you posted above, now i'm this getting this error


AVS Script

MPEG2Source("C:\Users\Abhijith Nair\Desktop\Death note vol1.d2v", cpu=0)
AnimeIVTC(mode=1, ifade=true, aa=0)
ClipTwo=trim(1918,30688)
ClipFour=trim(32368,32727)
ClipThree=MPEG2Source("C:\Users\Abhijith Nair\Desktop\ED.d2v", cpu=0).AnimeIVTC(mode=1, aa=0, ifade=true)
ClipOne=MPEG2Source("C:\Users\Abhijith Nair\Desktop\OP.d2v", cpu=0).AnimeIVTC(mode=1, aa=0, ifade=true)
AlignedSplice(ClipOne, ClipTwo, ClipThree, ClipFour)
Crop(2, 4, -2, -0)
Dehalo_alpha_mt(ss=3.0)
SMDegrain(lsb=true, tr=6, pel=4, subpixel=3, refinemotion=true)
lsfmod(defaults="slow", strength=50)
flash3kyuu_deband(range=17, output_depth=16, dither_algo=3, input_depth=8)

[poisondeathray]

Ok, i tried the code which you posted above, now i'm this getting this error

it's --no-fast-pskip

Not

--no-fast-pskip true

You're also missing a " - " at the very end, required for piping . Scroll back up and look at the example command line more closely




I am almost certain you will get banding with those settings, it might be slightly reduced, but it won't be eliminated. The problem is the source

[x264]

Well let me see how it turns out. Thanks for your help

[poisondeathray]

You can also try vapoursynth's fmtc for various dither operations . It works natively at 16bit, instead of "fake" 16bit .

It's like learning avisynth all over again (at least for me) , but my preliminary tests show it's promising, slightly better results than the native 8bit (or fake 16bit) filters in avisynth on various test gradients and CG renders

But I doubt anything will help much with the banding in the 1st sample; the problem is the source. You need very high amounts of dithering noise and enormous bitrates to retain that