Filtering help needed

[poisondeathray]

Also, how did you get 716x952 resolution ?? or nevermind - this is stacked fake 16 bit

[x264]

if it's hard to keep of the banding then how do most encoder's fix these kind of sources?

[x264]

Error

[poisondeathray]

if it's hard to keep of the banding then how do most encoder's fix these kind of sources?

You don't.

The amount and type of banding isn't normally as bad as this in other sources . There are different types and "causes" of banding. This one predisposes you because of the way the gradients are rendered in the animation

You can improve it a little, that's about it. By the time you encode it, the banding will be much worse unless you use enormous amounts of bitrate and crazy grain retention settings - not worth it

Error

Not sure what the error is (request linear: internal error (frame not cached) ) , I've never seen it before, but probably a problem with your script. Can you preview it in vdub or avspmod ? Or play it in a media player ?

[x264]

What kind of tweaks within x264 will prevent the banding from coming back?, does --tune grain help to retain the dither?

[x264]

Not sure what the error is (request linear: internal error (frame not cached) ) , I've never seen it before, but probably a problem with your script. Can you preview it in vdub or avspmod ? Or play it in a media player ?

Preview in avspmod is fine. I don't know what's going wrong when i set it up for encoding.

[poisondeathray]

What kind of tweaks within x264 will prevent the banding from coming back?, does --tune grain help to retain the dither?

Yes it does , but the default aq strength is 0.5, and probably too low for dark scenes like your 1st clip.

Basically anything that helps retain grain , noise and fine details. Bitrate is the most important

lowering crf, increasing aq, increasing qcomp, increasing psy-rd and psy-trellis, lowering deadzones, no-dct-decimate, negative deblock values

There is also a grain optimization patch available for x264 (not sure if tmod has it)


Not sure about your error, might be a problem with smdegrain (it no longer supports fake high bit depth) , unless you are using an older version . You can ask at doom9

[x264]

It is working now after i installed the NVIDIA drivers. I will post the results after the encoding process completes.

[poisondeathray]

It is working now after i installed the NVIDIA drivers. I will post the results after the encoding process completes.

Weird, not sure why that would affect it . Nothing in your script or encoding uses Nvidia .

I see you fixed the f3kdb settings
http://forum.doom9.org/showthread.php?p=1624839#post1624839

[x264]

I have never made any changes in x264 settings besides the presets and tune. Could you post the x264 settings which you would use for sample1? many of threads on x264 mentions that tweaking the defaults will do more harm than good, so i've never bothered to tweak them.

[poisondeathray]

I already told you , personally I wouldn't bother with this or use --crf 0

You have to experiment with the settings to see what quality loss or degree of banding YOU will accept . Everybody is different in what they will tolerate. How a particular script or series of filters reacts with some settings only comes with experimenting and learning .

Yes, as a beginner it's a good idea to stick with the presets, but in order to learn you have to do many tests and examine the results . The settings I mentioned above definitely affect grain retention, but at a cost of enormous bitrate required - simply put - grain and retaining dither is extreme expensive .

I can tell you from experience , the conditions in the 1st test clip are horrendous and predispose you to banding. You will require bitrates that make the clip many times larger than the original to make it close to perfect. If you tolerate some banding and defects you can reduce that amount, but it's subjective. Again, you have to do these tests to see what degree of banding and problems YOU will accept. Some people don't even notice these problems and defects (then that's great, use low bitrates, normal settings). The more you learn about video & encoding, the more you tend to focus on , and see, problems instead of enjoying the content....sad but true

When other sections aren't as bad, or the conditions are different, you can apply different filters to different sections. You can even apply some different encoding settings to different sections using x264 zones. This way, you can reduce the bitrate requirements. This is how encoders tweak their encodes and encode more effectively.

Hint: If you want to test the effect of various encoding samples on your script , encode a lossless intermediate, then use that as input to x264 , doing a series of encodes at various settings. This way you don't have to incur those filters as a bottleneck . Right now 0.02-0.03 fps doesn't look to appealing . Then perform a series of tests with a set of encoding settings and varying the script . Do these tests on a smaller sample, not a whole episode. This is the only way you learn how the settings react to various inputs

[x264]

I'm getting an error which says "FFVideosource : Short read while reading frame" when i tried loading the encoded mkv into ffvidesource.

ffvideosource("C:\Users\Abhijith Nair\ep.mkv", colorspace="YV12_10-bit_hack")

[poisondeathray]

can you open it normally ?

The ffms2 hack is for the stacked fake 16bit input , not normal encodes

ffvideosource("C:\Users\Abhijith Nair\ep.mkv")

[vhelp]

exactly as pdr says.. plus, put together a group of "tipicle" scenes, i.e, dark; light; dark/light; low_motion; high_motion; etc etc and use those as your standard guages when testing x264 encoding strategies for quality against the videos that will be encoded. later, when you develop the eye, you will be able to tell which encoding params will work best for each source.

pdr, is there a way to save "new" presets (cmd line params) to call up instead of retyping or copy/pasting ? i do a lot flip/flopping between encode qualitys, i.e., LQ vs HQ, and so on for instance. it would be easier to make future adjustments to preset files as they occur.

i encode mainly with:

x264 --preset veryfast, x1, x2, x3, ... video.mkv

but would like to get it to this path if possible:

x264 LQ in_video out_video.mkv
x264 HQ in_video out_video.mkv

thank you.

[poisondeathray]

pdr, is there a way to save "new" presets (cmd line params) to call up instead of retyping or copy/pasting ? i do a lot flip/flopping between encode qualitys, i.e., LQ vs HQ, and so on for instance. it would be

easier to make future adjustments to preset files as they occur.

LOL personally I use, highlight, copy, paste is not so difficult IMO . Just keep a text file around of your commonly used settings

Other people use send to with a batch file from the right click menu. e.g jagabo does this
https://forum.videohelp.com/threads/341663-How-to-Use-x264-Command-Line?p=2127376&viewf...=1#post2127376

So if you wanted to do both encodes in 1 click, just make your batch file with both . Or if you had a bunch of clips to encode , just use a batch file

[x264]

I would like to compare the encoded frames and the original DVD frames.

[poisondeathray]

I would like to compare the encoded frames and the original DVD frames.

There are many methods people use to do this , but since you cropped/resized the encode , it's harder to compare . It helps if they are the same dimensions and same framecount (e.g. if you test a clip, trim() it to the same number of frames)

1) interleave(a,b) . This will alternate frames back and forth as you advance . Sometimes it helps to label them (e.g. subtitle("a") )

2) stackhorizontal(a,b) or stackvertical(a,b) , either go frame by frame or use a media player

3) difference testing using subtract() , or use amplified differences

4) switch back & forth using tabs in avspmod (number keys)

.
.
Sometimes differences can only be seen in motion (e.g. when played, important for temporal characteristics) , other times you might need to go frame by frame and/or even zoom in

You can use PSNR/SSIM testing - objective measures and metircs

Another technique is to use histogram("luma"), often you can pick up subtle banding and defects

I'm sure there are many more techniques and methods, just naming a few

[vhelp]

pda--sorry, my connection is frustratingly slow and timing out while trying to post a new topic instead. you reponded too quickly

[x264]

Should i increase aq to something like 0.6 or 0.7?, one of the members of an anime forum told me that i should not raise the qcomp value above 0.8 for animation.

[poisondeathray]

Should i increase aq to something like 0.6 or 0.7?, one of the members of an anime forum told me that i should not raise the qcomp value above 0.8 for animation.

You can do whatever you want. I can't answer that question. Simple answer is do some tests and find out what works for YOU .

In general, I wouldn't raise qcomp above 0.8 , but aq strength is going to change depending on the content , filters, specific situation. You might even try a different aq mode like mode 2, or try different aq like haali aq, ore aq, or mix aq . They work better in some situations

In general, you wouldn't use high psy-rd or psy-trellis settings, or high aq settings in an animation setting either (they tend to ruin edges) , but in a high banding situation, you have to make compromises

[x264]

Ok, can you explain what aq's purpose is? what does it do?.

[poisondeathray]

can you explain what aq's purpose is?

It redistributes bits to flat gradients and dark areas . At a given bitrate, more bits will be allocated to those areas. At a given CRF, the bitrate will usually balloon up



Also, many people don't bother with debanding during encoding, instead use debanding filters during playback. This way you can have reasonably sized files . Gradfun2db was originally a playback filter. Filters in the gradfun family , f3kdb are usually fast enough for playback. Many software players have debanding filters as an option or you can do it through ffdshow

[x264]

Is it necessary to use a prefilter before using MC denoising?

[poisondeathray]

Is it necessary to use a prefilter before using MC denoising?

No, but in some cases it might be beneficial to use a denoised clip as the prefiltered clip to run the filters on . SMDegrain has a prefilter option built in

[x264]

Encoded sample, dark scene.
settings used x264:
--qcomp 0.7 --aq-strength 0.7 --preset veryslow --tune grain.
http://www.mediafire.com/?8d5w768m63r0fad

It looks better than what i was expecting.

[x264]

Left: Filtered | Right: Filtered+Encoded, screenshot taken by using this script http://pastebin.com/aL8NKMpB

[poisondeathray]

Yes it's definitely improved significantly .

The banding I'm talking about is most noticable in the armpit frames and on the ceiling frames. If you can't see it, don't worry about it

[x264]

SMdegrain is killing the details in the dark scenes
http://screenshotcomparison.com/comparison/19807

[poisondeathray]

SMdegrain is killing the details in the dark scenes
http://screenshotcomparison.com/comparison/19807

http://forum.doom9.org/showthread.php?p=1624990#post1624990

[sanlyn]

Yes it's definitely improved significantly .

The banding I'm talking about is most noticable in the armpit frames and on the ceiling frames. If you can't see it, don't worry about it

Very familiar. This video is discussed several times on Doom9. The banding is more obvious when it's played. Sometimes a still shot doesn't show it so well. I don't know why.