Filtering help needed

[jagabo]

The banding is more obvious when it's played. Sometimes a still shot doesn't show it so well. I don't know why.

Because your eyes are more sensitive to motion than detail.

[jagabo]

SMdegrain is killing the details in the dark scenes
http://screenshotcomparison.com/comparison/19807

When the amplitude of the noise approaches the signal level, or conversely, when the signal level drops to the level of the noise, the filter can't tell the difference between signal and noise.

[x264]

Very familiar. This video is discussed several times on Doom9. The banding is more obvious when it's played. Sometimes a still shot doesn't show it so well. I don't know why.

The one which was discussed on Doom9 is the R1 release. The DVD's i'm having is the R2J release which is much better compared to the R1.

[x264]

SMdegrain is killing the details in the dark scenes
http://screenshotcomparison.com/comparison/19807

When the amplitude of the noise approaches the signal level, or conversely, when the signal level drops to the level of the noise, the filter can't tell the difference between signal and noise.

I don't understand, can you explain in layman's terms?

[x264]

The IVTC'd video flickers when in motion, which filter should i use to fix this issue?

[poisondeathray]

The IVTC'd video flickers when in motion, which filter should i use to fix this issue?

Can you be more specific ?

What kind of flicker ? Luminance flicker ? aliasing flicker ? noise flicker?

After IVTC only , or other filters applied ?

[x264]

should i use psy-trellis for anime?

[poisondeathray]

should i use psy-trellis for anime?

You can do whatever you want . Try encoding a few samples at the same bitrate and vary the settings of psy-trellis

In general I tend to not use it, or very low amounts - it tends to make edges "noisy". But in a grain or dither retaining situation you usually have to make compromises

[jagabo]

SMdegrain is killing the details in the dark scenes
http://screenshotcomparison.com/comparison/19807

When the amplitude of the noise approaches the signal level, or conversely, when the signal level drops to the level of the noise, the filter can't tell the difference between signal and noise.

I don't understand, can you explain in layman's terms?

If there's more noise than picture removing the noise will remove the picture.

[x264]

How do i apply a mask to prevent the dark scene's from being oversmoothed? could any of you post the code for this process?

[jagabo]

See my post and poisondeathray's post immediately after:

https://forum.videohelp.com/threads/355377-Interlacing-problem?p=2235689&viewfull=1#post2235689

That allows you to filter differently based on brightness.

Or you can use Trim() or ApplyRange() to apply different filters to different ranges of frames.

[x264]

In MeGUI there is an option for setting quantizer matrice, what should i use for this option?

[poisondeathray]

In MeGUI there is an option for setting quantizer matrice, what should i use for this option?

These are for custom quantizer matrices (CQM's) . Use search if you want more info

People don't use this anymore, except in very specific situations. They've been largely surpassed by using AQ modes

[x264]

Could you name some of the best postprocessing filters?(to be used with TIVTC(clip2="postprocessingfilter")?, also should i set PP=0 when using an external PP filter?

[poisondeathray]

Could you name some of the best postprocessing filters?(to be used with TIVTC(clip2="postprocessingfilter")?, also should i set PP=0 when using an external PP filter?

PP=0 turns it off completely = no post processing . It's often used when tweaking the settings with display=true. You need to adjust the combing threshold for clip2

Have a look at the documentation and instructions for examples on how to tweak the settings

For clip2 , QTGMC.SelectEven() usually gives good results (at least better than the default, when combing is detected)

[x264]

I found this in the qtgmc thread

Although I know a number of people use it with success, QTGMC is not really designed for anime. QTGMC relies on identifying the motion from frame to frame. With a film source the frame rate is fairly high and movements between frames are usually small; the detail of film means good matches can often be found. Anime will run at a lower rate, may contain duplicate frames, have stylized/artificial movement, and be hard to track if there is a lack of fine detail. So motion matches on anime can be poor, which will lead to ghosting on the temporal steps of the algorithm (and possible blurring / halo strengthening). The same problems can apply to game captures to a lesser extent. --Vit

[poisondeathray]

I found this in the qtgmc thread

Although I know a number of people use it with success, QTGMC is not really designed for anime. QTGMC relies on identifying the motion from frame to frame. With a film source the frame rate is fairly high and movements between frames are usually small; the detail of film means good matches can often be found. Anime will run at a lower rate, may contain duplicate frames, have stylized/artificial movement, and be hard to track if there is a lack of fine detail. So motion matches on anime can be poor, which will lead to ghosting on the temporal steps of the algorithm (and possible blurring / halo strengthening). The same problems can apply to game captures to a lesser extent. --Vit

This is true, QTGMC isn't recommended for deblending either for the some of the same reasons like temporal ghosting issues - but it's still commonly used .

Nevertheless, it still produces better results than other filters with clip2 I would say most of the time, even on anime

Of course YMMV and it varies by source, so try some others out if it's not giving the results you want

[x264]

So when i set clip2=QTGMC(), the function will replace the combed frames with the deinterlaced frames from QTGMC. When using the clip2 parameter should i set PP=1(find/hint combed frames but don't deinterlace)?

[poisondeathray]

So when i set clip2=QTGMC(), the function will replace the combed frames with the deinterlaced frames from QTGMC. When using the clip2 parameter should i set PP=1?

QTGMC will return too many frames, you need selectodd or even

clip2 only works with higher PP modes, I can't remember which ones, it says in the instructions . But it works with default PP value for sure

It will replace only those which are detected as combed according to the threshold settings (this is why you need to tweak the settings using display=true)

[x264]

QTGMC will return too many frames, you need selectodd or even

The official doc. of qtgmc shows this > QTGMC().selecteven() < for same framerate.

It will replace only those which are detected as combed according to the threshold settings (this is why you need to tweak the settings using display=true)

Which all options do i need to tweak?

5 - motion-adaptive blend deinterlacing
6 - motion-adaptive cubic interpolation deinterlacing
7 - motion-adaptive modified-ela deinterlacing

Which of these modes should i use?

[poisondeathray]

Use the ones that work better on your specific source

[x264]

Use the ones that work better on your specific source

I don't know the difference between the 3 PP modes.

[poisondeathray]

Use the ones that work better on your specific source

I don't know the difference between the 3 PP modes.

I usually leave it at default (PP=6)

The modes 5-7 do replacement pixels using the mask where combing is detected, not the entire frame , but if you specify clip2, it doesn't matter because the method e.g. blend vs cubic etc... isn't used , instead clip2 is used

But when you don't use clip 2,

PP=5 Blend deinterlaing is bad. It makes everything blurry
PP=6 Cubic. this is almost the same thing as bob(). Leaves aliasing artifacts on orphaned fields, or combing areas
PP=7 Don't know never use it

clip2 -

When using PP > 1 this parameter may be used to specify a clip to take deinterlaced
frames from instead of using one of TFM's inbuilt deinterlacing methods. If PP < 5
then frames will be taken as is from clip2, if PP >= 5 (i.e. using motion adaptation)
then TFM will build the mask as usual and only pixels in moving areas will be taken
from the frames in clip2. Don't specify anything for clip2 to use TFM's internal
deinterlacing. For an example of using clip2 see the clip2_sample_avs.avs file.

[x264]

TFM(d2v="C:\Users\Abhijith Nair\Desktop\Death Note\OP\MainMovie\Desktop\VTS_01_1.d2v", slow=2, display=true, clip2=QTGMC(preset="placebo").selecteven( ) ).Tdecimate(mode=1)
this is what i came up with do you think any option needs to be changed?

You said that i tweak some settings using display=true, name those options?

[poisondeathray]

TFM(d2v="C:\Users\Abhijith Nair\Desktop\Death Note\OP\MainMovie\Desktop\VTS_01_1.d2v", slow=2, display=true, clip2=QTGMC(preset="placebo").selecteven( ) ).Tdecimate(mode=1)
this is what i came up with do you think any things needs to be changed?

display=true will hardcode the overlay if you leave it in there, you only use that for debugging and tweaking purposes while adjusting the combing threshold and other settings (remove it before encoding)

Then you use adjust the settings to deliver the results you want

[jagabo]

preset=placebo is probably a waste of time.

[x264]

Is it safe to use low cthresh values?

[poisondeathray]

It's not detected as combed with the settings you are using. Notice the overlay says "clean frame" . Display will say "combing detected, deinterlaced" or something like that when it applies the PP clip2 . So adjust the comb detection threshold settings until it is detected. Then go back over other sections to make sure there aren't false positives (you don't want to deinterlace "good" sections)

[x264]

and what if there are false-positives? how do i prevent qtgmc from deinterlacing the good frames?. I had to lower the cthresh value to 2 so that a combed frame would be deinterlaced, but now the previous good frames are also being deinterlaced.

[poisondeathray]

and what if there are false-positives? how do i prevent qtgmc from deinterlacing the good frames?

You adjust the settings until it says "clean frame" instead of "combing detected"