Anti Aliasing caused by field shifts

[Akai-Shuichi]

Is there any good ways to deal with anti aliasing caused by field shifts, as in the aliasing moves, but the picture doesn't. I've tried several AA filters such as daa(),maa(), aaa(), etc they don't seem to help much.

[poisondeathray]

not sure what you mean by "field shift" ? If there is a field shift, you can shift the field back - but we're probabaly using the term "field shift" differently

A temporal smoothing filter like ttempsmooth with strong setting might be able to "fix" your problem , but they can be quite damaging at high settings . Or QTGMC in one of the progressive modes (inputtype 1 or 2)

maybe you can post a sample or try to describe the problem better

[Akai-Shuichi]

Ok here is a short clip: http://www.sendspace.com/file/i2zexl

If you look at the text/characters you can see the aliasing is like causing the frame to jump a little bit.

[poisondeathray]

Ok here is a short clip: http://www.sendspace.com/file/i2zexl

If you look at the text/characters you can see the aliasing is like causing the frame to jump a little bit.

typically the term "aliasing" in video describes jaggy edges or stairstepping

If you're referring to the text moving (up, down , left, right, ), you need a stabilizer, like depanstabilize, stab, deshaker etc..

You also have residual combing (like horizontal lines, looks like serrated edges) . You can use vinverse for that , or better IVTC settings in the first place (you probably had an orphaned field)

EDIT: ok I think I know what you're getting at - the movement in the frame causing aliasing to result in a few frames?

[Akai-Shuichi]

Yes that is it.

Also about the IVTC settings. I used

tfm(order=1).tdecimate(mode=1,hybrid=1)

because the source was 29fps progressive and DGIndex gave me FILM 99.8%, but i noticed some interlaced parts (I'm guessing thats what combing is) so i set tdecimate to hyrbid=1. If there is a better way of doing that then please let me know.

[poisondeathray]

99.8% is pretty close

You have to use the debug mode to figure out which decisions TFM is using. It sort of explains it in the instructions . It might be as simple as setting clip2 to use a better deinterlacer like nnedi3 or adjusting the comb detection parameters - cthresh (if that section was detected as "combed" it would apply a different deinterlace according to clip2)

Ideally you want to fix it at that earlier field matching stage, instead of applying some AA filter aftwards (which will degrade everything)

[Akai-Shuichi]

I think I know what you are getting at, but I am not 100% sure how to write it in avisynth. Could you give me some sort of example?

From the directions I'm guessing that you have to use VFR 2-Pass for this.

[poisondeathray]

Without seeing a sample of the source, I can only guess

TFM(clip2=nnedi3())
TDecimate

This assumes that TFM decided to deinterlace that 1 field . In default mode, PP=6 , so it uses bicubic deinterlace (this is similar to bob() - it just resizes the field - that's why there would be aliasing)

[Akai-Shuichi]

Ok the combing is gone, but the interlacing is still there. x.x

[poisondeathray]

I was guessing it was a cadence break.

Does this occur in other sections? Because it's easy to replace with a still frame (there won't even be any movement) for a title section . 99.8% is fairly high so there can't be that many sections.

If this occurs in other sections you can try adjusting the other TFM settings according to the debug mode (see the instructions)

Did you try "force film"? I've seen examples where it will do better than using TFM/TDecimate default settings when high % is specified

If you upload a sample you might get more specific suggestions

[manono]

To reiterate, upload an untouched sample so he (we) doesn't have to guess. 10 seconds with steady movement should be plenty.

because the source was 29fps progressive and DGIndex gave me FILM 99.8%

Those two statements are contradictory. It can't be both progressive 29.97fps and Film. Film, by definition, is 23.976fps. I'm guessing you mean it's progressive 23.976fps with pulldown to output interlaced 29.97fps. But, as I said, we shouldn't have to guess. If you don't know what you have, untouched source video speaks louder than 1000 words.

And, as pdr suggests, 'Forced Film' is often the better choice (but not always) when the Film percentage is so high. Or do it this way (with the Field Operation set for 'Honor Pulldown Flags'):

TFM(D2V="c:\oursource.d2v")#use the D2V to effectively use 'Forced Film' on the soft telecined parts
TDecimate(Mode=1)#Mode=1 for anime

Unless you have evidence of it being a real hybrid, you're just asking for trouble by using that setting.

[Akai-Shuichi]

I'm currently uploading a sample. I'll try out force film and see if it is any different.

[jagabo]

because the source was 29fps progressive and DGIndex gave me FILM 99.8%

Those two statements are contradictory.

I was going to say the same thing. We'll see what the sample looks like.

[Akai-Shuichi]

Here is around a 2:00 clip. http://www.sendspace.com/file/lozi3h

[poisondeathray]

Well there is aliasing in the source in some sections. It's not from cadence break or mismatched field, and you're right - some of them are from the "bounce" in the fields, some of them aren't (e.g. the subway pan)

If you want a brute force "fix" that addresses all of them - I would IVTC (either force film or TFM/TDecimate) , followed by santiag(3,3) . It's not that damaging on this type of anime (because there isn't that much fine detail to begin with)

The few aliased shots I had a quick look at comparing (IVTCed frame numbers) when using santiag
0036 title sequence
0521-527 bird
0900-0901 subway pan
1001 sign

I didn't look at everything or in much detail, it's just a quick fix. It's "brute force" in that everything is processed - even "good" frames. Maybe someone else has a more targetted/specific approach

[Akai-Shuichi]

Thanks! This is exactly what I've been looking for!

I am amazed at fast this is compared to AAA() it is also seems to be much more effective

[poisondeathray]

Thanks! This is exactly what I've been looking for!

I am amazed at fast this is compared to AAA() it is also seems to be much more effective

But just a warning - It's also very damaging on higher quality material with fine details. The only reason it works "ok" here is this type of animation is low detail to begin with. If there was some way to selectively apply the filter it would be better . Some AA functions are better than others in certain situations

[Akai-Shuichi]

Yea, I know how bad it can wreck details. I've used AA filters on stuff from hulu which pretty much any anime from there is notoriously aliased and you kind of have to find a balance between the AA and detail preservation.