How to reduce jaggy edges/aliasing in HD>SD conversion?

[Killer3737]

As per the title I'm looking for some tips in reducing aliasing and jagged edges when downscaling HD to SD. The particular source I'm working with now is high quality, sharp and clear. I've uploaded a sample here (134Mb). (Yes for those that know me, it's a Foo Fighters concert again. ) I first encode it to a lossless intermediate with ffmbc and then encode that AVI with Avisynth in either HCEnc or CCE.

As I understand it with already sharp content you don't want to use something like Lanczos because it'll make it worse. Just for comparison though sake I've tried that, all three splines, and Bilinear. Expectedly the latter gave the best results:

AviSource("output.avi")
AssumeTFF()
Yadif(mode=1, order=1)
BilinearResize(720,480)
AssumeTFF()
SeparateFields()
SelectEvery(4,0,3)
Weave()
ColorMatrix(mode="Rec.709->Rec.601", clamp=0)
ConvertToYUY2()

Which you can see here. Playing it back most of the time it looks ok but certain edges still look jagged, particularly the neck and underside of his guitar in the first 20 seconds and then some of the crossing markings on the outside shot.

Is it just something I have to live with on some level when downscaling with consumer tools or could it get a little better? Also is the fact the source is 1280x1080 having any bearing on it?

[poisondeathray]

Try Iresize

You don't need ffmbc for MPEG2 Sources, you can use DGIndex, Mpeg2Source() . The ffmbc workaround is usually only for interlaced AVC transport streams

Mpeg2Source()
AssumeTFF()
IResize(720,480)
ColorMatrix(mode="Rec.709->Rec.601",clamp=0)


function IResize(clip Clip, int NewWidth, int NewHeight) {
  Clip
  SeparateFields() 
  Shift=(GetParity() ? -0.25 : 0.25) * (Height()/Float(NewHeight/2)-1.0)
  E  = SelectEven().Spline36resize(NewWidth, NewHeight/2, 0,    Shift)
  O  = SelectOdd( ).Spline36resize(NewWidth, NewHeight/2, 0,   -Shift)
  Ec = SelectEven().Spline36Resize(NewWidth, NewHeight/2, 0,  2*Shift)
  Oc = SelectOdd( ).Spline36Resize(NewWidth, NewHeight/2, 0, -2*shift)
  Interleave(E, O)
  IsYV12() ? MergeChroma(Interleave(Ec, Oc)) : Last
  Weave()
}

If it's still too much, you can change the resizing in the function to bicubic or bilinear.

Another technique is to lowpass using a vertical blur

If you're really picky, you could do some compositing and apply filters only on areas that need it, to avoid damaging "good" areaas

[Killer3737]

Ha, I've no idea what made me think it was H.264, or how I didn't notice it in any of the logs it wasn't. More sleep needed, clearly.

Anyway I'll give IResize a whirl, thanks.

[poisondeathray]

This animated gif simulates what you'd expect to see on a normal DVD setup (no AR correction, just in terms of motion/aliasing artifacts) . It's just Bob() after the script

A is the script from post #1
B is IResize

A



B

[Skiller]

Unless the HD source is very blurry, HD to SD conversions always need a vertical low-pass to avoid the jaggies. So either try IResize or use a Blur-Sharpen combination before the re-interlacing. Personally I think IResize is OK but I prefer Blur-Sharpen because it gives me more control over the required low-pass strength whereas IResize is quite conservative, often wasting a bit too much vertical detail in my opinion (I'm judging the output on a Sony PVM interlaced CRT studio monitor).

I would probably use this:

MPEG2Source("x.d2v")
AssumeTFF()

ColorMatrix(mode="Rec.709->Rec.601", clamp=0, interlaced=true)

Yadif(mode=1)

Y = ConvertToY8().Spline16Resize(704, 480)
U = UToY8().Spline16Resize(Width(Y)/2, Height(Y)/2, src_left=0.25*(1-Float(Width(Last))/Width(Y)))
V = VToY8().Spline16Resize(Width(Y)/2, Height(Y)/2, src_left=0.25*(1-Float(Width(Last))/Width(Y)))
YToUV(U, V, Y)

MergeLuma(Blur(0, 0.7, false).Sharpen(0, 0.4, false))

SeparateFields().SelectEvery(4,0,3).Weave()

The resizing part looks way more complicated than it is; it's a workaround for a known (minor) chroma shift bug when resizing in horizontally subsampeld color spaces (see here for details).
The low-pass is applied only to the luma.

For encoding with HCenc there is no need to convert to YUY2 by the way.

[pandy]

http://forum.doom9.org/showthread.php?t=168315

[Skiller]

@ pandy
The issue is not aliasing caused by the resizer itself, it's because of the resulting interlaced video exceeding Kell Factor (too much vertical detail). Interline Twitter is the correct term of the resulting artefacts.

[pandy]

@ pandy
The issue is not aliasing caused by the resizer itself, it's because of the resulting interlaced video exceeding Kell Factor (too much vertical detail). Interline Twitter is the correct term of the resulting artefacts.

Once again i advise to read whole thread on provided link - please don't involve Kell Factor as this is something else and is not related or limited to interlace video only.
As a general AA processing is related how to provide subjectively better video - mathematical approach used traditionally in signal processing may provide suboptimal perceived quality.

btw
Interline twitter means that resizer (downsample/resample filter) is insufficient to suppress aliasing i.e. Nyquist criterion are not meet - that's why special AA processing need to be applied.

[Skiller]

Once again i advise to read whole thread on provided link

OK, I just read that whole thread. I'm sorry, I do not understand how any of that is related to this issue. 2/3 of that thread is about AviSynth x64 and memory usage and I fail to see a significance between Interline-Twitter/Nyquist/AA and Sangnom/FTurn.

As a general AA processing is related how to provide subjectively better video - mathematical approach used traditionally in signal processing may provide suboptimal perceived quality.

Not sure what you're trying to say.

[pandy]

Once again i advise to read whole thread on provided link

OK, I just read that whole thread. I'm sorry, I do not understand how any of that is related to this issue. 2/3 of that thread is about AviSynth x64 and memory usage and I fail to see a significance between Interline-Twitter/Nyquist/AA and Sangnom/FTurn.

Well thread is named "FTurn and SangNom2 - plugins for faster antialiasing" and key antialiasing is the one you searching for.
Correct AA preprocessing need to be applied - usual resizers don't suppress sufficiently aliasing and if they suppress aliasing well enough then overall picture is perceived as blurry.

As a general AA processing is related how to provide subjectively better video - mathematical approach used traditionally in signal processing may provide suboptimal perceived quality.

Not sure what you're trying to say.

Common approach in resizers is windowed Sinc http://en.wikipedia.org/wiki/Sinc_filter filter - usually such filter is correct from mathematical perspective but it introduce pre and post ringing - ringing provide good spectral characteristic but have bad effect as it is visible - human eyes and our brain don't see world in same way as spectrum analyzer - thus different than windowed Sinc resizers are better from perceived/subjective quality perspective.

Mentioned by you interline twitter is just visual manifestation for aliasing and nasty behavior of poor resampling filters where phase flickering may be observed (this is nicely visible on progressive screens where high contrast thin line is rasterized (drawn) on screen - small angle and/or position changes may reduce perceived brightness of line and sometimes line can almost disappear - without AA it will disappear and then reappear).

There is many topics dedicated to AA preprocessing for example http://forum.doom9.org/showthread.php?t=167480 but as a general rule they concentrate on proper resizing algorithm (and windowed sinc kernels always giving suboptimal results so unless you doing something for spectral characteristic i.e. measurements then you should avoid windowed sinc - http://linustechtips.com/main/topic/149418-madvr-or-specifically-nnedi3-filter-a-stron...bled-in-games/ ) .

And yes i know what i'm saying.