Weird conversion results

[carlmart]

Some time ago I brought a query on a DVD I had which showed some strange things, like comb type distortions. The problem was diagnosed as "field blended garbage" and the script you suggested cured them.

Now've found another DVD which I believe is suffering from similar maladies. But I tried that same script and it didn't work. So maybe it needs some adjustments. Have a look at a this sample from the DVD:

http://www.mediafire.com/?41jn3ngyjew

[manono]

It's standard hard telecine and just needs a standard IVTC:

TFM()
TDecimate()

[carlmart]

Thanks, Manono!

Now please have a look at this other sample.

http://www.mediafire.com/?nmjzjhmxvkm

It's the same film and same images. But this comes from converting an MKV 720p file onto m2v.

As a result there's a jittering in the movement which I don't know how to prevent, because it was not in the original.

What do you suggest?

[jagabo]

My guess is your 720p file was 60 fps, probably with 3:2 frame repeats and your conversion to 23.976 fps didn't go well, resulting in some duplicate frames and some skipped frames. Try:

SelectEven()
TDecimate()

If that doesn't work there are some other possibilities...

[carlmart]

Here's the same part I had already uploaded, this time taken from an MKV 720p file.

http://www.mediafire.com/?sharekey=baad3af1c8fb5a28d6baebe61b361f7ce04e75f6e8ebb871

If I convert this file to mpeg2 I get a jitter effect.

What can I implement so it runs smoothly?

[jagabo]

That MKV file is already 23.976 fps. To make an NTSC DVD all you have to do is encode as progressive MPEG2 at 23.976 with pulldown flags.
http://www.mediafire.com/?1csemunsucs

[carlmart]

That MKV file is already 23.976 fps. To make an NTSC DVD all you have to do is encode as progressive MPEG2 at 23.976 with pulldown flags.
http://www.mediafire.com/?1csemunsucs

What script did you use?

[jagabo]

That MKV file is already 23.976 fps. To make an NTSC DVD all you have to do is encode as progressive MPEG2 at 23.976 with pulldown flags.
http://www.mediafire.com/?1csemunsucs

What script did you use?

DirectshowSource("test-014.mkv")

[carlmart]

It's standard hard telecine and just needs a standard IVTC:

TFM()
TDecimate()

Manono,


I have never used those filters. Can you help me with some variables to play with on each filter?

[manono]

You don't need any variables. Not with that sample, anyway, and not with most hard telecined sources. Load the TIVTC.dll (or place it in your AviSynth plugins folder), and use the 2 lines I gave you above. If it's not working then most likely IVTC is the wrong thing to use.

Oh, and make sure to use Field Operation->Honor Pulldown Flags when creating the D2V file, and not Force Film.

[carlmart]

You don't need any variables. Not with that sample, anyway, and not with most hard telecined sources. Load the TIVTC.dll (or place it in your AviSynth plugins folder), and use the 2 lines I gave you above. If it's not working then most likely IVTC is the wrong thing to use.

It worked alright. Thanks.

Oh, and make sure to use Field Operation->Honor Pulldown Flags when creating the D2V file, and not Force Film.

I had checked the pull-down in the HCgui screen. It ended up fine.

[carlmart]

It's been some time now that I have started tests with DVD as the playing medium.

As I am now seeing more apparent resolution due to the upscaling capability in my player and watching it on a plasma 42" 720p screen, I want to know how far I can take working on the image to get things looking better and with more resolution.

If the source is a "plain" DVD there's very little to compare it with, except my opinion or my wife's that it looks better.

That's why I am using some mkv 720p or 1080p files and down converting them to 720:480. Then I do have a higher resolution reference I can compare my results with.

Two have been my major filters in avisynth: lsfmod and Tweak(cont,sat). The first one is obvious, being a sharpener. The second is because I think that things "look" as having more resolution if you tweak contrast and saturation a little bit.

That mkv sample I had uploaded to Mediafire was my first test and where tweak() was more effective than lsfmod.

On another sample, also now on Mediafire results were not so clear. So I am now trying with another film.

Just to know if I may be onto something: may these filters, alone or combined, make a DVD "look" having more resolution?

The values I have been using were:

LSFMod(Smode=5)

and

Tweak(cont=1.0,sat=1.0)

[jagabo]

Tweak(cont=1.0,sat=1.0) should do absolutely nothing. Y * 1.0 = Y.

Personally, I find sharpening is rarely a good thing.

[carlmart]

Tweak(cont=1.0,sat=1.0) should do absolutely nothing. Y * 1.0 = Y.

Of course! Maybe it was the placebo effect then!

Personally, I find sharpening is rarely a good thing.

Let's see how it performs on the big screen with LSFmod. On the comparison test I just made it came out very well on a scene with many trees and leaves. Much better than the unfiltered one. Detail did come out.

But the plasma screen is better to see the likely artifacts.

In your opinion what might be a "good thing" then?

[carlmart]

There seems to be at least one problem in processing this file with lsfmod: it's slow.

My machine has an Intel Core2 cpu, with Intel chipset. It usually takes about two hours to convert a whole film.

From what HCenc is showing it will take more than 12 hours to go through 1st and 2nd pass. So it's at least six times more!

[jagabo]

If your video isn't blurry to begin with sharpening will create oversharpening artifacts. At first glance the results might look pleasingly sharp but if you look closely you will see bright and dark halos at sharp edges, aliasing problems (stair stepping), buzzing edges, and sometimes accentuation of moire artifacts.

If your source is blurry then some sharpening may be appropriate. But even this often leads to problems. Dumb sharpeners will increase noise along with sharpening edges. Smarter edge sharpeners will attempt not to increase noise in "flat" areas while sharpening edges. But if you look closely you will see artifacts in boundary areas. Some edges are very sharp and usually noisy, the rest of the picture isn't. It's usually obvious (without having seen the original fuzzy source) when this type of sharpener has been used. And in my book that's a bad thing.

With contrast and saturation you'll find the same thing. At first the higher contrast, higher saturation image looks better. But after a while you'll start noticing loss of details in the dark and bright areas and unnatural colors. Unless there's something wrong with your source you shouldn't need contrast and saturation adjustment.

Note that Tweak()'s contrast adjustment is really a gain adjustment. Both dark and light areas get brighter. ColorYUV()'s contrast adjustment is a true contrast adjustment. Darks get darker, brights get brighter.

Tweak: brightness' = brightness * N
ColorYUV: brightness' = ((brightness-128) * N) + 128)

[carlmart]

Unless there's something wrong with your source you shouldn't need contrast and saturation adjustment.

I think this is the key to the whole question. The source is not the problem: the final result is.

My question, probably one that many tried and would like their comments on, is how to make things look a bit sharper in DVD without going over the border onto artifacts of any kind. A lost battle?

[manono]

Tweak(cont=1.0,sat=1.0) should do absolutely nothing. Y * 1.0 = Y.

Nothing to the contrast or the saturation, for sure. It does perform a coring operation, though, and clip the luma to 16-235. Depending on the source luma, the difference could be noticeable. That's why I often (but not always) use it this way:

Tweak(Sat=whatever, Cont=whatever, Bright=whatever, Coring=False)

Of course, when converting hi-def sources to std-def, one of the most important things to do is adjust the colorimetry:

ColorMatrix(Mode="Rec.709->Rec.601")

jagabo, have you ever used LimitedSharpen and it's varients? I use LimitedSharpenFaster quite a bit and don't notice the kinds of artifacts you describe, particularly not the ringing which it was developed specifically to avoid, and which many other sharpeners introduce or accentuate. I often use it on lousy sources. When the source is good (like for hi-def scaled down to standard-def), others prefer to use SeeSaw or IIP (Integrated Image Processing).

From what HCenc is showing it will take more than 12 hours to go through 1st and 2nd pass. So it's at least six times more!

When you have a slow filter chain, it's sometimes beneficial in terms of the overall encoding time to either make a lossless AVI first, or run HC to create a lossless file, before then running the 2 passes on the lossless intermediate file. I usually run 5 passes using CCE and almost always create a Lagarith AVI first when doing heavy filtering, but even when using HC the encode will often take less time overall by first creating the lossless file.

Of course, much of the slowdown may be because of the Hi-Def source you're using, or can you usually encode Hi-Def sources with no filtering in 2 hours?

[carlmart]

Tweak(cont=1.0,sat=1.0) should do absolutely nothing. Y * 1.0 = Y.

Nothing to the contrast or the saturation, for sure. It does perform a coring operation, though, and clip the luma to 16-235. Depending on the source luma, the difference could be noticeable. That's why I often (but not always) use it this way:

Tweak(Sat=whatever, Cont=whatever, Bright=whatever, Coring=False)

What do you exactly set by doing this?

Of course, when converting hi-def sources to std-def, one of the most important things to do is adjust the colorimetry:

ColorMatrix(Mode="Rec.709->Rec.601")

Can you elaborate a bit over this, so I understand what I would be doing?

jagabo, have you ever used LimitedSharpen and it's varients? I use LimitedSharpenFaster quite a bit and don't notice the kinds of artifacts you describe, particularly not the ringing which it was developed specifically to avoid, and which many other sharpeners introduce or accentuate.

Same to me just now. The existing halos were accentuated, but there was little ringing to see.

Now I have applied coring=false and colormatrix, and the halos seem to have improved. Let me burn the DVD so I can see it on the plasma.

I often use it on lousy sources. When the source is good (like for hi-def scaled down to standard-def), others prefer to use SeeSaw or IIP (Integrated Image Processing).

Suggestions on what values to start with?

When you have a slow filter chain, it's sometimes beneficial in terms of the overall encoding time to either make a lossless AVI first, or run HC to create a lossless file, before then running the 2 passes on the lossless intermediate file. I usually run 5 passes using CCE and almost always create a Lagarith AVI first when doing heavy filtering, but even when using HC the encode will often take less time overall by first creating the lossless file.

I didn't quite understand how to implement this.

Of course, much of the slowdown may be because of the Hi-Def source you're using, or can you usually encode Hi-Def sources with no filtering in 2 hours?

Yes, HCenc takes about two hours to encode HD sources with no filtering, only Spline36Resize.

[manono]

What do you exactly set by doing this?

You keep it from clipping the luma values. You may want them all 16-235. I usually want to keep the overshoot and undershoot and adjust the values elsewhere, in the Tweak Filter, in YLevelsS, or in the AGC filter.

Can you elaborate a bit over this, so I understand what I would be doing?

Read the doc included with the ColorMatrix filter. Also, scroll down to the bottom of this page, read the Color Errors due to Chromaticity Differences section and play with the Previous and Next buttons to see how the colors change under the different circumstances:

http://www.sigmadesigns.com/public/Support/chromaticity.html

Suggestions on what values to start with?

I only work with lousy sources and don't reencode Hi-Def for Standard-Def, so I've never used the 2 filters. I only work with LimitedSharpenFaster.

I didn't quite understand how to implement this.

As I use CCE, I make a lossless Lagarith AVI and then serve it into CCE using AVISource. You install Lagarith following the instructions, open your .avs in VDub(Mod), and choose Lagarith under Video->Compression. I Configure it for YUY2 (and for Multithreading), as I'm using CCE. You'll want YV12 for HC. In addition, although I don't use HCEnc, the latest version has the abilility to create a lossless intermediate file. You check the box in Settings 3, "use lossless file". You'd then use what it creates to run the much faster second pass afterwards. I think it goes ahead and runs the 2nd pass automatically. This means you run the slow filtered script only once and with luck maybe the 2nd pass takes only an hour or so. Instead of your 12 hours, maybe it'll take only 7 (or 8) hours total. Although the lossless file will take a lot of hard drive space, the time savings makes it worthwhile, I think.

[jagabo]

Tweak(cont=1.0,sat=1.0) should do absolutely nothing. Y * 1.0 = Y.

Nothing to the contrast or the saturation, for sure. It does perform a coring operation, though, and clip the luma to 16-235.

Yes, I forgot about that.

jagabo, have you ever used LimitedSharpen and it's varients? I use LimitedSharpenFaster quite a bit and don't notice the kinds of artifacts you describe, particularly not the ringing which it was developed specifically to avoid, and which many other sharpeners introduce or accentuate.

I downloaded and tried it before replying to the OP. I saw all the usual sort of artifacts when used on already sharp material (test patterns and a few normal DVDs I looked at). Less than a simple Sharpen() but still there. Granted, I didn't fiddle with any other options. Sample 8x Nearest neighbor enlargements of LSFMod(Smode=5):



An upscaling player or TV may further accentuate the defects.

It's possible the OP's current equipment is low pass filtering the DVD before upscaling to reduce moire artifacts. In that case a bit of oversharpening might improve the final on-screen result. But I wouldn't screw up my recordings just to compensate for a poor player/TV.

[carlmart]

If anyone wants to have a look at some tests I made, they are at Mediafire.

The original mkv file:

http://www.mediafire.com/?gowljxutmjl

The unfiltered encoding to m2v:

http://www.mediafire.com/?wwlndemhmz0

Encoded to m2v with LSFMod(Smode=5)

http://www.mediafire.com/?ngkozyjndym

Encoded to m2v with Tweak(Sat=1.0, Cont=1.0, Bright=1.0, Coring=False) & ColorMatrix(Mode="Rec.709->Rec.601")

http://www.mediafire.com/?ntztmmmi4zl

Just now I used my laptop to look at the original files fed directly to the plasma in 720p. Truth is I can't find apparent differences between them, which should be there. Maybe the original is not as high resolution as it might be.

In any case the encoded ones look quite good and much better than my original DVD for that film.

One thing I discovered in the upscaling my DVD player does is that it seems to force resolution somehow, because the halos got quite visible in the DVD I made from the tests. I wonder if I can correct that on the TV setup for the HDMI input. At first I had thought the halos were on the original file or on the encoded ones, but the existing halo is quite dim and you have to look for it to find it. It was good to check that on the original files, compared to the burned ones.

[jagabo]

Encoded to m2v with Tweak(Sat=1.0, Cont=1.0, Bright=1.0, Coring=False)

All that is doing is brightening the picture by 1 unit and generating some integer/floating point rounding errors.

sat' = sat * 1.0 (does nothing)
cont' = cont * 1.0 (does nothing)
bright' = bright + 1.0.
Coring=False -- don't limit results to 16-235, limit to 0-255 instead

[carlmart]

Encoded to m2v with Tweak(Sat=1.0, Cont=1.0, Bright=1.0, Coring=False)

All that is doing is brightening the picture by 1 unit and generating some integer/floating point rounding errors.

The only I wanted to keep was coring.

sat' = sat * 1.0 (does nothing)
cont' = cont * 1.0 (does nothing)
bright' = bright + 1.0.

You're just randomly putting in numbers. You ought to read the documentation.

Sorry, just did. Default for bright is 0.0. It was not randomly: it was a mistake, thinking (wrongly) that correction was also above 1.0, and would be alright keeping that value.

[jagabo]

With Tweak(Sat=1.0, Cont=1.0, Bright=0.0, Coring=False) you're saying don't do anything to the image but convert it to floating point and back to integer (and the errors that will introduce) just for fun:

sat' = sat * 1.0 (do nothing)
cont' = cont * 1.0 (do nothing)
bright' = bright + 0.0 (do nothing)
Coring=False (do nothing)

That's a lot of work just to do nothing.

Here's a little tip that I think you will find helpful: use the Interleave() command to interleave two video streams together.

v1=DirectShowSource("video.mkv")
v2=Tweak(v1,Sat=1.0, Cont=1.0, Bright=0.0, Coring=False)
Interleave(v1,v2)

This gives you an easy way to compare before and after results, or two different filter paths, of AviSynth filtering. Open the script in VirtualDub and you can easily flip back and forth between filtered and unfiltered frames with the arrow keys. You'll see the above script makes no visible changes.

Or try something like this:

v1=DirectShowSource("video.mkv").Lanczos4Resize(720,480)
v2=ColorMatrix(v1,Mode="Rec.709->Rec.601")
Interleave(v1,v2)

[carlmart]

Thanks, I will try to use the Interleave tricks on what I am doing now. From what I saw there's not much I can do when the source is good. And the conversion looks very nice only with ColorMatrix(Mode="Rec.709->Rec.601"). So why not leave that alone?

But Manono mentioned the lousy sources he worked with, so I am using that tip to perhaps deal with one of mine.

I have several DVDs that were originally mastered in letterbox 4:3. The one I uploaded is one of them.

http://www.mediafire.com/?z5wjujjyimm

For now I just corrected for size and telecine problems, which already looks better than the original. If you want I can take the filters out for you to see what it was. I used:

MPEG2Source("d:\tmp\VTS_01_0.d2v")
TFM()
TDecimate()
spline36resize(720,480,0,66,0,-66)

From what I see grain is a problem. Sharpening might improve things a bit, perhaps, but I am not sure.

Does this one qualify or not as a lousy source?

[manono]

The link is to a non-letterboxed 4:3 film that's already progressive.

When I mentioned bad sources, I meant bad sources. They're NTSC DVDs of old Indian films sourced from PAL VHS tapes mostly.

[carlmart]

The link is to a non-letterboxed 4:3 film that's already progressive.

OK. Then I can upload the unprocessed original file trim. It's letterboxed 4:3. I can demux that same part with DGIndex. Will that do?

When I mentioned bad sources, I meant bad sources. They're NTSC DVDs of old Indian films sourced from PAL VHS tapes mostly.

OK. They are worst than mine then.

[carlmart]

Here's the original file, unfiltered and non-resized.

http://www.mediafire.com/?mzzytyhlyj5

[manono]

Oh, sorry. I didn't notice that it was 16:9 and the result of your efforts. I thought you had uploaded the wrong file.