Need help restoring poor quality DVD - Posterization - Color Banding

[pentajet]

First: I am a newbie. I am trying to learn but presently I know quite little about encoding. I am trying to restore an anime clip. The original DVD is very poor quality having jagged edges and severe aliasing problems and DCT blocks on some scenes. I repeat that these problems are present on the original DVD - they are not a result of the encoding. This is what I did so far:

1) Used Sandnom to deinterlace and get rid of the aliasing with aa=90. The results were IMPRESSIVE. Whoever created Sandnom is a genious.

2) Used TlSophote to get rid of the remaining larger jagged lines. This also got rid of most of the noise in the video. I used the default settings. Although the problem was not resolved completely I got pretty satisfactory results.

3) Used ffdshow Postprocessing SPP Deblocking to try to resolve the deblocking problem. The situation improved but I did not achieve the results that I was looking for.

I don't even know if the problem that I am experiencing is DCT blocks or not. It occurs on some scenes only especially static scenes with large panes of color like the grey faces of the robots in the cartoon.

This is a link to an example of the problem: http://www.badongo.com/vid/148141

When the truck falls down the mountain and the robots come to see what happened to it the top side of the truck (which is facing the robots after it fell on the ground) displays several "dancing" colors melting one into another and creating a very ugly effect. The same effect can also be seen when the robots try to put the truck back on its wheels (always on the flat top part of the truck).

Is there any way to improve the situation further. I repeat that this is not a problem of the encoding - the original is even worse. And I can;t see the problem on my monitor. It can only be seen on the TV when it's set to high contrast.

I tried several AviSynth filter (dozens of them including ColorSmooth. MIPSmoother, Convolution3d, Dup so many I can;t remember them all). None of them had any effect. The only thing that improved it slightly was the ffdshow filter which I set to 200% (full).

Any suggestions?

[lordsmurf]

MSU Deblock filter in VirtualDub does well.

[pentajet]

Thanks for the suggestion but I tried MSU Deblocking 2.2 and it had no effect at all. I am not sure if the problem I am experiencing is actually blocking as I don't see any blocks. It occurs on static flat panes which are supposed to be colored uniformly (for example light grey in the example in the video) but the uniform color is replaced by two similar colors that start "dancing" one into the other.

If anybody could download the file above (it's just 4Mb - 4 scenes) and tell me exactly what my problem is so that at least I can a proper research I would be grateful.

[lordsmurf]

Posterization.

Want to see that affect? Put a fancy wallpaper image on your Windows desktop, then drop it down to 256 colors. That's posterization.

Sample and definition is here:
http://www.cambridgeincolour.com/tutorials/posterization.htm

Panasonic DVD recorders do that, even in SP mode, it sucks.

Your anime is probably an unofficial HK or a homemade disc, or a shrunk retail disc, isn't it? I cannot imagine this problem existing on official anime releases, never seen that.

[pentajet]

It's the unofficial Italian version of a US cartoon from the '80s. Thanks Lord Smurf. Your advice has been very useful. If anybody has any suggestions on how to improve the video I would be grateful. So far I tried Smart Smoother HQ and grandfun2b. Grandfun2db has done an impressive job making the colors on flat panes more static but it actually increased the banding/posterization. Smart Smoother HQ does not do a lot - it helps but not much.

[lordsmurf]

Some sort of temporal filter, maybe. Use VirtualDub, install a lot of vdub filters (as you see them and think you need them), and just experiment.

That's exactly what I would do.

[Wilbert]

I tried several AviSynth filter

Try: GradFun2db: http://forum.doom9.org/showthread.php?t=108681

[pentajet]

Thanks Lord Smurf. i tried the "mplayer temporal noise reducer" in ffdshow and it slightly improved it. U had already tried Grandfun2b and it actually procuded more blocks than beforehand. I also tried ffdshow DeBand and the results were superb. The temporal noise and the banding are completely gone on all the cartoon except the scene which I uploaded here and a couple of others (the one above is the worst example in the whole series). Unfortunately it produced a good deal of ghosting. I think I will live with it 'though. Considering the awful source I was working with it's a miracle that I got this out of it. If anybody has suggestions on how to reduce the ghosting and blurriness in the picture I would appreciate it.

I also read that the scrpt mftoon might do miracles on anime and its feature to darken the lines might help in further eliminating the aliasing in the clip. Unfortunately I can't seem to make it work. i downloaded mftoon, awarpsharp, warpsharp and masktools but still everytime I try to use mftoon VdubMod returns "avsscrpt error".

[pentajet]

Looks like I shouldn't thought it's done, so early. The video indeed looks good with ffdshow's deband and most of the posterization is gone but it seems I can't encode it with the filter appliad. The ffdshow update notes say "deband edge emulation - only works when processing full frames".. I have no idea what that means.

I played a little with the color controls and it looks that the ffdshow filters are working during encoding because the "Picture Proeprties" filter definitely works but I set deband to maximum which during post-processing playback creates a very visible ghost image and it was not there on the encode so it's definitely not working (I did it to rule out the possibility that xvid might be re-creating the banding during encoding).

I also tried saving the source to an uncompressed format and applying the filter afterwards but NOTHING. Now I'm even more desperate than before since I know that this deBand filter definitely can fix my video and is the only thing that seems to work but I can't make it work. I am losing my sanity on this thing!!!!!!!

[pentajet]

Problem solved using the DirectShowSource command instead of the ffdshow filter.