VHS to DVD with Canopus ADVC110 - mostly filter/cleaning questions

[jagabo]

For the most part I recommend not using noise filters.

The Dot Crawl filter is for removing Dot Crawl artifacts, not general static noise. I saw no dot crawl artifacts in the image you posted (although the jpg compression may have reduced them). Dot crawl is most noticeable at sharp edges -- as in the link I posted ealier.

[Z-C]

After reading in wikipedia and thanks to your reply I understand better what dot crawl is. Yet, in the mean time I did try the dot crawl filter anyway. It actually does a pretty good work at removing noise. Unfortunately, it also causes quite a serious blur. alteast in my PAL video.

Regarding not using denoising filters - May I ask why not? Isn't the image I posted very noisy? the rest of my films are usually at good quality when it comes to other possible errors, but the noise is really disturbing.
I understand that noise filters also blur/make stuff look more like plastic/anime, but there should be a balance between diminished noise level and not-to-noticeable blur - or am I wrong here?

I need to see the output on the TV and not on vdub preview to know better, but it seems the denoise filters I used did a pretty good job, without too much blur(there is some though).

[lordsmurf]

Every video has it's own need. Unless I see the video or I know precisely what error is happening, it's pretty much impossible to make specific suggestions. With audio, it's easier to do that. But video is so big and you have to be sure the hardware in the analog domain is being used to its full extent before being transferred to digital data.

That's frustrating, but at least we're trying to give a push in the right direction.

[Z-C]

lordsmurf - dont get me the wrong way. You're being very helpful and I appreciate what you do. After all, all the help you supply here is a result of nothing more than your good will.

Please tell me what i need to post here. i did post an example frame - is that not enough? do you need me to upload a short clip?

I have an opinion of which filters are better and what settings to use(after doing quite some experiments). I just want to see if I'm seeing eye-to-eye, with the more experienced users, and if I'm not missing something, or overadjusting.

[guns1inger]

For noise removal I have been using Convolution3d for a while, and have been generally happy with what it does, however I have been working on some tapes of late that are 2nd and 3rd generation, over saturated and extremely noisy. Convolution3d wasn't able to remove the noise without killing too much detail. I decided to give PeachSmoother a go, something I hadn't used for a while, and it has done a very nice job. I would give it a try before settling on just a 2d noise filter, even though you do pay a price in performance.

[Z-C]

Thank you guns1inger.
But, aren't those avisynth filters?

What do you mean by pay the price in performence? -slow processing speed? or killing details?

[guns1inger]

Yes, they are, but honestly, it is worth having a look at. A simple script with a few lines of code can do wonders, and doesn't preclude you from using your vdub filters as well, if necessary. The cost with all good noise filters is speed. Especially if they are doing temporal filtering as well.

[Z-C]

guns1inger: thank you again I took your advice and tried what you did.

I have a small issue though when using convolution3d. It seems that I get an error saying that convolution3d can only work in YUY2 color mode.
I did try using the converttoyuy2 function, which seems to work. However, I'd like to avoid the mode convertion. Is that possible? can't convolution3d work in YUV?

My second question is - when does avisynth apply the filters? I opened the avisynth script directly in CCE and started encoding with it. It works fine, but I'm just curious - when does the convolution3d filter kicks in? because the process didn't seem slow in any way.

[jagabo]

I have a small issue though when using convolution3d. It seems that I get an error saying that convolution3d can only work in YUY2 color mode.
I did try using the converttoyuy2 function, which seems to work. However, I'd like to avoid the mode convertion. Is that possible? can't convolution3d work in YUV?

YUY2 is a form of YUV. Since convolution3d only works in YUY2 you can't avoid the conversion. Actually there may be a way to improve things. I don't know what DV decoder you are using but if it's outputing RGB or YV12 (and hence requiring the conversion to YUY2) you may be able to force it to output YUY2 directly. You'll have to look into the decoder's settings.

My second question is - when does avisynth apply the filters? I opened the avisynth script directly in CCE and started encoding with it. It works fine, but I'm just curious - when does the convolution3d filter kicks in? because the process didn't seem slow in any way.

AVIsynth filters frames as it serves them to the MPEG encoder.

[guns1inger]

How much things slow down depends on the filter, and how hard it is working. PeachSmoother seems to be much slower than Convolution3D, which is one of the reasons I tend it use it less often. Both get slower the more noise they have to remove.

[SerbianBoss]

there's YV12 version of convolution3d filter but temporal filtering is off.

Just to say that convolution 3d is excellent filter.Degrainmedian is also good and much faster than convolution.

@guns1inger and others

try combination of peachsmoother and vague denoiser. Is's good combionation if source is very noisy.It cleans very well.

One more thing.

If you have installed panasonic dv codec and filter with avisynt filters picture is little sharper.But if you usind cedocida dv codec(YUY2 or YV12 colorspace) picture is little lively and it's better noise reduction.

[Z-C]

jagabo: I used Panasonic DV codec which decodes to RGB. So I changed to canopus DV codec which I thought decodes to YUY2... but I guess I was wrong? Nothing special in settings for those two.

Here's a little update on my experiments:

Convolution3d gives the best results by far, above all Vdub filters with these settings - Convolution3D(0, 32, 128, 16, 64, 10, 0).
However, there is a downside - the picture is more blurred than with any Vdub filters, but it's not really noticeable. It seems that using other settings in Convolution3d still blurs the picture the same, but leaves much more noise intact.

MSU denoise gives the best results on the Vdub filters part. But for some reason, after frameserving and encoding with CCE - they video stream is slow(not exactly slow motion, but still slow) and jumpy - it's so intense that you can't even watch the video without making your eyes hurt. Why does this happen?

SerbianBoss: Is that mix(peachsmoother and vague denoiser) good for semi-noisy sources? Or will it hurt the video too much(too much blur etc...)

In any case - peachsmoother is my next go. However it'd be great if you guys could tell me why the movie encoded using Vdub filters gives me all that problems, and what I can use to avoid converting Color modes? cedocida codec? I am not familiar with that one so a bit hesitated to use it...

EDIT: One VERY important note - I burned the same clip on a dvd using different denoise filters and settings. some were severely denoised, and others weren't. On the PC you could easily see the difference. But on my TV set I couldn't notice a difference even if I tried. So I'm wondering, is it worth all the hassle? Or in the future when I get an LCD/Plasma TV set to replace my ancient analog TV I'd be thankful I cleaned the movies from noise?

Thank you all again!

[jagabo]

Noise reduction removes detail as well as noise. Once that detail is removed you will never be able to get it back (unless you go back and capture your video tapes which are degrading with time). And when your video is displayed at 50 or 60 fields per second on a TV the noise is not nearly as visible as when you look at still frames on the computer. This is why I recommended you not bother with noise filters.

[Barnabas]

Z-C,

Noise is MUCH more visible on the computer and an LCD TV. Noise I can't see at all on my 52" CRT TV, looks VERY noisy on my 27" LCD HDTV. I had someone bring me a tape that was a copy of a tape done at slow speed to cleanup. The noise was terrible. While I did improve it a bit with the use of filters, there is NO way terribly noisy video will ever look like new when filtered.

There is a $100.00 VirtualDub noise filter I tried last year that did in incredible job of removing noise from DV video shot on low light. I forget the name, but it did better than any other filters I have tried. I kept that same footage on my computer for testing other filters, and found that the filters I mentioned to you previously were fairly close in quality, but you have remember, each video is different, and has to be filtered accordingly.

[Z-C]

jagabo: You make it sound so critical. I mean, even if some detail is loss, the picture generally looks better. and even though it does blur some stuff, you can't really notice it unless you compare frame-to-frame and examine it carefully. Atleast that was the case so far. It might differ with other footage.

I'm still not sure to what extent I want to use the denoising filters, but I'll probably look for a middle ground.

I must use some denoising filters because, like Barnabas said, on LCD TVs the noise is going to be very noticeable and disturbing. And since I have so many tapes(something like 150 hours of footage), I wouldn't want to repeat the capturing process again when I get a progressive TV.

Barnabas: Any chance you can locate the name of that filter?


And the more important part - could someone explain me why my video stream jumps all the time and playbacks much more slowly if I use Vdub for filtering instead of Avisynth?

[lordsmurf]

Video samples are required. I often have people mail me samples, because uploading/downloading just won't cut it for thorough feedback.

[FulciLives]

My favorite "video noise" filter for interlaced video is Convolution3D

Here is a link to a guide on how to use it properly with interlaced video ---> CLICK HERE

When using Convolution3D I suggest you use the YUY2 version instead of the YV12 version. I believe someone already stated the reason in a previous post.

- John "FulciLives" Coleman

P.S.
When using Convolution3D with an interlaced video source you have to separate the frames into fields ... apply the filter to the fields ... then combine the fields back into a frame. The guide that I linked to covers how to do that properly.

[Z-C]

FulciLives: I have already read your guide before which was excellent!

I still don't understand why I need to convert to YUY2, if the DV was captured in YUV.

SerbianBoss and all: I tried PeachSmoother, and vague denoiser separately. The results were much less satisfying than Convolution3d.
However, thanks to your advice, I tried the combination. The results were astonishing! It cleans up as well as Convoltion3D, but with significally less blurring!

I still don't get it. I thought combining two denoising filters would mess up my video. But it seems it keeps the details quite well. How did you find this combination?

By the way, do I need to separate the fields in those filters like in Con3d? Because I didn't, and it seems to work just find.

I'm pretty sure I'm going to make this combination my normal denoising choice.

[jagabo]

I still don't understand why I need to convert to YUY2, if the DV was captured in YUV.

There area many different YUV data packings:

http://www.fourcc.org/yuv.php

[guns1inger]

One thing I have learned from working with a range of footage is that there is no absolute answer. Every video is different, and requires it's own solution. No single denoiser is perfect for every occassion, and every one of them needs some serious tweaking to get the best from it. Don't expect to use the same settings as the last clip on something new, and to get the same results.

As I said earlier, I have been using various settings in Colvolution3d for a while now quite happily, but with the tapes I am working on at the moment, it just wasn't satisfactory. In this instance PeachSmoother produced more satisfying results more quickly (in part because it has a readout to help you narrow down some of the numbers). The moral of story is that you need to have options, so learning a couple of different filters is well worthwhile.

PeachSmoother actually works better on a combined source, whereas Convolution3D prefers a seperated source. I guess it is just how it's written.

[Z-C]

guns1inger - I agree with you in this case. Unfortunately I have so many tapes to convert that I don't know how much time I will have to find the optimal settings for each movie.

That's why I'm saving all the avisynth scripts using different denoising filters and settings. Hopefully, in most cases the results will be satisfactory with those "templates".

guns1inger - According to documentation, PeachSmoother adjusts itself to the source, so it should be able to handle a variety of sources with the same settings, unlike Con3d. Or is this statement false?

I don't know if vague can adjust though, I hope it can since the combination between the two seems to work really well!

[guns1inger]

It can. You can see it in action by setting the parameter ReadOut to True and watching the numbers tick over. However, depending on the the footage, this may be slower, and unsatisfactory. The footage I am working on at the moment is live dance concert footage. There are camera flashes and strobes going off, and if I leave PeachSmoother to adapt itself, it tend to get noise pulses. So I turn on the readout and run it through the footage (or at least representative samples of the worst parts) to get an idea of the numbers it is calculating, then use these to set the NoiseLevel and Baseline parameters. Baseline is usually pretty close to the bottom number. Noiselevel I tend make larger than the peak value seen in the readout. There is still room to adapt, but it is based around these settings so it is faster and more consistent.

If you have a range of environments, but the changes are slow or subtle, it might be OK to leave it up to it's own devices.

Here are the settings from a particularly noise, oversaturated piece of footage with flashes and strobes and smoke.

PeachSmoother(ReadOut = False, Dot = False, NoiseLevel = 7.6, Baseline = 4.3, NoiseReduction = 60, Stability = 30, Spatial = 100)

In my case, it leaves a nice level of detail while keeping everything clean. When I get some time I will also have a look at combining it with VagueDenoiser to see what effect that has.

[lordsmurf]

Unfortunately I have so many tapes to convert that I don't know how much time I will have to find the optimal settings for each movie.

I don't know how to say this without coming across like an ass....

... but that's really just too bad.

Video takes time. That is the first rule of video work.

I've been plodding along on personal DVD projects for the better part of 5 years now, and it'll probably take me 5 more before I finish everything I want to do. Sure, I could rush through all those videos, but it would be wasted effort if they came out looking like crap.

[Z-C]

lordsmurf - Don't you think the degrading that will occur in your analog footage is worse than applying certain filters with mediocre settings(because they weren't fully optimized)?

I am aware it's going to take time. But I'm setting myself a goal to finish my movies in approx. one year.

guns1inger: readout supplys the numbers for both NoiseLevel and Baseline? If using fixed noiselevel and baseline numbers can give better results then it's worth a shot. But does it really matter that much? or is it mostly boosting the speed? I don't care about slow processing..
Regarding the example you put here - I will try it out and play with it to see how it effects my movies.
I am using NoiseReduction of 80. But could you tell me what Stability does? I didn't understand what it means from the documentation, and frankly when I played with it - I didn't notice any difference.

[lordsmurf]

No.

By "analog" I assume you mean magnetic tape media. Tapes degrade, absolutely, but a 5-year or even 10-project is still plenty fine. Your tapes will still be just about as good as the day they were recorded.

[guns1inger]

Depending on the footage, specifically sudden changes such as strobe lighting or camera flashes that last only a frame or two, leaving PeachSmoother to it's own devices can result in pulses of noise appearing as it starts to re-estimate noise levels again. By using fixed values for NoiseLevel and Baseline, you force PeachSmoother to stay on course. In order for this to work well, you need a couple of conditions to be true - relatively constant footage - preferably from a single take or recording - and you need to have taken readouts from the bulk of the footage.

If changes in your footage tend to be subtl, or occur over a number of frames instead on only one or two, then you may be better off leaving PeachSmoother in adaptive mode.

Stability seems to be a buffer for areas where the filter isn't sure if it has motion or not. It dictates the amount of colour blending. From what I can see, you need to make fairly large changes in the setting to get obvious results. Try setting it to 50 or 70 and see if you notice a difference.

I know stills don't do the same justice as video, but these are from the video I'm working through at the moment. Left side is original, right-side is PeachSmoother (with the settings from above, give or take) and the vdub filter colormill

[SerbianBoss]

guns1inger does this samples is divx or dvd?

[guns1inger]

DVD. These came from VHS source, capped through a TBC-1000 and then A/D pass-though on my Panasonic DV camera to the PC as interlaced PAL DV avi. These were processed through avisynth encoded at CBR 7600 kbps as they are part of a short multi-angle test. The final clips are encoded at around CBR 8700 kbps + audio.

The screen grabs were taken with VLC from the interlaced PAL mpeg2 footage. The tape is ten years old, but some of the footage on it is 5 - 6 years older than that, so this is 2nd and 3rd generation dub in some cases.

[grannyGeek]

this is a great thread!
so good to see some concrete discussion going on.
Z-C, you are in really good hands!

May I ask a related question?
@FulciLives -- Re Convolution3d --
every sample script I have seen posted on several forums includes the commands to SeparateFrames / Weave
I think this has caused most of us newbies to think this is required for all video, not just interlaced.
Convolution3d ReadMe doesn't discuss Interlaced / Progressive, but your post here implies that progressive doesn't need the Separate / Weave commands.
Z-C apparently found that out in practice.
Or did I mis-understand ?

@ Z-C, I just wanted to mention that my own workflow usually includes using scripts to trim my captures into shorter segments for filter setup.
Often one scene will want a bit of denoising, but the next one won't, and the one after that needs some color correction.
I set these sections all up with individual aviSynth scripts in VDubMod, and then make a final "master" script that imports all the section scripts, and concatenates them all back into a single video clip.
That script is what I send to Quenc to be encoded to mpeg.
So far, this is working better for me than trying to find a single "Fits-All" filter combination to use across the entire length of the movie.

Keep posting back your progress, I'm getting a wonderful education from this thread.

granny

[guns1inger]

Re: splitting the clip up for different needs - definately a good suggestion. I was thinking as I was posting earlier about using PeachSmoother in adaptive and non-adaptive modes, how nice it would be to have some type of keyframe constuct in avisynth for just this type of occasion.