VHS to DVD with Canopus ADVC110 - mostly filter/cleaning questions

[Z-C]

Hello everyone.

I've got a Canopus ADVC 110, for the main purpose of having my old vhs/hi8 home videos transferred to digital format for better preservation and editing. After the editing and cleaning part is over I intend to get the movies into DVD-video so they will be easy to use on a standalone DVD Player.
Just for you to know - Most of my videos(if not all of them) are PAL.

I've been doing some reading, but I must admit that the more reading I do, the more confused I get. So I guess it's time I came here asking for some guidance. I'll number my questions to make things easier for all of us.

1. As far as I understand, the DV program I use has no importance, since the canopus does all the encoding work within the external device. Is that true? Or are there any tweaking programs for better output while doing the analogue-DV part. I mean if that's really the case then why should I use scenalyzer and not a free and easy tool like WinDV?

2. I have bad sources, which I would like to improve, or atleast maintain current quality, because these are my home videos that are supposed to last for quite some time, Is improving their quality asking too much?

3. From what I've been reading, the best ways to clean and improve my captured videos is using either VirtualDub or AviSynth. Is the one better than the other? As far as I understand, The most commonly used method to do the cleaning part is using AviSynth + Convolution3D filters. Will that be enough or should I use some other stuff too? Of course I want to get the cleanest output like everyone.

4. I'm a bit confused about the interlacing/progressive part. I've been reading some stuff about needing to convert my captured DV files to progressive, in order to use AviSynth/VirtualDub to do the cleaning. Is that true?
If I move to progressive, then will my TV sets, which aren't LCD/Plasma, be able to playback the results? I guess they should because I can play DVD movies just fine right? But then again - is that the best option quality-wise?


As you can see most of my questions are in the cleaning part. I still haven't reached the MPEG-2 encoding part, but I have some more experience in that field, so that should be easier.
Like everyone else - I want to get the best quality I can with the hardware I have. I haven't got more money to spend on a TBC(the advc was expensive enough), but as far as I've read the cleaning process could do quite a nice job.

I would truly appreciate your help, as I am a bit 'stucked'.

With Regards,
Guy.

[Barnabas]

Hi,

You are correct, using WinDV will yield the same quality as scenalyzer does

I do a great deal of VHS to DVD work, and have a Canopus ADVD-55. I suggest you stick with VirtualDub as it's MUCH easier to use than AVIsynth.

For filtering, there are tons of great filters for both, but my favorite filter for VHS to DVD work is the DotCrawl Comb Filter by Scott Elliott. I am amazed at just how much better things like text on VHS tapes look after using this filter. It also removes noise in the process, so may be all you need.

Keep in mind that you need a VERY good VCR, something like the JVC's that have digipure. They have digital noise reduction that eliminates the bending on things like poles and vertical edges. The same circuit also provides TBC

You don't need to de-interlace to use VirtualDub with filters.

A plasma set can also play interlaced video.

[redwudz]

1. The transfer program you use to get the video into the computer really makes no difference. It's just a data transfer. I use WinDV myself. Scenalyzer may have a few added features if you need them.

2. A TBC or other stabilizers may help with VHS tape conversions to DV. Filtering may make the video look better or change it, but any filtering usually takes something out, sometimes it's quality. But if it looks better to you than when you started, then that's an improvement.

3. VD and my editor of choice, VD Mod, both work well with DV. You do need a DV codec installed for VD. I don't use AVIsynth often, but I have used it usually as a add on for programs like VD. It can edit, but it's not really an editor like VD.

4. I just leave it interlaced, as my final product is DVD. Some filters may work better with a progressive source, but the ones I use most often don't seem to care.

I have a ADVC-100. I transfer with WinDV, filter and edit with VirtualDub Mod as needed, save out the edited audio in WAV format and convert it to AC3 audio with ffmpeggui. I then frameserve the video from VDM to TMPGEnc encoder and encode to MPEG-2. Finally, I combine the AC3 audio and the MPEG-2 video in TMPGEnc DVD Author, make menus and burn. With frameserving, you don't need to save the edited copy on your hard drive, just send it to the encoder directly.

I use the Panasonic DV Codec with VDM. Filters used most often: Levels, White Balance, RGB. For noise, depends on how bad and what kind. Many filters are available, look to http://neuron2.net/ for starts. There's one specifically for VHS.

Using AVIsynth can speed up filtering considerably. And it does a few dozen other things. Not exactly 'User friendly', though. You might have to work up to it after some studying.

I can't afford a TBC, either, so I just make do with what I have.

With filtering, less is usually better. I use it to correct problems, not to make the video better.

[jagabo]

Many VirtualDub filters will mess up interlaced sources. One example is the Blur filter. The workaround for this is to use the Deinterlace -> Unfold Fields Side-by-side before Blur, then Deinterlace -> Fold Side-by-side Fields Together after the blur. This would prevent co-mingling of the two fields.

So unfold the fields before using filters that isn't interlace-aware, apply the filter(s) and then fold them back together.

Some filters have the option of treating the video as interlaced -- Resize, for example. Those don't need to be unfolded.

The unfold-filter-refold process isn't perfect but it's the only real option for some filters and interlaced video in VirtualDub.

[Z-C]

Wow! I didn't think I'd get replies so soon. Thank you!

You sure have helped, yet I'm still somewhat confused about certain issues.

Both of you said to stay with interlace, yet jagabo said to deinterlace. From what I've been reading deinterlacing is really a bad idea, but how can I make sure the filters I use don't require deinterlacing?

About having a highend VCR - I'm afraid I don't have more money to spend on this setup. I want to squeeze the best with what I have. However, most of my analog media is on small hi8 tapes from my Sony camcorder, so a VHS tapes VCR wouldn't do much good anyway. However, is there any chance my camcorder has some kind of filtering? it used to be a highend analog unit.

Regarding Vdub and Avisynth. I know Vdub is easier, but I'm a pretty fast learner and I don't mind using Avisynth - if that would really make things look better. So just to make sure - are you offering Vdub for simplicity or for quality?

Regarding the same issue - Is there any guide explaining Vdub/AviSynth and it's filters for the purpose of cleaning captured analog media? I was only able to find one which was, to say the least, very poor.

redwudz: What you've said really concerns me now. The raw DV footage I get when capturing is not satisfactory. So I was hoping to get a somewhat better quality in the end results. How do you know how many filters to use? is the way to do it is simply by trial and error? do you try to apply several filters separately until you get satisfactory results?

Thank you all again for your useful help. I hope you could help me with those developments, and I would also be happy to hear more comments on my first questions.

With Regards,
Guy.

[Barnabas]

Z-C,

If the filters you try won't work with interlacing, you will know Right away.

Since most of your stuff is on hi8 tapes, your Sony camcorder should be ok.

The only really good deinterlacing plug-ins cost a ton of money. If the source is interlaced, leave it interlaced.

As there are tons of filters for VirtualDub and AVISynth, no guide could cover them all. Some filters do come with a guide, others, nothing, some are not even in english.

While I do use AVIsynth from time to time, I MUCH prefer VirtualDub or VirtualDubMod.

It IS possible to improve your video a bit, but don't expect miracles. Since you need to encode to MPEG-2, encoders such as TMPGenc has lots of filters you can use too. Filters are indeed trial and error, so it's best to be able to preview them before you spends hours encoding.

[Scorpion King]

I use VirtualDub. About adding filters I must agree with those above. Filters can make a poor video somewhat better but used improperly filters can also hurt it. One method I use is to select a frame, write down the frame number, copy that unaltered frame to the clipboard then into my photo editor program. There I can zoom and pan the image and decide what I want to try and do with it. I then add a selected filter in VirtualDub, adjust as I think, then copy that filtered frame out to my photo editor program and compare it to the original frame side by side. Once I'm satified I have my desired filters adjusted properly I will select about a minute worth of video and save it out to view full screen. I find it very difficult adjusting and arranging filters and this method seems to work best for me, although it is kinda time comsumming. I've tried AVISynth in the past.

[jagabo]

Both of you said to stay with interlace, yet jagabo said to deinterlace. From what I've been reading deinterlacing is really a bad idea, but how can I make sure the filters I use don't require deinterlacing?

What I recommended isn't really deinterlacing the video. It's simply peeling the two pictures apart, filtering them separately, then putting them back together again.

Here's an interlaced frame of video:



When you watch this on TV you don't see both fields at the same time. First you see the field with the red 1. 1/60 of a second later the red 1 has faded away and you see the green 2.

Using the blur filter as an example, if you simply blur this interlaced frame the two fields will blend with each other:



When you watch this on TV, instead of seeing a red 1 followed by a green 2, you will see both the red 1 and green 2 at the same time, for two fields in a row.

VirtualDub's Unfold Fields takes the interlaced frame, separates the two pictures, and places them side by side (note that videohelp may shrink this image because it's very wide, click on it to see it at the original size, it's half as high and twice as wide as the earlier interlaced image):



Now if you blur this image and fold the fields back together you get this:



When you watch this on TV you will see a blurry red 1, then 1/60 of a second later you will see a blurry green 2.

Regarding Vdub and Avisynth. I know Vdub is easier, but I'm a pretty fast learner and I don't mind using Avisynth - if that would really make things look better. So just to make sure - are you offering Vdub for simplicity or for quality?

AVISynth requires that you create script files. These are plain text files with lists of commands (filters). Some commands are easy, some are more difficult -- you'll have to specify several arguments to get what you want. Most AVISynth filters can work in a YUV colorspace (native video colorspace). VirtualDub filters only work in the RGB colorspace. When you convert from YUV to RGB (and back again for your final MPEG2 compression) you will get a small loss of color precision. This loss is usually not noticeable but there are some colorspace converters that don't do a good job.

The raw DV footage I get when capturing is not satisfactory.

What exactly is wrong with it? Some things are fixable (tint, brightness, contrast), some things are partly fixable (noise), some things are not fixable (flagging, TBC errors). One thing to keep in mind: video on the computer usually looks darker than video on a TV. This is because computer monitors have very different gamma (brightness) than televisions.

[Z-C]

Thank you again for your replies.

Is there any way to check if my sony camcorder offers any kind of playbacking improvement filters? or there's no chance for that to happen?
All the guides I found refer to JVCs VCRs.

Now let me get this straight - To make things short - the Vdub filters can do the work just as good as AviSynth, with less headache?

Barnabas: Could you explain why you prefer Vdub? just to understand it's benefits better.

Scorpion King: Your method sounds really nice. However, is taking 1 frame/1 minute clip really sufficient to make filter decisions for an entire movie which can even get to 2-3 hours long?

I also understand what you guys say about 'no guide can cover all the filters'. But where can I get started then? I guess there are numerous of more commonly used filters for captured analog media restoration, right?

One more thing - when I use filters I reencode the movie? If so, am I better off encoding the movie to MPEG 2 when using filters? or should I have 2 different encoding processes made? one for restoration and the other for the end result - the mpeg 2 compression.

EDIT: jagabo I only saw your reply after posting my comment.
Thanks for the clear explanation with pictures - it really helped!
So basically I get a blurred interlaced output in the end? but is that modification worth being able to use certain filters? Please correct me if I'm wrong, but it seems to me that the blurry image will only diminish overall quality.

I also appreciate the AviSynth explanation. if that's the only difference then I really should use Vdub? I understand that the quality difference won't be really noticeable if it even exists?

With Regards,
Guy.

[Scorpion King]

is taking 1 frame/1 minute clip really sufficient to make filter decisions for an entire movie which can even get to 2-3 hours long?

It is when the entire video looks like crap

[Barnabas]

I like VirtualDub better for MANY reasons.

1: It's a visual GUI
2: Tons and Tons of features
3: Tons of great filters
4: Ability to frameserve

For what you want to do, here is the sequence you will probably do.

1: Capture to DV avi with your Canopus
2: Open your DV file in VirtualDub
3: Apply and test various filters in VirtualDub
4: Frameserve to your encoder
5: Encode to MPEG-2
6: Author your encoded video
7: Burn to DVD
8: Make some popcorn and enjoy your cleaned video!

[jagabo]

Some filters you'll want to consider in VirtualDub (AVISynth has similar filters):

Levels (adjust brightness, contrast, gamma)
Red Green Blue Adjustment (adjust colors as RGB values)
HSV Adjust (adjust colors as HSV values)
VHS (adjust misaligned color channels, other color problems)

In addition there are many spacial and temporal noise filters, sharpening filters, etc that you might consider. But I usually don't care for noise filtering with video. If you filter enough to make a difference people start looking like they're made of plastic. Many people oversharpen -- you end up with halos around all sharp edges.

[JohnnyMalaria]

Our Enosoft DV Processor will let you do some processing of the captured DV video while it is being captured in real time. In particular, you can adjust:

Brightness
Contrast
Color
Phase (a.k.a tint/hue)
Individual Cr and Cb (very useful for correcting white balance problems. e.g., indoor shots looking too yellow)

It features a real-time Vectorscope and Waveform Display along with a split screen option to help make the adjustments.

You can use it to do the capture, too. Simply choose the input device (i.e., your Canopus) and capture to AVI. While it is capturing, you can adjust all of the above in real time.

It works in the same color space as the DV is stored (YCrCb) and, therefore, you can correct some things more readily this way than with tradition RGB filters.

Once captured, the file can be opened in Virtual Dub or whatever for additional work if needed.

Right now, it's beta and, therefore, it's free(!)

You must have at least Windows XP and a processor with SSE2 support.

Click the highlighted link at the start of this message for more info. (If it isn't highlighted, go here: http://www.enosoft.net/enosoftdvprocessor.html )

If you do try it and find any problems with it, please do let us know by posting to our Users Group or by emailing me (john AT enosoft.net)

[Z-C]

Thank you again for your help. I will go the Vdub way and post some results here for further guidance(if needed) here.

Barnabas: That's a great schematic, thank you! One thing isn't clear yet: when I apply filters in Vdub, do I actually reencode? If so, isn't doing that, and then encoding to mpeg2 too much?

jagabo: I appreciate your examples, these would get me started! are these built in filters or do I need to download them? If so then where?

JohnnyMalaria: Sounds good, I will try that! But it actually works on the DV files that are being transferred right? How does it work? does it reencode the files on-the-fly?
I don't want to reencode my files too much, that might cause quality loss.

EDIT:
Another issue I didn't bring up until now, because I had enough other questions first , is regarding something I keep reading about NTSC problems.
It seems that encoding with NTSC is creating a lot of issues. The biggest issue I saw was concerning DV NTSC which works on YUV- 4:1:1 instead of 4:2:0. I heard this is the main reason why DV is considered a bad capture.

Are all these problems solved when working in PAL(like I am)? Maybe I had the wrong impression but it seems that PAL is superior for these type of projects.

With Regards,
Guy.

[Caple]

Barnabas: That's a great schematic, thank you! One thing isn't clear yet: when I apply filters in Vdub, do I actually reencode? If so, isn't doing that, and then encoding to mpeg2 too much?

You don't reencode if you frameserve it. The filters are applied as you mpg encoder does the job.

[JohnnyMalaria]

JohnnyMalaria: Sounds good, I will try that! But it actually works on the DV files that are being transferred right? How does it work? does it reencode the files on-the-fly?
I don't want to reencode my files too much, that might cause quality loss.

Yes, it works on the DV as it is being transferred. The video is never complete decoded and re-encoded. For adjustments to things such as brightness, we are able to adjust parts of the encoded signal that have no effect on the quality. (Technically, we work in the DCT domain and simply change the DC coefficient of the DCT blocks). A lot of DV "generation loss" comes about because the video is fully decoded (often to RGB) and then re-encoded. The quality of the DCT and IDCT algorithms plays a crucial role here as does the encoding algorithm. The DV specification leaves a lot of room for implementation differences. The encoding of the DCT data to compressed bitstreams is where most of the variation between implementations occurs. Some are very fast but very inefficient. Ours is very rigorous - ensuring the maximum efficiency of the compression. We can afford to be more rigorous than others since we save a lot of processing time elsewhere. When ever possible, our processor avoids any of the re-encoding. For brightness, contrast, phase and chroma adjustments, we never decode beyond the DCT stage. (That also saves a *huge* amount of processing time.) We also work exclusively in the native DV "YUYV"-type color space.

Another issue I didn't bring up until now, because I had enough other questions first , is regarding something I keep reading about NTSC problems.
It seems that encoding with NTSC is creating a lot of issues. The biggest issue I saw was concerning DV NTSC which works on YUV- 4:1:1 instead of 4:2:0. I heard this is the main reason why DV is considered a bad capture.

It depends.

The 4:1:1 means that every line of pixels is sampled every fourth pixel to get the color.
The 4:2:0 means that every other line of pixels is sampled every 2 pixels to get the color.

Or - for 4:2:0 - the color information in half of the lines is never even recorded.

As long as your final video remains in the same sampling space, typically things are okay. It's when you convert from, for example, 4:2:0 to 4:1:1 that things can go awry since you have to re-invent color information that isn't there.

[Z-C]

Great explanations guys! thanks again

After understanding much more of the theory behind things, thanks to your guidance, I think I am ready to move on to the implementation part.

I have started out with one of my more recent tapes(3 years old hi8 tape), which is actually already at decent quality.
I've been reading about the types of errors and solutions(filters), yet there are some quality diminishing errors, that I don't even know how to name, therefore I'm surely unable to find a right fitler for.

I've attached a picture from the clip I captured, just to make sure with you guys that I'm on the right path.
As you can see there are severe interlacing lines. That's because I'm viewing with Vdub right?(on powerdvd it looks good - no lines).
The second issue is the distorted bar on the bottom of the screen. is that a timing error that I need to crop, or is that the area I will not see on the TV tube. Again I will say - This is from a hi8 tape I filmed and not from an S/VHS source.

Third and most disturbing issue, is the fact that the movie has a lot of artifacts that move when playing. You can also see them in this picture. Is that what you refer as noise? or is it something else?

[Barnabas]

!:Correct, those are interlacing lines. You won't have them in Power DVD

2: All video tapes will display that "distorted bar". It's best to mask it, although, if you leave it in, the overscan on TV's will cover it.

3: Tape DOES have much more noise, that's why it's a good idea to apply some filtering.

You don't show if you are in the US or not. As long as you are capturing to NTSC, use the DotCrawl Comb Filter. I always use it for tape to DVD conversions.

[Z-C]

1. What will happen if I get an LCD/Plasma TV set? Most likely I will get one in a few years. will the movies display those interlace lines? or did the makers took that into consideration and interlaced media will be displayed correctly?

2. If it's the overscan then I'm not concerned. But why spend time masking it if it's unseen?

3. Regarding the noise - is the DotCrawl Comb filter intended for noise reduction?
I am actually using europe PAL system and not NTSC. will that filter work then? or do I need something special for PAL?

I apologize if my questions seem to repeat themselves - but they really do not. I do feel that I'm making a great progress with your help. and in the meantime - I'm trying to read as much as I can too.

With Regards,
Guy.

[lordsmurf]

1. They should have deinterlace filters that will play the content much cleaner than anything you could do in software on your own.

2. Because all that noise takes up bitrate.

3. Comb filters in most modern components using s-video connections reduce or eliminate dot crawl pretty well.

[Barnabas]

A LCD/Plasma TV set will play interlaced ok. I have a 27 inch LCD HDTV in my bedroom, and it plays interlaced stuff fine.

I would mask it out if it was me, as it might be visible on non CRT TV's. Also allows for better encoding. Cleaning up the video before encoding both speeds up encoding time, as well as getting you better compression.

You won't be able to use my favorite filter then since you are in PAL land. My 2nd choice would be the "nonlinear Smoothing loss-free 3D" filter. It's in Russian, but you only need to hit configure each time to change the advance through the different levels. Sure would be nice is someone here who also speaks Russian would translate what the text of the 4 different levels.

Go here for it and tons of other great filters:

http://www.thedeemon.com/VirtualDubFilters

[JohnnyMalaria]

If you are going to watch your compiled DVDs on a TV, why not leave the interlacing alone?

[jagabo]

A dot crawl comb filter addresses a specific issue with NTSC video. It isn't a general noise reduction filter.

http://en.wikipedia.org/wiki/Dot_crawl

Some VirtualDub filtering examples:

http://www.micahworld.com/doramatraders/dvdtools/

[JohnnyMalaria]

A dot crawl comb filter addresses a specific issue with NTSC video. It isn't a general noise reduction filter.

http://en.wikipedia.org/wiki/Dot_crawl

Some VirtualDub filtering examples:

http://www.micahworld.com/doramatraders/dvdtools/

But PAL video can also suffer from the same problem (I used to see it a lot with my PAL equipment).

I understand that a hardware comb filter would likely be NTSC only or PAL only (since the chroma carrier frequencies are so different), but for a capture YUV frame, wouldn't the same processing work equally well on both?

[Z-C]

I'm convinced. I'll mask that bar at the bottom.

I got more confused with the noise part though. Dot crawl=noise? If so, should I use that to clear my noise or general noise reduction filters that are solely intended on cleaning noise.

I would really appreciate recommendations for noise/artifacts reduction filters. I guess you guys have tried ample of them and know which ones are the best.

I will get "nonlinear Smoothing loss-free 3D", but is that really the best one? kinda odd that no one has translated it if it's so good..

[zoobie]

Actually, Scenalyzer is free. It's just their capping program that puts on a logo. Use WinDV or similar to cap and simply import into Scenalyzer.

You'll get pretty close to native DV from noisy hi8 if you denoise and perhaps slightly sharpen. You really want to leave most of the data as it is. Save your tapes for the next generation Blu-ray coming soon...

[Z-C]

Hi again.

I've been spending the last day trying to play around with denoising filters.
As far as I saw the filters' efficiency was as follows:

1. 2d cleaner by Jim Casaburi
2. MSU denoise filter
3. NRS By antonio forana - performed really well, but it seems like the motion gets more blurred after this :/.

The rest really didn't make a difference, or messed up the video when they did:
alparysoft denoise
dynamic noise reduction
and even the nonlinear smoothing loss-free 3D.

In general, the results weren't too satisfying, but were ok. So basically I'm here asking - do you guys know of any better filters for noise reduction?

Another issue I encountered is that sometimes when trying to use hard settings the movie either gets blurred in general, or it gets blurry during motion(frame by frame it's fine, but when streaming it seems more blurry). Is this a known side effect of noise reduction? or is it looking like that because of the problematic Vdub preview and it will be alright once done and playbacked on a TV set?

Thank you guys for your help,
Guy.

[jagabo]

I use noise filtering very sparingly because you always end up losing detail as well as noise. A wool sweater may end up looking like smooth silk because all the little shading detail gets blurred away. People end up looking like like plastic manikins. Temporal noise filters also introduce temporal artifacts: smearing, trailing, creepy crawly colors...

You're better off leaving the noise and encoding at a high bitrate.

[lordsmurf]

Restoring is about making it better. Sometimes it's about picking the lesser evil between two errors. It's not about making the video perfect again, that will never happen.

[Z-C]

I don't hope to make the video perfect. I'm just trying to make it the best I can.

I agree, finding the balance is hard. However, and I don't mean to sound rude so don't take this the wrong way, many of you have dealt with with noise reduction so much, because it is such a common error. Yet, you didn't really share your filter recommendations or gave me examples of the settings you use.

Sure it varies, but I am trying to get a clue of what the mainstream is when it comes to noise reduction. I checked all filters I could find, but would trully appreciate it if you could share your own experience with the filters.

So which filter would you recommend?
and is dotcrawl good for noise reduction? or that only works for NTSC video transferred with composite?