Advice on what filters to use when converting VHS to AVI

[jagabo]

Does anyone know if one can set a time limit for capturing in VD-eg 4 hours, so that it does not continue capturing if I leave it on at night?

Capture -> Stop Conditions...

[sanlyn]

Edits & Premiere Elements: If by "edit" you mean joining a lot of clips together in a time line and adding audio, etc., Adobe should be able to save your edits as Lagarith AVI. But be careful: like VirtualDub, if you apply Adobe filters you've converted the clips to RGB. If they're already in RGB as a result of your other work, it's not a problem.

Most people would cut AVI clips from their basic capture, perform any cleanup work required, then join in an editor.

[avz10]

Most people would cut AVI clips from their basic capture, perform any cleanup work required, then join in an editor.

This is what I will basically do. I will delete duplicate scenes with VD as far as I go, as the files are very big.

Where I will need help, will be with the clean-up process. At this stage I have captured 3 VHS cassettes, many more to go.

What will be the most practical way to post short clips so that you get an idea of the quality of the videos and how many frames are adequate? Two problems that have been discussed are noise and levels.

Thanks!
Albie

[sanlyn]

You should usually post a short few seconds of the original YUY2 capture. Which sample you post depends on the problems you are asking about. The 6.avi posted earlier has a little too much contrast that could easily be fixed in Avisynth. That avi looks much better and cleaner than previous captures and doesn't have a lot of noise. The problems I see in those two image sequences are the horizontal streaks (some call them "rips"). Most of these can't be repaired, at least not easily or completely. I'll explain:

You can open a YUY2 capture in VirtualDub directly without an Avisynth script or other processing, for purposes of reviewing and analyzing the images. Opening a YUY2 capture in this manner will convert to RGB32 with a Rec601 standard-definiton color matrix (which is the standard for PAL and NTSC), but your purpose is to simply review the video rather than save it as a new AVI. One thing you can look at is the way the interlaced fields are related and how noise (such as rips) behave. With the video in VirtualDub, load the built-in "bob doubler" filter set as Top Field First. Bob() is a simple and not very accomplished deinterlace routine that's easy to use to check various aspects. With that filter loaded, view your video in the "Results" window. You can see how the rips vary from field to field; note how the rips extend over several consecutive fields. Rips can be repaired in several ways, but if they extend over several fields or frames it becomes very difficult to fix. Two bob'd fields equal one full interlaced frame: a rip that extends over 4 fields will be seen in 2 frames during play. The more fields and frames are involved, the more likely it is that you'll simply have to live with the defect rather than create odd artifacts and distortions that will look worse.

Another technique that you can use in this review is the ColorTools histogram. Histograms can help identify color and levels problems rather specifically. ColorTools is an RGB histogram showing how the colors/levels look when displayed on an RGB device (PC monitors and TVs). Avisynth has some histograms that show how visual data is stored in YUV; these histograms and graphs have to be loaded with a script. The most commonly used histogram is the "Levels" histogram that works only in YV12:

AviSource("whatever source.avi")
ConvertToYV12(interlaced = true)
Histogram(mode="Levels")

The histogram will appear within the loaded image, on the right-hand side of the picture. That histogram has darkened areas on the left and right sides; the luma (white) component should stay inside those boundaries for standard RGB 16-240 output. Sometimes there could be a small "peak" at the left-hand end that lies in the left-hand colored area -- that little peak is usually the RGB 0 black borders, which can be ignored. If there is a lot of white data that creeps into the left or right border area, it would indicate too much contrast or over-saturation.

I usually use the levels histogram to set basic level adjustments. It's best to do that in YUV first. Once a video is converted to RGB display, the YUV data is stretched or expanded to show how those same elements will display in an RGB device. The RGB histogram will show the same problems, but with luma and all three colors showing as separate graphs.

The 6.avi clip looks OK except for the problems mentioned. The first images are rather green, the second is a little red. Whther you want to correct those separately is up to you -- but as you are aware, there is great variations between scenes and between clips. Correcting each one-by-one would take forever. As was mentioned earlier, most viewers won't be so demanding. The only time you might want to treat a specific scene separately is when that scene is too wildly ugly or strange to your taste. But most of the time, home videos will be acceptable without a great deal of correction. In this case, the images in 6.avi could stand a slight lowering of contrast or saturation with a statement like this:

ColorYUV(cont_y=-20)

That statement uses a negative value for contrast in the "Y" channel. Negative values would "shrink" the luma graph toward the middle. Positive values would increase contrast, or "stretch" the luma component at each side in a histogram. The "gamma_y" settings brighten or darken the middle portion, so one could make the image look brighter or darker without much effect at the darkest and brightest extremes. The best way to understand how filters like ColorYUV and Tweak are working is to review their Avisynth documentation and use a histogram to view the effects of different settings. It wouldn't take long to get an idea of how those settings work. Using them is the best way to understand their effects.

Sometimes full corrections can't be made, especially with home videos and VHS. You just do the best you can.

[jagabo]

If you're having problems understanding how ColorYUV() works you can use the attached animated script. Open the script in VirtualDub and you'll see the effect ColorYUV() has on the picture.



At the bottom is the image being processed -- a series of vertical bars of different intensity, dark to bright. At the top is the luma graph created by Historgam(). As you scrub around in VirtualDub you'll see the shape of the graph and the brightness of the bars change. Modify the script to change off_y or gamma_y instead of cont_y.

[2Bdecided]

I will delete duplicate scenes with VD as far as I go, as the files are very big.

...but you know, then you might be left with specific bits of damage on the copy you kept, which you then need to clean/fix or accept, while the copy you deleted might not have had that problem at that specific point.

Even if you don't go for the whole "synchronise three copies" median/average restoration, it's worth keeping the other copies around in case you find damage on one version that can be easily and quickly fixed by copying frames over from another copy.


If you're short of space, the obvious thing to do is to work on a few tapes at a time. Or buy another HDD. It's always useful to have one as a backup anyway. You should avoid the situation where you only have one copy of something you've spent ages working on.

Cheers,
David.

[avz10]

Apologies for not posting earlier. I have had a busy week, with working (to bring in some money) and capturing.

I have progressed from 1986 to 2001 and have edited a bit with VD-especially deleting some errors as well as some of the duplicate scenes. I have also been able to sort the clips/videos in years, but a lot of work still needs to be done. But, this is still a work in progress. Hopefully, by next weekend, most of the capturing would have been done.

With the capturing and editing, I have been able to create a number of shorter files. This is very important, as most of these files have video of equal quality-this means I can use AviSynth with different settings on most of the files.

This is where I need help.

You should usually post a short few seconds of the original YUY2 capture.

Have you got any idea what number of frames would be of help?

Another technique that you can use in this review is the ColorTools histogram. Histograms can help identify color and levels problems rather specifically.

I am very familiar with levels in Photoshop, but have not used color histograms. At least the basic levels can help a lot. I will need to learn about the color histograms.

Jagabo, I have difficulty opening the script. This is what I have changed it to:

function AnimateColorYUV("E:\Videos\1 Projek\4.avi")
{
#    ColorYUV("E:\Videos\1 Projek\4.avi",off_y=value).Subtitle("off_y = "+string(value))
#    ColorYUV("E:\Videos\1 Projek\4.avi",cont_y=value).Subtitle("cont_y = "+string(value))
    ColorYUV("E:\Videos\1 Projek\4.avi",gamma_y=value).Subtitle("gamma_y = "+string(value))
}

ImageSource("bars.png", start=0, end=512)
ConvertToYUY2(matrix="PC.601")
Animate(0,512, "AnimateColorYUV", -255, 255)
TurnRight().Histogram().TurnLeft()
ConvertToRGB(matrix="PC.601")

If I use script, VD crashes.

2Bdecided thanks for the advice. Due to the space problems I currently have, I need to buy another external hard drive ASAP. I cannot lose all these hours of work!

So, at this stage, I need to be advised on the size/number of frames I can post for your help. I also need advice if these need to be sharpened and/or de-interlased or what other tools might help.

Thanks
Albie

[jagabo]

That script makes no sense. You should have run it exactly as it was. Just to see the effect of changing off_y, gain_y, and gamma_y with ColorYUV(). Make sure that bars.png is in the same folder as the script.

If you want to see the effect it has on your video just replace the ImageSource() line of the original with AviSource("E:\Videos\1 Projek\4.avi").

[sanlyn]

Well...this will be rather a lengthy reply. Working with histograms seems difficult at first, but once you get into it they make a lot of sense.

2Bdecided thanks for the advice. Due to the space problems I currently have, I need to buy another external hard drive ASAP. I cannot lose all these hours of work!

People manage workflow in different ways. Many would have one drive for original captures and a second drive for work files that have been filtered and edited. From the capture drive, cut out sequences in small groups and transfer them to the PC for processing. Then transfer the finished working clips to the second drive and grab another group of clips from the captures for processing on the PC.

So, at this stage, I need to be advised on the size/number of frames I can post for your help. I also need advice if these need to be sharpened and/or de-interlased or what other tools might help.

PAL video at 25 FPS = 25 frames for 1 second of play. A few seconds of a problem sample are usually all one needs. The size of a few seconds of YUY2 clip will vary due to image details, motion, etc. The size limit for forum posts is 100MB. Usually you won't need 100MB of sample video.

Filtering and cleanup depend on the individual clips. The samples you posted in 4.avi and 6.avi would not require a great deal of work. Home-made VHS tapes are usually interlaced. DVD is usually interlaced as well. Modern playback hardware can handle interlaced, progressive, and progressive with pulldown. However deinterlacing does involve some level of loss, and deinterlacers vary in quality. QTGMC is the best to date. But proper deinterlacing also takes time. It's usually done because certain conditions of damage or other problems require it. A deinterelaced video can be re-interlaced if necessary. Interlace/unblending/restoring frame rates, etc., depend on individual conditions. It isn't always necessary. Many filters work best only with progressive video; one way of handling it without deinterlacing is to use SeparateFields() for processing, then Weave() after processing to restore the original field structure. Whether any of this is required depends, again, on individual conditions.

Sharpening also depends on individual cases. Strong denoising can have softening effects, in which case sharpening can enhance the remaining detail. Sharpening is usually one of the last steps. Sharpening before denoising is not recommended; it would sharpen noise and defects with other elements. The same is true for resizing, although I wouldn't see the need for resizing here. Offhand I don't see that the 4.avi and 6.avi samples need much sharpening.

Another technique that you can use in this review is the ColorTools histogram. Histograms can help identify color and levels problems rather specifically.

I am very familiar with levels in Photoshop, but have not used color histograms. At least the basic levels can help a lot. I will need to learn about the color histograms.

If you look at the levels histogram that comes with Photoshop, you'll note that the default histogram view is a graph of an average of all elements in the image: luminance + red + green + blue. The histogram controls have a down-arrow that lets you choose to see individual luma and color channels. For example, looking at the red channel individually can help identify problems such as red over saturation.

Almost all histograms work in the same way. Dark values are graphed at the left, bright values at the right. The histogram image shown by jagabo shows darks at the lower left, brights at upper right. When the markings on that histogram creep into the colored borders at the top and bottom, it indicates that those values lie outside the desired RGB 16-240 range. Some objects such as RGB-zero black borders will appear in the colored area at the lower left; black borders are normal and acceptable. IF you think those borders are confusing your view of more important elements, you can temporarily crop borders to obtain a better view. Remember that cropping has restrictions that are documented in Avisynth's article on the Crop() filter. Cropping in this way is a temporary measure for analyzing the image -- borders can be restored later.

The internet has many Photoshop and photo camera tutorials about histograms. There are similar tutorials for advanced controls in After Effects and Premiere. I recommend the two tutorials below because they illustrate with images and histograms the kinds of analysis that can be performed. It explains dark and bright clipping, tone and contrast, and other issues. This site is about camera histograms -- but remember that they work like those in Photoshop, Avisynth, VirtualDub and other apps. They deliver the same information. The basic principles of levels and color work are the same for video and photos. Different histograms and vectorscopes might display with different layouts, but they all say the same things:
Part 1: http://www.cambridgeincolour.com/tutorials/histograms1.htm
Part 2: http://www.cambridgeincolour.com/tutorials/histograms2.htm

The image below is from frame 80 of your original 4.avi sample. It's a vast improvement over earlier captures. A green arrow at the upper right indicates the green flutter across the top border. Other than this noise, most viewers wouldn't see many problems -- and even the green stripe or most of it might be hidden by TV overscan. But if you want to persue some cleanup, it's up to you. I'll provide a sample of cleanup later.

[Attachment 18871 - Click to enlarge]

Meanwhile, below are two Avisynth "Levels" histograms of frame 80. The histogram shows luma (top), the U channel (middle), and the V channel (bottom). Dark values are at left, brights at right in each of the two charts. The white arrow in the left-hand image points to dark values that lie below RGB 16 in the colored border. In this case it's not a problem: those blacks are the black border. If the borders are cropped away (histogram on the right), that colored area is cleaner except for a tiny left=hand "peak" -- it happens to belong to the black borders in the words "Merry Christmas". Overall, then, this histogram indicates an acceptable range of RGB 16-240 values.

[Attachment 18872 - Click to enlarge]

Below, left: A YUV vectorscope of the same frame (with borders cropped off). The colored area on each side is to0 be avoided. There is some slight activity on the lower left border -- again, that's from the lettering in the video. The only "problem" this vectorscope indicates is that most of the vdata lies in the middle area; the image is a little bright and has somewhat high black levels. But overall it would look quite normal to most viewers.
Below, right: an RGB histogram using ColorTools. Here you see all three color channels and the average luma. Again, the darks at the middle-left are a bit high (high gamma), but it looks OK. Blue is a bit overdone (the blue in the shirt has a slight "glow") and flesh tones are a bit green (elevated green channel in the RGB chart).

[Attachment 18873 - Click to enlarge]

[sanlyn]

An example of cleaning the green stripe at the top border. There are still some mild greenish "blips" along the top afterwards, but that was almost erased with cnr2. This cleanup was done in Avisynth with chubbyrain2, smoothuv(), and cnr2.

The script opens the AVI and converts to YV12 for the plugins. It works with no deinterlace. The first step is to separate the interlaced fields. SeparateFields() creates a new clip that is named "a" with the statement "a = last". The frame size of this new "a" clip of individual fields is 720x240. After the "a" version is filtered, the bottom 200 pixels are cropped away, because you only want to keep the top 40 pixels. This filtered and cropped version of the clip is named "b". The full-field version "a" is then overlayed with the filtered and cropped "b" version.

AviSource("path to video\4.avi")
ConvertToYV12(interlaced=true)
assumeTFF().SeparateFields
a=last

# --- Tell Avisynth to work with the "a" version -----
a
chubbyrain2()
smoothuv(radius=7)
Cnr2("xxx",4,5,255)
crop(0,0,0,-200,true)

# --- Name the filtered and cropped clip as the "b" clip ----
b=last

# --- overlay the original "a" clip with clean border from cropped "b" ----
overlay(a,b)

# --- restore the original interlaced structure ------
# ---  and clean up the black borders a little  ------
weave()
Crop(16,0,-8,-8).AddBorders(12,0,12,8)

# --- return the results of the last step -----
return last

There was no attempt to fix other problems, such as slight chroma bleed and the horizontal rips. Those streaks would require much more work; most of them can't be repaired because they overlap too many frames.

Working with the greenish cast and the somewhat pale look of the image is optional. In VirtualDub I used ColorMill, the contrast/brightness filter, and gradation curves. All corrections were mild, none were extravagant. You don't want to pump contrast here -- as this scene was apparently photographed in subdued light and would look odd if it were "juiced". The encoder was TMPGenc Plus 2.5, with audio converted to AC3 with TMPGenc MPEG Editor. A pixel sampler (csamp.exe) and the ColorTools histogram were used to take pixels readings and help determine the corrections desired.

Below, top image: filtered version with cleaned top border, before color work.
Below, bottom image: filtered version after color work.

[Attachment 18874 - Click to enlarge]

[Attachment 18875 - Click to enlarge]

An RGB histogram of the final color version. At the far right you can see that Red and Blue are slightly clipped in the original. In this case, what was clipped doesn't appear to be very important.

[Attachment 18876 - Click to enlarge]

This sort of thing takes time and patience. How much work and learning you want to do depends on what you want. Overall, the original clip doesn't have much noise and would be acceptable to most viewers.

[2Bdecided]

2Bdecided thanks for the advice. Due to the space problems I currently have, I need to buy another external hard drive ASAP. I cannot lose all these hours of work!

Good move.

So, at this stage, I need to be advised on the size/number of frames I can post for your help. I also need advice if these need to be sharpened and/or de-interlased or what other tools might help.

The videohelp upload limit is 100MB per file. That's more than enough, and more than most people will bother to download. A couple of seconds of a given "problem" is normally fine.

Cheers,
David.

[avz10]

Thanks for the replies. I am still capturing videos and one can see a clear difference between some camcorders and also between direct captures from VHS and VHS-C cassettes.

I have numerous clips now and I can actually cut them to use a specific filter on a part of a clip, e.g. if a scene is very dark. But, with regards to the typical video streaks, I do not think I will do anything about them. There are just too many.

I just want to stay to some basics, like working with the histograms, look at noise and then the difficult question for me is if I should de-interlace. The step after the editing will be to encode to mpeg and create dvd's, but as stated before, I will keep the original AVI files.

[jagabo]

Since your end product is DVD you should not deinterlace. Doing so will reduce the smoothness of motion. If you have to deinterlace for filtering purpose, do so in a way that lets you re-interlace. Ie, SeparateFields(), Bob(), QTGMC(), Yaddif(mode=1), etc.

[avz10]

A few more....

[2Bdecided]

I think the black level is fractionally too low in most of the clips, though this is a matter of taste. I'd use something like...

levels(0,1.0,255,10,255,coring=false)

...where that value of 10 in the middle is the black level boost. You could use a bit less (e.g. 5 - very subtle) but don't use much more. It'll just bring some details out of the shadows in some clips.

Obviously some clips have completely different highlight levels - a couple of the indoor shots only use half the possible range, hence look dark. You can fix it manually, or use a BIT of HDRAGC across the whole lot. I have no great insights into that. I usually end up blending the result, e.g. ...

a=last.levels(0,1.0,255,5,255,coring=false)
b=last.hdragc()
overlay(a,b,opacity=0.5)

...but a smart person would play with the parameters properly!

o (and to a lesser extent a couple of the other clips) has horrible flickering between fields. You'll need to bob or separate fields and apply deflicker. I have no great insights into that either, though the milder examples will be handled automatically by decent denoising.


The most important thing across all of it (if you've decided not to try to fix all the tape damage) is some reasonable luma denoising and fairly aggressive chroma denoising. I would try mvdegrain2 from http://avisynth.org.ru/mvtools/mvtools.html (download the filter, see the code example near the bottom of that page). I usually add a little of the luma noise back in at the end to avoid a plastic look. It would be better to ensure only the high frequency part of the noise was added back.

The final result will look better IMO if you crop off the jagged borders and then add black to get back to 704x576. Only crop even numbers of pixels.

Cheers,
David.

[2Bdecided]

Here you go - some ideas for a general script for everything - no specifics for each clip yet (that's can be sanlyn's department )

avisource("a.avi")+avisource("b.avi")+avisource("c.avi")+avisource("d.avi")+avisource("e.avi")+avisource("g.avi")+avisource("h.avi")+avisource("i.avi")+avisource("j.avi")+avisource("k.avi")+avisource("l C.avi")+avisource("m C.avi")+avisource("n.avi")+avisource("o.avi")
assumetff()
bob(0.0, 1.0) # lossless (perfectly reversible) bob deinterlace
#o=last
#a=last.levels(0,1.0,255,10,250,coring=false)
#b=last.hdragc()
#overlay(a,b,opacity=0.5)

levels(0,1.0,255,10,255,coring=false) # raise black level

converttoyv12() # need YV12 for denoiser

source=last #save original

#denoiser:
backward_vec2 = source.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
backward_vec1 = source.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec1 = source.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec2 = source.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
source.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=1)

clean=last #save cleaned version
#return clean # return cleaned version to check it if required

diff1=subtract(source,clean).Blur(0.25)
diff2=diff1.blur(1.5,0)
diff3=subtract(diff1,diff2) #diff3 is high-ish frequency noise only

sharpen(0.4,0.0) # sharpen cleaned version a little

#mix high frequency noise back in
overlay(last,diff3.levels(128,1.0,255,0,127,coring=false),mode="add", opacity=0.7)
overlay(last,diff3.levels(0,1.0,127,128,255,coring=false).Invert(),mode="subtract", opacity=0.7)

#put cleaned chroma back in with warp sharpening
mergechroma(clean.aWarpSharp(depth=20.0, thresh=0.75, blurlevel=2, cm=1))


#crop junk in borders...

#very clean:
left=18
top=6
right=10
bottom=6
w=704
h=576
#OK for DVD:
left=12
top=2
right=8
bottom=0
w=704
h=576

crop(left,top,-right,-bottom)

# add equal (mod2) borders

nleft=((w-width(last))/4)*2
nright=w-width(last)-nleft
ntop=((h-height(last))/4)*2
nbottom=h-height(last)-ntop

addborders(nleft,ntop,nright,nbottom)

#return last # preview deinterlaced version if required

#re-interlace:
assumetff()
separatefields()
selectevery(4,0,3)
weave()

Just a starting point.

Any bits you don't want can be commented out.

Hope this helps.

David.

[sanlyn]

Will keep my eye on that script, 2B, thank you. MVTools (and MaskTools) are still a mystery to me, so I keep a lot of these ideas in dedicated folders. One of these days I'll figure 'em out. I'm assuming that some of that MVdegrain2 is to smooth interlace problems. Anyway, will give it a run on all.

n.avi and o.avi have some serious chroma channel problems. n.avi is the worst. I also note that most of the "disturbances" are in odd fields in all samples. Will work on some of that this morning. I suspect player misalignment.

The tapes appear to be bouncing around in the tape path. A partial cure for this is an old trick of repacking tapes a few times. Packing refers to the way the tape is wound on the reels. Likely they've been played many times over the years. If you look at the way the tape is wound on the reel you'll see many bumps and ridges. You repack a tape by fast-forwarding to the end (without playing and without pause). Then fast-rewind to the start (again, without playing and without pause). Repeat this one or two times until the tape is wound more smoothly. Then allow the tape "rest" to for a day or two. It often helps many tracking problems.

[2Bdecided]

I'm assuming that some of that MVdegrain2 is to smooth interlace problems.

Just denoising really, though it'll smooth out a variety of evils. e.g. the flickering red jumper in e.avi.

Qtgmc as a deinterlacer also does a useful bit of smoothing (but not usefully so on such ropey sources), and it's a visibly better deinterlacer which (even when re-interlacing) could in theory lead to better denoising - but I don't have time for that.

I bet MCtemporalDenoise or mvdegrain3 would prove useful, but as the OP isn't that familiar with AVIsynth I thought we'd start easy. mvdegrain2 is really easy - the one plugin, and those lines in the script, and you're there.

Cheers,
David.

[sanlyn]

MVdegarin2 worked well, even with bob(). Tried it with QTGMC as well. Difficult to tell the difference. Both had a bonus: cleaning up some edges, smoothing some of those horizontal streaks, and cleaning some subtle horizontal bars (PAL hanover bars ??) that you can see in bright areas. Thanks again.

[avz10]

Thanks for all the replies. I am still busy capturing, so there are still quite a few hours that needs to be captured.

The basic editing in VD also takes time. I use my laptop for that, connected to the external drive.
Currently, I am capturing the VHS C cassettes, with a marked difference on VD comparing to the 1st copy on a regular VHS cassette.

I think the black level is fractionally too low in most of the clips

I agree. I have found in Photoshop that it really makes a difference if the levels are properly done.

o (and to a lesser extent a couple of the other clips) has horrible flickering between fields. You'll need to bob or separate fields and apply deflicker.



n.avi and o.avi have some serious chroma channel problems. n.avi is the worst.

o and 3.avi from #1 was taken with my father in law's first camcorder- a second hand JVC. I had it serviced and the result was n.avi. Luckily, not too much was taken with that camcorder and I will edit them separately (with guidance!!)

The most important thing across all of it (if you've decided not to try to fix all the tape damage) is some reasonable luma denoising and fairly aggressive chroma denoising.

I think most of this has been answered in 2Bdecided post. But I would like to exclude most of the previous mentioned bad quality clips/tapes if we are to use a "generic script for the other videos. Tose are really the worst of them all, but percentage wise, luckily a small portion. I would appreciate a little bit of guidance here-i.e. the too much factor.

If you look at the way the tape is wound on the reel you'll see many bumps and ridges. You repack a tape by fast-forwarding to the end (without playing and without pause). Then fast-rewind to the start (again, without playing and without pause). Repeat this one or two times until the tape is wound more smoothly. Then allow the tape "rest" to for a day or two.

I am doing that before capturing, but have never allowed the tape to rest.

Thanks!!!

Albie

[sanlyn]

If you look at the way the tape is wound on the reel you'll see many bumps and ridges. You repack a tape by fast-forwarding to the end (without playing and without pause). Then fast-rewind to the start (again, without playing and without pause). Repeat this one or two times until the tape is wound more smoothly. Then allow the tape "rest" to for a day or two.

I am doing that before capturing, but have never allowed the tape to rest.

Alllowing them to sit for a while isn't really necessary, but a good idea to repack. Those tapes must be must be in horrible condition. Eek!

a.avi: This isn't a major fix, mostly minor color in Avisynth. There's not much one can do: mixed lighting and a camera's autocolor are a challenging combination. Can only give it a try. There is greenish-cyan daylight or other colored and bright light source from the left. The young woman on the left is also discolored by reflection from the large red lamp behind her. Bright highlights are burned away.

Below is frame 44 from the original, unprocessed. A YUV histogram and RGB histogram. Overall it's not too bad, but is oversaturated (red) and a little bright. The clipped highlights are indicated at the right-hand side of the histograms. Basically, contrast is a little too high. That's not difficult to fix.

[Attachment 18938 - Click to enlarge]

Frame 44 after some ColorYUV treatment. Not perfect, but mixed lighting is a monstrous chore. When reducing black levels and brightness, an image can be restored to look more natural by adjusting gamma (midtones) with little or no effect on extremes of dark and bright. It also affects one's perception of "saturation".

[Attachment 18939 - Click to enlarge]

The script below is the basic script I ran first, without further denoising. Later I ran 2Bdecided's MVdegrain2 idea, distilled from the script posted earlier without more sharpening or levels work. I used QTGMC in the posted mkv instead of bob(), but they both looked alike to me.

I know it's impossible to fix all the rips and streaks. But now and then you find one or two that are easy to fix. Here, the closeup of two elegant ladies was marred by three embarassing streaks across their hair and noses. Because there was very little movement, ReplaceFramesMC2() worked well. Had they been quickly turning their heads or talking, forget it -- it would never work. I fixed frames 17, 18, and the dark "blip" in frame 69. The other streaks crossed too many frames to be workable. Getting all the green out of their hair was a chore in YUV. Better to tweak that in VirtualDub.
Top image: frame 17 before ReplaceFramesMC2.
Lower image: after.

[Attachment 18940 - Click to enlarge]

# Avisynth plugins & Scripts:
#---ReplaceFramesMC2.avs
#---santiag.avsi
#---RemoveSpots.avs
#---awarpsharp.dll (v2)
#---DeBlock.dll
#---LSFmod.avsi

AviSource("path to video"\a.avi")

ColorYUV(cont_y=-15,off_y=-8,gamma_y=-12)
ColorYUV(cont_v=-40,cont_u=-25)
ReplaceFramesMC2(17,2)
ReplaceFramesMC2(69,1)
ConvertToYV12(interlaced=true)
santiag(1,2)
AssumeTFF().SeparateFields()
RemoveSpots()
MergeChroma(awarpSharp2(depth=30))
DeBlock()
LSFmod(defaults="slow",strength=75)
Weave()
Crop(12,8,-8,-8).AddBorders(10,8,10,8)

I saved the results to Lagarith YV12, then later tried several versions of 2BDecided's idea. In this case I used QTGMC, but bob() worked just as well. Note that this version of MVDegrain2 requires an older mvTools.dll :

#---required: mvTools.dll version 1.11.4.5, 2008
#---QTGMC.avsi, but can use bob() or yadif to deinterlace
#---This script works only in YV12

AviSource("path to video\a_open_0.avi")

assumeTFF()
QTGMC(preset="very fast",sharpness=0.6)
source=last # <--- save the original input

#--- denoiser: ---
backward_vec2 = source.MVAnalyse(isb = true, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
backward_vec1 = source.MVAnalyse(isb = true, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec1 = source.MVAnalyse(isb = false, delta = 1, pel = 2, overlap=4, sharp=1, idx = 1)
forward_vec2 = source.MVAnalyse(isb = false, delta = 2, pel = 2, overlap=4, sharp=1, idx = 1)
source.MVDegrain2(backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,idx=1)

clean=last # <- save the cleaned version
diff1=subtract(source,clean).Blur(0.25)
diff2=diff1.blur(1.5,0)
diff3=subtract(diff1,diff2) # <- "diff3" is higher frequency noise only
sharpen(0.4,0.0)            # <- sharpen cleaned version a little

#--- mix high frequency noise back in ----
overlay(last,diff3.levels(128,1.0,255,0,127,coring=false),mode="add", opacity=0.7)
overlay(last,diff3.levels(0,1.0,127,128,255,coring=false).Invert(),mode="subtract", opacity=0.7)

#--- put cleaned chroma back in with warp sharpening ---
mergechroma(clean.aWarpSharp(depth=20.0, thresh=0.75, blurlevel=2, cm=1))

#---- re-interlace: -----
assumeTFF().SeparateFields().SelectEvery(4,0,3).Weave()

[sanlyn]

o.avi: The first thing I noticed -- other than the horrible purplish red -- was how vastly different the even and odd frelds are. Each set of fields look like something from two different videos. I used SeparateFields() to capture images of how different they are:

Top: Frame 56 - Even Field. Lower: Frame 56 - Odd Field.

[Attachment 18943 - Click to enlarge]

[Attachment 18944 - Click to enlarge]

They were different enough to cause a flicker that wouldn't go away and stubborn color problems. Below is attached an o_avi_field_demo.mkv I made showing the separated frames (using bob() to expand Even and Odd fields into full frames) and the way the histogram responded. The video has been slowed down to 8 FPS.

At first I used SeparateFields() to try to make the fields look similar. This worked well, but the interlace artifacts here and elsewhere looked very annoying. So as an afterthought I used three plugins to r5epair nthe bad interlace: santiag (anti-alias script), QTGMC (deinterlace and some denoising), and Vinverse (help with combing artifacts). Rather than rewrite the script to work everyhting in the deinterlaced domain, I'm posting the entire process as-is:

# Avisynth plugins & Scripts:
#---santiag.avsi
#---QTGMC.avsi  # -< can use yadif or bob()
#---Vinverse.dll
#---awarpsharp.dll (v2)
#---LSFmod.avsi

AviSource("path to video\o.avi")
#--- try to make Even and Odd fields look alike......more or less -----
AssumeTFF().SeparateFields()
i1=last
e=i1.SelectEven().ColoRYUV(gain_y=-1,gamma_y=-5,cont_v=-150,gain_v=-1,cont_u=-100)
o=i1.SelectOdd().ColoRYUV(cont_v=-125,gain_v=3)
# --- reweave separated fields -----
Interleave(e,o)
Weave()
#--- tweak overall color balance for both fields ---
COlorYUV(cont_y=15,gamma_y=-25,cont_v=-40,cont_u=-20,off_u=-3)
Tweak(coring=false,sat=1.4)
ConvertToYV12(interlaced=true)
santiag(1,2)
# --- the next five lines are optional - repair bad interlace ---
QTGMC(preset="fast")
VInverse()
MergeChroma(awarpsharp2(depth=20))
LSFMOD()
AssumeTFF().SeparateFields().SelectEvery(4,0,3).Weave()

# --- Fix borders and output as YV12 foe encoders ----
Crop(12,8,-8,-8).AddBorders(10,8,10,8)

[2Bdecided]

I think most of this has been answered in 2Bdecided post. But I would like to exclude most of the previous mentioned bad quality clips/tapes if we are to use a "generic script for the other videos. Tose are really the worst of them all, but percentage wise, luckily a small portion. I would appreciate a little bit of guidance here-i.e. the too much factor.

IMO mvdegrain2, if you add a little of the noise back in afterwards like I did, isn't "too much" for any of your latest clips. Someone else might have a different opinion - "good" denoising is really subjective.

Even first generation PAL VHS is far noisier than any modern digital source, though I find the noise is more obvious on the PC than on a TV. For best results, burn the result of whatever AVIsynth script you try to DVD and check on a TV. It's easy enough to take all your short clips, apply one setting to all of them, then a different setting (e.g. adding more or less noise back in, using a different denoiser, etc), and then dump all the lot onto a DVD to compare.

IME when you watch that DVD, you wonder what the differences really are. Differences that look really obvious when you A/B frames on a PC seem to disappear when you just watch things one after the other on a TV. You can drive yourself mad.

You can do a split-screen effect of original vs processed, or process1 vs process2 in AVIsynth by using crop to crop each clip in half, and then stackhorizontal to put them side-by-side. Then at least you can see the difference easily - though of course just watching one after the other is more representative of normal viewing. Split screen can cause you to like improvements which are either invisible, or annoying, when applied to the whole video and watched normally (i.e. without reference to the original). Human psychology is an interesting field!

Cheers,
David.

P.S. the danger with more and more processing is a) you over do it (the result looks like plastic), and/or b) some specific undamaged bit of your footage doesn't interact well with some specific filter/operation, and the output has some weird artefacts/problems added to it that aren't in the original, and/or c) it takes so long that you never finish the job.

P.P.S. thoughts on different denoisers: camcordercolourdenoise (or whatever it's called) for virtualDUB is sometimes ideal for some camcorders. Also, low light footage from certain camcorders can look terrible after mvdegrain2 so if you're unlucky you might have to find something totally different for that. Other than that, mvdegrain2 really is a good starting point for the general noise you seem to have. Several of the other denoisers are really quite similar (essentially motion compensated selective averaging), and so will give similar results. I haven't seen any patterned noise on these new clips where neatvideo might be able to work far greater miracles than anything else (though it's always a useful option - but sometimes time consuming to get just right). If you do try it, don't use neatvideos temporal options (they just blur frames together), just the spatial ones as a gentle pre-processor for some other motional compensated denoiser. The default settings in NeatVideo are anything but gentle.

[sanlyn]

I didn't go to NeatVideo at all. I don't think the vids warrant a filter that aggressive. Camcorder Color Denosie had no effect on the areas one would hope for, nor did I see a need for it. Cnr2 did a better job on rainbowed areas, as did MCTD with Merge Chroma. ChubbyRain2 was used earlier to remove most of the green "flashing" and discoloring across the top and to scrub the border stain. I used 2BDecided's MVdegrain2 idea on some of the clips that seemed to need it, but it wasn't always necessary.

The chroma channels in n.avi are too corrupt to work with IMO. Besides....highlights are completely blown away in that clip. I believe that n.avi and o.avi was shot with the defective camera avz10 mentioned. o.avi (the swimming pool shot) I managed to salvage, if imperfrectly. k.avi is so dark there's very little data to work with, but I've been performing some Frankenstein experiments on it that just might bring it to life (pun intended).

[2Bdecided]

The chroma channels in n.avi are too corrupt to work with IMO.

n.avi? orange fish, green water? What colour do you want them to be?! It is blown out though, but what do you expect - old camcorder AGC, dark water, sunlight on the edge. The whole tape won't be like that. Once the camera is pointed mostly at some grass or stone or something it'll be fine.

k.avi is so dark there's very little data to work with

Oh you exaggerator you. I shoot worse than that today (camera gain disabled and take what I get - fix it in post). k.aci is using (almost exactly) half the luma range, but the luma isn't especially noisy, so it's easy to bring it back. Chroma is wrecked.

Most of the clips have bad interlacing artifacts

I've only watched on a PC, but I can't see any broken interlacing on any of the new clips (not sure about o.avi, but rest I'm fairly sure are OK).



I think you're pushing Albie in a really difficult direction by demonstrating shot-by-shot level correction in AVIsynth. It's easier and quicker in an NLE (especially a better NLE with decent scopes). I know you can get it closer eventually with a combination of tools including AVIsynth, but have you forgotten how many videos he has? You spent a year on one tape.

However, one really time consuming thing you're doing which I think Albie won't but I probably would is fixing the tape damage that comes out as lines on the screen. If only there was a decent automatic way to spot those really nasty multi-frame multi-line lines. The ReplaceFramesMC2 you're using is a real treat for that. Masked to only hit the bad areas, either fully auto, or with a dead simple quick manual interface, would be great. IIRC there are professional tools exactly like this, but I don't know if there are free ones.

Cheers,
David.

[sanlyn]

I'm in agreement with most of your comments.I imagi9ne ALbie will pick and choose with these routines, certain he won't try to fix hoping to demonstrate what the filters do. As for those ugly multi-frame streaks -- no way, and most of them can't be fixed at all. I've fixed a few really easy one, especially if they were embarassing. But ALbie can pick and choose here.

The color and levels in almost all except the very worst samples can be fixed in a NLE or in VirtualDub, if they need it at all. I've worked with a co9uyple of the worst. The shot with the boy and dog is hopeless, I agree. Yes, that's an orange fish and likely it's old green water. What bothered me is the red shadows. It's easy enough to make the dog black instead of reddish black, but I'm guessing the shadows are red because they're standing on reddish or yellow stone and brights are so badly clipped that some bright blue and green are gone. In any case I don't think much can be done.

k.avi is pretty dim (we've had much worse in an earlier clip), but there's a lot of nosie in it. Color and levels aren't so difficult, and the interior lighting will be a bit warmish, but -- wow, that noise. Note the background shadows: they bounce around pretty badly. Here's where I am so far, with work still remaining (frame 33, original at top, "in progress" below it) and after your MVdegrain routine was used:

[Attachment 18973 - Click to enlarge]

[Attachment 18974 - Click to enlarge]

Sometimes when I view that clip I'm thinking it wasn't in focus. Maybe. The only VDub filter here was ColorMill.

Interlace: it's rather fuzzy and noisy on playback, even after denoising with various plugins. I used QTGMC ("fast") and Vinverse to make things look smoother. The difference is subtle but noticeable. Albie can use those two lines of code if he wants, or not. I realize that coloor balance is a personal matter, but I'm hoping to show that if levels are off the charts the image can play "cleaner" with some very minor ColorYUV adjustment. Suiting those adjustments scene by scene would of course be a real chore, and I'm certain that Albie knows it; I'm mainly looking at the most horrible cases. Fortunately, most of the samples look OK.

[avz10]

I do not know how to thank everyone for all the advice!

I'm still capturing, but have now accumalated a massive amount of gigabytes! I do try to cut and delete duplicate scenes, but it is quite a job. I obviously keep the best clips. To keep all the captured videos, will just be (at this stage) too much.

I will need to subdivide the clips even further, for instance to isolate a clip of the quality of o.avi, so that I can process it with the recommended script.

I have a few questions left:

o.avi and 3.avi from #1 was taken with my father in law's first camcorder- a second hand JVC. I had it serviced and the result was n.avi.

I am posting clips after the service (before I bought my next camcorder in 1996) and want to know if more work is needed on these videos than just using 2Bdecided's script on 19 July (#46).

Secondly- After processing all the videos, is the best way to encode with HCEnc with these settings as posted here with my previous project?

And, before I encode to mpeg, do I need to change some of the settings on the original captured videos e.g. the color setting?

https://forum.videohelp.com/threads/355843-Virtualdub-filters-For-8mm-Film-That-Was-Cap...d-On-Vhs/page4
#108

---------------------------------------
                | HCenc - MPEG2 encoder - rel. 0.24.0 |
                ---------------------------------------
   
  MPEG profile@level:    MP@ML
  input:                 e:\videos\1. projek\rgb to yv12 nuutste.avs
  output:                E:\Videos\1. Projek\Welgegung mpeg2.m2v
   
                --------------------
                | encoder settings |
                --------------------
   
  profile:               BEST
  frames:                0 - 84067
  framerate:             25.000
  aspect ratio:          4:3
  chroma format:         4:2:0
  bitrate Kb/s:          6500
  max. bitrate Kb/s:     9000
  pulldown:              no
  closed gops:           yes
  VBV check:             yes
  scene change det:      yes
  interlaced:            auto, TFF
  goplen,B-pic:          15 2
  dc_precision:          10
  intraVLC:              auto
  scan method:           auto
  bias:                  0
  chapter frames:        0
  time code:             0  0  0  0
  CPU:                   SSE4 (4.2)
  priority:              idle
  SMP active:            yes
  matrix:                MPEG
  luminance gain:        no
  adaptive quantization: yes, strength 2
  deadzone quantization: auto

Using then e.g Premiere Elements to finalize the editing and add a few titles here and there?

Again, a big thanks!

[2Bdecided]

If you're talking about simple edits and just a few titles, then all but the pickiest would be happy to take the final MPEG-2 encode and re-edit that as long as the editor supports smart rendering - i.e. as long as the vast majority of footage which doesn't have titles added and doesn't have an edit within that GOP is passed through without-re-encoding. Does Premiere do that for DVD MPEG-2? I don't know - I've never used it.


If you're talking about anything more complicated, then you wouldn't normally use Premiere after HCenc. HCenc (if you choose to use it as your MPEG-2 encoder) would be the stage immediately before DVD authoring, after all clean-up, editing, titling etc. Importantly, if you intend to fix any video levels in Premiere, you really shouldn't be using DVD-ready MPEG-2 as the input.


So what should you do? Well, if you're not encoding to the final format from the AVIsynth script (i.e. you're going to do more fiddling around after AVIsynth), then ideally you need an intermediate format to save the output of the AVIsynth script to, and to subsequently load into Premiere. Sanlyn will tell you to use lossless . I sometimes use DV-AVI, but sometimes use very high bitrate MPEG. There are dedicated intermediate codecs available to buy which do a great job - near lossless, fairly efficient size (larger than DV-AVI), very fast to encode (as fast as anything; much faster than MPEG!) and very fast to decode (NLEs really fly with them as the source; much nicer than working with lossless).



I may be in a minority of one, but I don't find HCenc to be especially good for encoding home movies. That's not meant to be harsh criticism: I don't find anything free to be especially good. The praise heaped on HCenc seems appropriate when encoding commercial films, but not when encoding interlaced home movies. With blurred de-noised VHS it should be OK though.

Cheers,
David.

[avz10]

re-edit that as long as the editor supports smart rendering -  i.e. as long as the vast majority of footage which doesn't have titles  added and doesn't have an edit within that GOP is passed through  without-re-encoding. Does Premiere do that for DVD MPEG-2? I don't know -  I've never used it.

I have always used Pinnacle, and currently the latest version- Studio 15 Ultimate collection. I especially liked the storyboard view as well. Unfortunately, I could not load AVI videos with the Lagarith codec. I am also not sure if it uses smart rendering.

https://forum.videohelp.com/threads/352796-Unable-to-load-AVI-files-into-Pinnacle-15-fo...ighlight=avz10
See #25

Avz10,

I tried several types of codecs and none of them worked with Pinnacle.

Honestly better switch to another video editor like as VEGAS/POWER DIRECTOR. You will like the PD storyboard.





Claudio

Quite a lot of comments were made earlier in this post when I stated that I used PowerDirector. PD does support Smart Rendering.

So, I am at a loss here.

Regards and thanks

Albie

[sanlyn]

Using then e.g Premiere Elements to finalize the editing and add a few titles here and there

If you're talking about simple edits and just a few titles, then all but the pickiest would be happy to take the final MPEG-2 encode and re-edit that as long as the editor supports smart rendering - i.e. as long as the vast majority of footage which doesn't have titles added and doesn't have an edit within that GOP is passed through without-re-encoding. Does Premiere do that for DVD MPEG-2? I don't know - I've never used it.

It doesn't. AVI or DV-AVI.

A suggestion: TMPgenc Mastering Works 5 is a smart rendering editor, and an encoder (DVD, BluRay, AVCHD, etc), can output lossless AVI (huffyuv, Lagarith, etc.):
http://tmpgenc.pegasys-inc.com/en/product/tvmw5_spec.html#13

Has a list editing mode and a timeline editing mode for video, audo, titles, transitions, effects, color correction, etc.):
http://tmpgenc.pegasys-inc.com/en/product/tvmw5_timeline.html

Can edit AVI, standard MPEG (HCenc, TMPgenc encoders, DVD recorders), and more. Free tgrial...good for 30b days or something like that. I'm using this, HCenc, and TMPgenc Plus 2.5 for posts in all these threads, both h264 "regular" MPEG.

Still working with all your captures.