Re-Newbie Needs Startup Advice

[sanlyn]

So, you're envisioning me doing some processing on a lossless file on one machine, and then taking that file to another machine for more lossless compression. Is that the reason to use Lagarith?

Not so much, but I envision you possibly going back to that capture if it has been archived to another hard drive, or you're using a different PC (old one died, or you upgraded, or you're traveling with captures or parts of them on a portable hard drive, etc., etc.) If that level of compatibility and portability are of no concern, it's not a problem.

Post-processing: it is not a push-one-button, one-step process. You might want to make some fixes in Avisynth, save that file and then feed it into VirtualDub for something else. You might want to take that cleaned-up video, join it to pieces from another video, combine scenes or other elements into a new video, etc. etc., etc. Each of those steps would be done with lossless intermediate files that require lossless recompression. Encoding to a final delivery format is a last-step procedure. You encode to that lossy delivery format once, and only once. You can recompress lossless media with lossless compression as often as needed.

Repairing bad frames: manono covered that in the previous post, above, listing three effective methods. By the way, that sort of frame repair usually requires deinterlace or inverse telecine operations if they are to work correctly. If you think you would do that with VirtualDub or a typical consumer NLE, you're in for a rude quality-loss shock. That sort of (often essential) procedure sometimes requires colorspace changes between YV12, YUY2, RGB, and back again; the best available and least destructive tool for such purposes would be Avisynth. None of that makes VirtualDub useless; most people would use both apps, each having distinct advantages.

[boobounder]

Thanks Manono. Yes, I meant to interpolate. And I know there's a couple of spots with more than one consecutive blue screen, so special thanks for the third recommendation.

Thanks Sanlyn. I get it now. Lagarith it is.

[QUOTE=sanlyn;2299899]

Repairing bad frames: manono covered that in the previous post, above, listing three effective methods. By the way, that sort of frame repair usually requires deinterlace or inverse telecine operations if they are to work correctly. If you think you would do that with VirtualDub or a typical consumer NLE, you're in for a rude quality-loss shock. That sort of (often essential) procedure sometimes requires colorspace changes between YV12, YUY2, RGB, and back again; the best available and least destructive tool for such purposes would be Avisynth. None of that makes VirtualDub useless; most people would use both apps, each having distinct advantages.

This raises new questions

I have been playing with deinterlacing,area-based in VirtualDub (I have seen no telecining in my practice runs).

So, do I take care of interpolating the blue frames before or after I deinterlace?

And you're saying I should deinterlace in AVIsynth rather than VirtualDub? What filters would you recommend in AVIsynth?

Also, got a pointer about the colorspace changes you discuss?

[manono]

So, do I take care of interpolating the blue frames before or after I deinterlace?

And you're saying I should deinterlace in AVIsynth rather than VirtualDub?

I forgot this stuff would be interlaced. The source should be progressive for the interpolators to work well. I always do my deinterlacing in AviSynth using either Yadif if I want quick results or QTGMC if I want better quality. You could bob, interpolate, and then reinterlace, but that might make for more interpolating artifacts. Or you could single-rate deinterlace, interpolate the missing frames, and then keep it progressive. Or you could duplicate frames to replace the blue frames using AviSynth's FreezeFrame command. That'll make the replaced parts play jerky, though.

In any event, all the work requires you going through the video frame-by-frame searching out the blue frames. There is no automatic solution.

[boobounder]

There are not that many, so this would be OK.

Is there a filter that will count them or flag them so I don't miss any?

[El Supremo]

FWIW, I use a very simple HW setup for digitizing VHS - a JVC SR-VS30 dual miniDV/SuperVHS deck, and just capture VHS through the Firewire output. All digitization is therefore handled internally to the unit with very good fidelity (considering the source quality is so limited ). I see these decks are still out there on eBay. My capture software is good old Scenelyzer Live, which still works under Win7.

I'm then left with DV codec, bitrates and file sizes (13.6GB/hour) in AVI wrappers which are great both as archives and for further editing, and which have plenty of resolution for capturing whatever detail is present in VHS source material. Depending on number of consecutive blue frames, you may be able to sort them out automatically with Scenelyzer's optical scene recognition settings.

Anyway, best of luck!

All the best!

[sanlyn]

I might note for the O.P.'s benefit that multiple re-encodes are not the best idea with dirty crappy crummy noisy damaged VHS. But if you don't see that well, don't mind imbedding artifacts with VHS-to-DV capture and just want to do some extra work trying to clean them up, and aren't particularly sensitive to color problems or to losing 20% of your original color information during capture, then VHS to DV is what El Supremo recommends. I don't recommend it. VHS->DV artifact fans can take it from here.

[boobounder]

Agreed.

El Supremo: this is similar to how I did most of my tapes 10 years ago. Most of the DV files turned out watchable. The tapes I kept are mostly the ones whose digitizations didn't turn out to be very watchable. So I'm trying a more thorough approach.

[boobounder]

So, where I am right now is this I've decided to add capture software and codec to my chain:

My hardware chain is updated a little: (the original for most copies JVC HR-VP676U) VCR > composite cables > Toshiba D-KR2 > S-video and audio composite cables > Diamond VC-500 > USB > Windows 7 PC with a second, brand new, video-dedicated, SATA HD > VirtualDub using Lagarith

The resulting file will be interlaced.

It seems to me that the next step is to count the number of blue screens coming from my VCR (perhaps it's not enough to be a big problem).

Can someone recommend a filter to count/flag these in interlaced video in either VirtualDub or AVIsynth (or something else)?

And this software

[sanlyn]

Sometimes you need a fast machine to capture with Lagarith YUY2, and turn off stuff like the internet and antivirus during the capture. If after a short time you see dropped frames in VDub's readout, go for huffyuv YUY2. On my old 2GHz PC I had to do that with a tape that played as if it were wobbling on its last breath. You appear to have a dirty/damaged tape (hence, the blue frames). Try the old hack of repacking the tape -- that is, without playing the tape, wind it all the way to the end, then wind it all way to the start. You might have to do it twice. You want to get most of the humps and wrinkles out of the windings on the reel. Let the tape rest for a day to allow vinyl's natural elasticity to restore the tape's shape. Remember, repack is done without playing.

Some "auto" frame finders have been suggested in the past. The result was that after struggling with the gimmicks for weeks, the owner eventually went back to manual means. But other readers might know of new developments. Even if found, they remain to be repaired.

[boobounder]

Agreed.

I have a small enough number of blue frames, and not over much of the tape ... so just editing them out is a viable option. Thus all the questions about doing it automatically.

[boobounder]

I'm guessing here ... but it seems to me that some filters, like deinterlacers and denoisers should be done with zero blue frames. Is this correct?

[sanlyn]

Denoisers don't always require deinterlacing. If you deintelace or inverse telecine, it's for all frames, and there are ways of doing a quickie temp job to see what your're dealing with. At this point we haven't seen anything you're working with. Sometimes bad tapes track better on different machines, etc., etc. A lot of things can happen. Or not. Advice: don't capture 20 hours of tape. Get short captures, cut some brief edits to have them checked. Not only are you contemplating with the cart before the horse, you don't have a horse or a cart yet.

[boobounder]

OK. Back to capture then.

If I'm bringing in VHS, should set Lagarith to YUY2?

[sanlyn]

Yes. YUY2 is a colorspace that is closest in structure to the way VHS is stored as YCbCr.

[boobounder]

Here's the most basic thing I could find: the warning message from the start of a VHS. Captured with Lagarith YUY2.

I see 3 things:

1) Some chroma across the top of the letters (like the N in the bottom row).
2) Some dots at the bottom of the letters (as in "PLEASE" in the second row).
3) Some sort of glitch around frame 56 and after.

Is this what I should notice? If I see this, what scripts and/or filters should I be considering for the whole video?

[SixFiftyThree]

Just wondering why UT Video isn't (often) brought up for lossless codecs? I haven't used it but I've read good things.

[sanlyn]

All you need to is to convince the entire world to install it. I hear there are problems UT and with colorspace conversions between BT601 and BT709, or you have to be careful with it in that regard, or something to that effect. I recently saw a post by poisondeathray that mentioned it.

[sanlyn]

Here's the most basic thing I could find: the warning message from the start of a VHS. Captured with Lagarith YUY2.

I see 3 things:

1) Some chroma across the top of the letters (like the N in the bottom row).
2) Some dots at the bottom of the letters (as in "PLEASE" in the second row).
3) Some sort of glitch around frame 56 and after.

Is this what I should notice? If I see this, what scripts and/or filters should I be considering for the whole video?

I'm afraid you'll have to show us an image or brief piece of video to explain what you mean. I don't have an "N" in the bottom row of my captures nor a PLEASE in a second row. I've had a "glitch" now and then. But basically I have no idea what you are referring to in those letters, some of which look like garbage on many tapes.

[boobounder]

I'm confused. I attached a clip the post you quoted. I don't see it now.

I also attached a clip to this post, and I don't see any sign of it either.

I clicked the "Upload files/Manage attachments" button below. A dialog opened. I pointed to my clip, and clicked Upload. Then I waited for confirmation that the clip was attached. Close the dialog as directed after the file was attached. Now I see no sign of it. What gives?

[boobounder]

Trying again.

My "Upload Files/Manage attachments" dialog now lists my file under "Current Attachments" next to a "Remove" button. I'm assuming it's OK to click the "Close this window" button in that dialog. I still see no sign in this editing window that anything is attached.

[boobounder]

Success!

Not sure why that didn't work before ... I think you can tell from my tone that I couldn't figure out anything I was doing differently.

[sanlyn]

Sometimes a few users have reported posting hangups of one kind or other. The server can get rather busy. I sometimes have problems myself and just canbcel the post and restart. However, you do have to wait for that "Manage attachments" upload window to finish up before closing it. Sometimes it takes a while.

Here's the most basic thing I could find: the warning message from the start of a VHS. Captured with Lagarith YUY2.

I see 3 things:

1) Some chroma across the top of the letters (like the N in the bottom row).
2) Some dots at the bottom of the letters (as in "PLEASE" in the second row).
3) Some sort of glitch around frame 56 and after.

Is this what I should notice? If I see this, what scripts and/or filters should I be considering for the whole video?

Tape Leader is garbage. No one keeps it. All kinds of weird things happen. And they're usually ugly. But just for demo purposes I cleaned it up in Avisynth, which was quite a hassle because there is some red-green clipping and other problems. Also, your tape player was obviously adjusting tracking at the start. I might add that your clip isn't YUY2. It's converted to RGB, which caused some cleanup problems. To keep a sample in the original colorspace, set the colorspace and compression first and then select "fast recompress" to avoid a colorspace conversion.

I recompressed your sample as Lagarith YUY2. It's attached below as "test 140205a_Lagarith_YUY2.avi". You can see that properly compressed its size is about 25MB. The RGB sample was ~150MB.

The tape leader has more than the glitch you mentioned. There are similar glitches in:
frames 13-14 (bottom)
frames 15-16 (top)
frames 26-27 (top)
frames 56-57 (middle)
frame 58 (top)
frame 101 (middle)
frames 101-102 (lower middle)
I used two Avisynth plugins to fix those a bit. They're called ripples, streaks, line twitter, and a few other names. At least you didn't have floating gray hum bars, which are a nightmare. Some of those glitches crossed multiple frames, being in the odd-numbered field of one frame and the even-numbered field of the next. If you use SeparateFields() and brighten the image in VirtualDub, you'll see how they play out.

The fonts are going to have the same soft edges, and some dot crawl, discoloration, and left edge ghosting. Some of the dots were smoothed out but discoloration in messages like that are impossible, especially with composite output. Not even a good y/c filter can clean it. There's also some terrific background noise, block noise, and banding. Upon re-encoding it would look even worse, just pure chaos. In the attached demo "Test_Denoise_demo.mkv" I had to use some fancy 16-bit plugins to get a smooth surface. This sort of never looks perfect or even "good".

This is not the point where I can discuss how I did the demo cleanup. You wouldn't keep garbage leader anyway. You might have some of those glitches later with old/damaged tape. Par for the course. We can address that when we come to it. The demo is only a brief example of what's possible.

[lordsmurf]

To start with, I think it best to play my tapes on the unit that recorded them. This has only composite output.

This is only true if the tapes are misaligned. (Or rather, the recording deck was.)

[boobounder]

Many questions ... I'll put them in separate posts to avoid confusion.

Yes, I know tape leader is garbage. I chose it because (presumably) it would have fewer problems for me to try and spot.

First question: I'm not sure why that file was RGB. I clipped it (in VirtualDub) from a file that was captured as YUY2. I assumed that clipping it would change the colorspace. So, was I supposed to "set the colorspace and compression first and then select 'fast recompress'" in the regular editing screen of VirtualDub before choose "Save as AVI"?

[sanlyn]

Yes. Color Depth (i.e, colorspcae) + compression + "fast recompress".
Another way: if all you're doing is a quick cut, which you can also do in Avisynth, just set "direct stream copy".

The leader has a solid background, plain characters, and minimal motion that make it easier to spot playback booboos. Also makes it easier to see that many of them were open to being cleaned up.

[boobounder]

Oh yikes ...

Is there an easy way, in say VirtualDub or some external utility, to tell the colorspace of a file? (If I knew how to check what I uploaded, I could've made sure it was YUY2 instead of incorrectly assuming that).

[sanlyn]

If you captured YUY2, that's what you started with. So you make some cuts in VirtualDub. You specify Lagarith as the compressor but you don't specify a colorspace, and you're in "full processing mode". In "full processing mode, VirtualDub converts to RGB. You might specify some other output (which in this case you didn't do), but the vid becomes RGB anyway. If you specified Lagarith + YUY2, and you're still in "full processing mode", the sequence is YUY2 in -> RGB convert for full processing -> YUY2 for output.

We all know that VirtualDub -- like all Windows programs -- displays images as RGB. But that's only for display, which is a special display cache that is separate from the actual video itself. What "full processing mode" means to VirtualDub is that the video is decoded and converted to RGB and readied for possible VirtualDub processing. For output you might specify some super-duper ten-gallon compressor and cyber-geek colorspace, but the video is still going through an RGB conversion stage anyway. So if you don't need any VirtualDub filters and you want to avoid a real in-place RGB conversion, don't turn on "full processing mode".

A quick way to check a video's general-spec colorspace and codec is with MediaInfo. After you first install it and use it for the first time, you'll see a "MediaInfo" menu choice on popup menus if you right-click on a video file. MediaInfo will often report YV12-style spaces as "YUV", and often reports both YUY2 and RGB as "RGB" (because they are rather alike). SO if you wanna be more certain, get out a quickie template .avs script that you save for yourself whenever you need to pop it up quickly, use Avisynth's info() function:

AviSource("drive\path\video_name")  ### <- or MPEG2Source or DirectShowSource, etc.
info()

Avisynth is usually okay at identifying YV12, YUY2, RGB, etc.. What Info() gets wrong is that it always assumes a video is BottomFieldFirst, even when it isn't. Literally, the best way to check field order and that sort of thing is to use SeparateFields(). If the file is BFF but you use "AssumeTFF()" with "SeparateFields()", motion will jerk back-and-forth because the wrong field is being displayed first. And vice versa. So if motion is haywire, try stating the opposite field order. If you SeparateFields() and see motion of any kind but every two fields are exact duplicates for 30, 60, 90, or 1000 fields, the clip is not interlaced.

[boobounder]

Excellent. I wish I'd had those last night.

Second question (in 3 parts): Regarding these excerpts:

But just for demo purposes I cleaned it up in Avisynth ... I might add that your clip isn't YUY2. It's converted to RGB ...

The tape leader has more than the glitch you mentioned. There are similar glitches in:
frames 13-14 (bottom)
frames 15-16 (top)
frames 26-27 (top)
frames 56-57 (middle)
frame 58 (top)
frame 101 (middle)
frames 101-102 (lower middle)
I used two Avisynth plugins to fix those a bit. They're called ripples, streaks, line twitter ...

a) Are ripples/streaks/line twitter (at least in part) caused by they YUY2 > RGB conversion that I didn't understand?
b) Can I see the AVIsynth script that you used for this? Even if I don't use it, I am taking notes on everything.
c) Did the 2 filters you use require you to specify the glitchy frames, or do they run through and fix all cases of ripples/streak/line twitter? Basically, for this problem, do I spot it once and run a filter on the whole thing, or look for every occurrence and isolate them?

[poisondeathray]

There are not that many, so this would be OK.

Is there a filter that will count them or flag them so I don't miss any?

Yes, you can use WriteFileIf() and AverageChromaU() > some number to detect blue frames to write the frame numbers to a text file

BUT - if that is the intended detection parameter, you can't have predominantly blue frames . In your video sample, "if video is distorted, please adjust tracking, blah blah" - every frame would be flagged as "blue" , and thus false positives. You can tweak the detection value or threshold to whatever you like . You probably want to get more false positives, so you don't miss any

Probably a sample or better description of "how blue" or what hue/color/saturation this "blue" really is would be required . Along with some "normal" frames would be ideal. (for the blue detection script purposes, the sample doesn't need to be "lossless", if other folks are giving input on cleanup etc... they probably want a lossless or near lossless sample )

You also have to decide if you want field numbers vs. frame numbers . (i.e Are you working interlaced and keeping interlaced, or bobbing to progressive at some point in the workflow - or maybe you haven't decided or learned enough to decide yet )




Also, you can probably modify auto dupe interpolaters (e.g filldrops) to do it for single blue frames or fields, instead of using Y' (YDifferenceFromPrevious) as a duplicate detection parameter, but I don't know how to do it for consecutive strings of fields or frames

[boobounder]

Thanks poisondeathray.

I am interested in this long-term, but for now I'm just learning the basics.

I just happened to have one test tape with those blue frames, and from that tape actually has vignettes, so I could easily just cut one out entirely. So ... I'm going to wait and see how often this problem comes up before worrying about it too much. No doubt I'll come back to it some months down the road.