Pre-processing DV video - What does it?

[Colmino]

I tried to tackle this in a different thread and it turns out I was talking about the wrong thing. So I'm trying again.

Pre-processing is - if I'm getting it right this time - the removal of compression artifacts from the DV video. I read somewhere that it can take around 8 hours per hour of DV video to get the best quality pre-processing.

First of all.. What does everyone out there use for their DV pre-processing? I know Avisynth / VirtualDub are supposed to have filters which can accomplish this. Fine. What are they, and which one(s) is/are the best? And is there an even better solution in the commercial market?

Apart from that, I am once again confused about what this entails, exactly. Near as I can tell, if one pre-processes DV video, they'll end up with raw video, or else they'll re-encode it with some sort of lossy compression... perhaps back to DV. Seems to me that would defeat the purpose. But at the same time, an hour of raw video is, according to another thread, some 60 GB. I'm not quite grasping what the standard procedure is when it comes to pre-processing. The very fact that there seems to be only two (undesirable) choices is what caused me to assume that people generally didn't bother.

Thanks in advance for clearing it up!

[redwudz]

I've used DV for some time, but I have never heard of 'pre-processing'. Do you have a link to where you got this information? DV is DV. It's what you get from your source, be it a ADVC-100 or a video camera. It's a file. You can 'post-process' it in Virutualdub or other programs. You can filter DV or do all kinds of manipulations in VD with it, but that is after it is downloaded to your PC. You won't end up with 'raw' video because you didn't start with it. The bottom line is that you can only work with what you are given. DV is compressed video that is output from a hardware encoder.

[pijetro]

Just to get rid of a few pre-conceived notions, here we go...

Pre-processing is - if I'm getting it right this time - the removal of compression artifacts from the DV video

Nope!!! Your DV video shouldn't contain compression artifacts. If it does, return the camera :P

First of all.. What does everyone out there use for their DV pre-processing?

Dude, this is the first time i've heard of that term on this forum. The term perhaps is more related to pre-editing, prior to export of a different type of media, ie: DVD, web, tape, etc...Pre-processing is a generic term for people who deserve to get conned..Much like "Pro-Active"

....they'll end up with raw video, or else they'll re-encode it with some sort of lossy compression... perhaps back to DV

You can work with DV all you want. It's simply a transfer. The only recompression happens at transitions, so there's no quality loss whatsoever, if it's sent back, or saved as DV. .The term raw video, can be two-fold, depending on who you're talking to...
As far as I'm concerned (along with most people), raw video means capturing an .AVI without any compression. I won't get into that subject, but know this, DV isn't raw, in fact it does have a slight compression, but for some video editing people, it's considered raw, since 13gigs an hour of transfer seem huge.

The great thing about Vdub, or AVISynth, is that you can use their filters before serving the video back out. Whether it's noise removal, cropping, or resizing..So, yes, perhaps there is pre-processing going on, but not to the extent or finality that you seem to think.

Good luck!!!

[Colmino]

Nope!!! Your DV video shouldn't contain compression artifacts. If it does, return the camera

Okay, I won't split hairs. Let me just put forward what I do know: 1) DV, as recorded by camcorders, is a video format that utilizes lossy compression (5:1). 2) Artifacts which cannot be attributed to optical phenomena or CCD inadequacies, and which bear a striking resemblance to typical compression artifacts, are easily identifiable, to me, whenever I watch my camcorder recordings, especially whenever the image is busy. 3) I'm not the only person to have referenced these artifacts as being the consequence of the lossy compression the DV format utilizes.

Let me put it another way. Take any DV file you want, pass it to VirtualDub, and have VD reencode it as DV. Do that a few times. I haven't tried this myself, but I have a suspicion that the results will exhibit compression artifacts.

Pre-processing is a generic term for people who deserve to get conned..

The only software I have thus-far located which may do what I'm talking about happens to use the term. And I gather they aren't out to con people with a mediocre product. See www.visinf.com/gstech.jsp.

The only recompression happens at transitions, so there's no quality loss whatsoever, if it's sent back, or saved as DV.

I can understand there being no loss if one merely cuts bits out, but something as involved as artifact removal - a thoroughly "destructive" process - must make it impossible to avoid either re-encoding the file or saving as raw, like I originally suggested.

The great thing about Vdub, or AVISynth, is that you can use their filters before serving the video back out. Whether it's noise removal, cropping, or resizing..So, yes, perhaps there is pre-processing going on, but not to the extent or finality that you seem to think.

So all I really need to figure out now is what all's involved in "serving the video back out". I'm used to this meaning "save the results to the hard drive", and that's probably still the most failsafe method. But perhaps infeasible with video.

[redwudz]

I looked at the site you mentioned. Your link doesn't work. 'Pre-processing' or not, if you started with a DV file, that's what you have to work with. If you want higher quality, don't use DV. Yes, it has artifacts, it is a compression format. That said, if you have a DV file, you can filter it. Any filter removes something. You won't improve quality, you will just remove noise, etc.

[pijetro]

Artifacts which cannot be attributed to optical phenomena or CCD inadequacies, and which bear a striking resemblance to typical compression artifacts, are easily identifiable, to me, whenever I watch my camcorder recordings

Well then, no matter what you do, you won't fix the problem. The term "lossy compression" shouldn't get mixed up with the term "artifacts". My JPGs are also lossy, but compared to uncompressed TIFF, i'll trade the space for quality any time

Take any DV file you want, pass it to VirtualDub, and have VD reencode it as DV.

In essence, you are recompressing in this case, which does lead to video degredation.

artifact removal - a thoroughly "destructive" process

Not really. If anything, a constructive process. I'm beginning to know where you're going with this...

I checked out the link you've mentioned. First of all, what the website you refer to calls grain, we here usually simply call noise. When you say artifacts, I'm assuming some sort of pixelation...

So, what to do.

Don't worry about lossy compression, or get worked up about the theory of degraded videos. The fact of the matter is, most people want to put DV material back out on DVD or tape..The importance over all of this would be getting quality footage. If budget, camera, lighting doesn't allow for this, and you're stuck with graininess, then there's a solution....


AVISYNTH
There's a multitude of filters to use, which all behave and clean noise differently. The filters come with documentation, so you can pick and choose which one works for you...

But don't forget the big picture. If your source video is constantly showing "artifacts", or "noise", you could save a lot more time by working out the bugs at the pre-production, rather than post-prodution stage. Take into consideration that filtering noise out properly consumes gobs of time!!!

Good luck!!!

[Colmino]

I looked at the site you mentioned. Your link doesn't work.

Yeah, my bad. Remove the period and that should fix things.

'Pre-processing' or not, if you started with a DV file, that's what you have to work with. If you want higher quality, don't use DV. Yes, it has artifacts, it is a compression format.

I'm not complaining. Just trying to make as many improvements as current technology allows.

That said, if you have a DV file, you can filter it. Any filter removes something. You won't improve quality, you will just remove noise, etc.

I believe that if a filter does its job correctly, it is possible to improve the quality, by virtue of the removal of noise. Let's stipulate the extreme case, where somebody takes the time to painstakingly perform the manual fixing of obvious compression artifacts, from within Photoshop or whatever fits the bill. There you have it: reduced noise without side effects.

Anyway, since I plan to be moving the video to DVDR at the end of the process, it makes sense to eliminate the DV compression artifacts even if the filter isn't good enough to avoid reducing overall image quality, just so the Mpeg2 encoder doesn't throw a fit.

[Colmino]

When you say artifacts, I'm assuming some sort of pixelation...

The pixelation isn't so much of a concern. More of an issue are the moments when the image is too busy. Blockiness abounds.

The importance over all of this would be getting quality footage.

Generally infeasible, in my case. These aren't journeyman semi-professional efforts we're talking about. These are home videos taken by a variety of non-technical individuals. ;P

Take into consideration that filtering noise out properly consumes gobs of time!!!

Yeah, that's not a problem. I'm not churning out one DVD a day, after all. Just preserving family videos on a disc that can be played in most living rooms. Question, though. What would be a good, straightforward example of filtering noise out properly? I mean, which filter(s) are standard and what settings and whatnot?

[disturbed1]

Filters for avisynth http://www.avisynth.org/warpenterprises/

Most all come with help files that explain what the filter does and how to use it, along with a webpage link.

In Virtual, when you hit the add filters button, select a filter, there is also a help button there that will explain how to use the filter.

The pixelation isn't so much of a concern. More of an issue are the moments when the image is too busy. Blockiness abounds.

This is pixelation. Is this happening after you compress to mpeg? Or are you capturing to mpeg in realtime? If the DV footage has this in it, take it back, this should not happen with the native DV footage (unless the cam is attached to a barrel rolling airplane).

If it is in the mpeg file, use a better encoder.

[Colmino]

This is pixelation. Is this happening after you compress to mpeg? Or are you capturing to mpeg in realtime?

Nope. Regular DV footage, not captured.

If the DV footage has this in it, take it back, this should not happen with the native DV footage (unless the cam is attached to a barrel rolling airplane).

Apart from conflicting with what I've personally observed, this just doesn't seem to be entirely absolute. Given a worst-case scenario, any lossy compression will reveal its drawbacks. How about if I pipe white noise via analog from a TV to the camcorder and have it record that? Not that it needs it; the moments of fast camera motion I have witnessed have proven to be sufficient.

I don't think it has to do with the quality or state of functionality of the camcorder. I see identical artifacts from three different camcorders, and the one I personally use, just to put the quality concern to rest, is an imported Panasonic NV-GS100.

[pijetro]

Why don't you post a clip, or URL or something, so we can quantify your claims? I can see that this is going in circles...

You keep mentioning about lossy compression..This is bogus..Even if your equipment was capable of a lossy format, you'd still get, what you consider artifacts..

Perhaps a new type of media is in order ie:MircoMV, or DVDRam, or whatever...

[BJ_M]

that link you posted ( http://www.visinf.com/gs/ae/ is the correect link)
is for grain surgery 2 .. excellent product ... made for film to video transfers and video to film .. it can remove or add grain and some noise reduction - you can do the same with freeware filters at about the same quality.. grain surgery 2 only works with Aftereffects ...

[Colmino]

Why don't you post a clip, or URL or something, so we can quantify your claims?

Generally, I'm inclined to submit that I know what I'm seeing. I'm not utterly clueless. So trust me. But truthfully, if I had a clip to show you, and the means to show it, I would. I haven't yet gotten around to researching what hardware I'd need to capture DV, nor which software is best for doing so. (As an example, I'd want software that knows when it was forced to "drop frames" so I could start over, rather than just praying it didn't happen.)

You keep mentioning about lossy compression..This is bogus..

Alright, so the artifacts I'm seeing aren't compression-related, even though they occur in direct relation to on-screen action. I'll buy that. Doesn't change the fact that I'd like to reduce 'em.

it can remove or add grain and some noise reduction - you can do the same with freeware filters at about the same quality..

Would you be willing to name some names? Your link is certainly more complete than the one I'd found, and it seems like Grain Surgery is a rather thorough product, for what it does.

As for converting video to film... If there's something out there that can turn NTSC into 24fps without too much in the way of image corruption, I'd be interested in knowing about it. That would be a fun toy.