Backup raw AVIs - Best strategy?

[tripecac]

I have about 60 hours of footage I need to capture (with WinDV), edit (with Vegas), and then make into DVDs. In addition to backing up the project files and final (compressed) video, I'd also like to backup the raw AVI files themselves, but am wondering about the best strategy.

Someone said that WinDV captures at a rate of about 13 GB/hr. For 60 hours, that means roughly 800 GB of raw AVI files.

OPTION A - DVDs
Each DVD can hold about 4 GB of data, correct? If I break up the raw AVIs into 4 GB chunks (one per DVD), that's 200 DVDs. I'd want at least 2 copies (1 for home and 1 offsite), so that's 400 DVDs total. At around 30 cents per DVD, the total cost of backing up 60 hours of footage onto DVDs would be $120. Plus, of course, many "hands on" hours spent burning, validating, and labeling each DVD.

OPTION B - External Hard Drives
Alternatively, I could store the AVI files on hard drives (via an external hard drive enclosure). 800 GB could fit on two 500 GB drives, and I'd want one copy here and one offsite, so that means I'd need four 500 GB drives. They cost around $90 each, so that's $360 total. This is three times as expensive as the DVD option, but it's a lot less tedious, since I would only have to shuffle and label 4 things instead of 400.

OPTION C - Blu-Ray
Blu-ray discs hold 25 GB, so I'd need about 32 for the offsite copy, and another 32 onsite. That's 64 blu-ray discs, and at $10 each, that's $640 which is crazy. Plus, I don't have a blu-ray burner yet, so this isn't a serious option.

So I am curious:

What raw AVI backup strategies have you guys used? Which do you prefer?

Thanks!

[AlanHK]

You can use a lossless compression like Lagarith
That should reduce the size by at least 60%.

[filmboss80]

Option D - Keep the camera original source tapes (MiniDV, I assume, since you are using WinDV capture). The cheapest backup would be a digital tape-to-tape copy. Reserve disc back ups only for specific shots you cannot live without. Not every second of the raw footage is priceless, is it?

[tripecac]

My original source tapes are standard 8mm. I'm converting them on the fly with a DV camcorder and simultaneously capturing via WinDV. No MiniDV tapes are in the loop. I supposed I could have MiniDVs backing up the analog simultaneously, but I would have to change the MiniDV tape after an hour, which makes the capture process more tedious. I'd need 60 MiniDV tapes (1 per hour) for offsite and another 60 for onsite. 120 total, $2 each = $240, and MiniDV is sort of a technological dead-end, since computers can't read it directly.

The lossless compression idea makes a lot of sense. My main worry would be that years from now I might not have a program which can uncompress the files. I'll have to look into it some more.

[tripecac]

Okay, I read up on video compression.

I've downloaded and installed the Lagarith codec. What program do you use to do the actual compression (or load-as-avi, save-as-lagarith)?

[thecoalman]

I don't have any experience with Lagarith but from the link it's comparing it to the huffy codec which produces much larger sizes than DV so I'd have to guess it's not going to save you any space and may require more.

I'd keep them as DV, archive to both tape (you can send your footage back to tape from the computer) and a external/internal hardrive. Store each in a separate location. I keep my external hardrives here at home for easy access to the footage and store the tapes at a relatives house.

[edDV]

I think you said your original source is Video8 and Hi8. Those tapes are backups as well. Don't toss them. If you are that serious about backup, a simultaneos cap to MiniDv tape could be your primary backup. DV tape can be less than $3/hr in bulk. 750GB hard drives cost around $120 on sale, sometimes less. You would keep DV format files on the HDD for immediate access and a third form of backup.

As said above, "compression" to Huffyuv or Lagarith would result in larger files and considerable effort.

[AlanHK]

I don't have any experience with Lagarith but from the link it's comparing it to the huffy codec which produces much larger sizes than DV so I'd have to guess it's not going to save you any space and may require more.

I'd assumed that "raw AVI" was uncompressed. But apparently DV AVI is compressed by some scheme.
Sorry if that was misleading.

Anyway, you could try encoding a few minutes and confirm.

[vhelp]

If you are already capturing these using a DV codec then you shouldn't bother with any other
codecs because there are none that is comparitable in size. Plus, the process alone is adding
an additional step of lossy'ness during the decoding->upsampling->converting-to-codec of choice.
The lossy part is seen as the 'rounding' error that is part of the conversion process from one
codec to another. And sometimes (depending on the dv codec) even the upsampling has been
known for causing lossy/sloppy video.

When talking 'lossy' here its assumed that once you leave one codec domain, that because of
the processes involved in the conversion and upsampling (most if not all dv codecs do this ie,
[NTSC 4:1:1 -> upsampling to 4:2:2]) the original image detail is different, even if ever so slightly.
The image could be brighter or less detailed, etc. That is an example of 'lossy'ness when leaving
one codec domain for another.

The other side of the 'lossy' part is the color space conversions, 4:1:1 vs. 4:2:2, and also you
have to consider the equation formula used in all these conversions processes. Although the
lossy'ness may not be so noticeable, it is still another lossy part of the pie. Anyway.

So storing to another codec is a waist of time here. Your actual final storate (once left from
the 8mm tape domain) is the codec format you use to capture it into. If DV is the codec, then
you already know that this is the smallest space saving medium.

But, if you use another codec (one that is larger than DV) and can find an alternative smaller codec
to convert to, then you still have the issue with the lossy'ness with the:

--> decoding->upsampling->converting-to-codec

If you want to [8mm -> dv codec] for archival (storage space) purposes then you might want o
consider storing them on DVD-r/+r discs also. And, when the prices of Blu-Ray re-writables and
disc come down within reason you might want to consider that as the final archival medium because
now you can store a full hours worth of video to one disc. But the prices have to come down on
the media too. I can see this as the next wave of archival in the not-too-distant future.

-vhelp 4750

[tripecac]

By "raw" AVI I mean whatever WinDV is writing to disc. Type 2 AVI, it says. For about an hour and a half of video, the size is 20 GB. This is much more efficient than the AVIs created by my kworld USB 2.0 device, so I'm guessing it uses different types of AVIs.

Here's a side-by-side comparison:

kworld USB 2.0 (capturing analog video)
- length: 2 minutes
- frame size: 640 x 480
- frame rate: 29 frames/second
- data rate: 220,976 kbps
- audio bit rate: 3072 kbps
- size: 3.11 GB

Firewire/WinDV (capturing DV)
- length: 2 minutes
- frame size: 720 x 480
- frame rate: 29 frames/second
- data rate: 2253 kbps
- audio bit rate: 1536 kbps
- size: 420 MB

The WinDV capture is about 1/8 the size of the kworld capture. Does this mean it is a compressed AVI? Or is it just using different sample rates? Sort of like WAV files (16 bit x 44.1 KHz vs 24 bit x 96 KHz)? I wouldn't call the 16x44.1 (CD-quality) WAVs "compressed"; they're just less dense. Is that the same case with AVIs?

[ßsimian]

tripecac - yes there are different "flavors" of avi and compressions. bitrate is a major factor in size - bits/second of data. DVavi the winDV captures is the same as what's recorded on a miniDV cam. a good slightly compressed avi that is accepted and easily used in video editing software.

a good storage choice also in that it can be printed back to miniDV tape for storage using a miniDV cam or tape deck.

[Vidd]

I would go with OPTION B - External Hard Drives .

Buy a reliable brand of drive like Western Digital.



In a couple years when Blu-Ray media is cheaper, burn a backup copy of the drives.

[tripecac]

Okay, I see... So since DV-AVI (which I'm using) is already compressed, I shouldn't bother expecting to get it any smaller, right? Makes sense.

Okay, so now for an update:

I'm suspecting that my capturing isn't as high-quality as it could be. My current captures look much better than with my old analog capture device, but the colors still seem faded and the resolution is blurry when viewed at full screen (1280x1024). I don't think it's DV-AVI's compression that is doing this; it seems to be more of a hardware sensitivity issue (either the analog outputs on my Hi8 camera or the analog-to-digital converters in the DV camera are weak).

So anyway, I think what I will do is skip the massive "raw" AVI backups for now. What I can do instead is go ahead and create video DVDs based on the medium-quality captures I have now, and then backup those DVDs. We can squeeze about 2 hours of footage on a video DVD, right? So that's 30 DVDs. 60 if I want a set offsite (which I probably do). That's much more manageable.

The downside is that if my 8mm tapes are destroyed, all I'm left with is compressed (video DVD) copies of the medium-quality captures. It's better than nothing, but not ideal.

So my plan would be to later have a second set of captures, with better hardware, aimed at getting high-quality video. Hopefully by that point, hard disks will be bigger, blu-ray will be cheaper, and online storage will be feasible. I can then make backups of the "raw" high-quality captures, and treat them like masters from now on. Right now I don't consider my DV-AVIs masters.

Does that make sense?

As long as I trim the captures so they start on the first frame of video (and not empty video), I can replace them later and reuse the same Vegas projects, right?

[thecoalman]

I'm suspecting that my capturing isn't as high-quality as it could be. My current captures look much better than with my old analog capture device, but the colors still seem faded and the resolution is blurry when viewed at full screen (1280x1024). I don't think it's DV-AVI's compression that is doing this; it seems to be more of a hardware sensitivity issue (either the analog outputs on my Hi8 camera or the analog-to-digital converters in the DV camera are weak).

You can't use a computer monitor to accurately gauge the quality of the capture for a variety of reasons. Try authoring a high bitrate DVD test disc to view on the TV, preferably a scene with a lot of action. Convert the DV-AVI from to something like 720x480 @8000kbps. Assuming a decent software encoder your resulting DVD should be fairly close to the original camera footage.

[edDV]

By "raw" AVI I mean whatever WinDV is writing to disc. Type 2 AVI, it says. For about an hour and a half of video, the size is 20 GB. This is much more efficient than the AVIs created by my kworld USB 2.0 device, so I'm guessing it uses different types of AVIs.

Here's a side-by-side comparison:

kworld USB 2.0 (capturing analog video)
- length: 2 minutes
- frame size: 640 x 480
- frame rate: 29 frames/second
- data rate: 220,976 kbps
- audio bit rate: 3072 kbps
- size: 3.11 GB

Firewire/WinDV (capturing DV)
- length: 2 minutes
- frame size: 720 x 480
- frame rate: 29 frames/second
- data rate: 2253 kbps
- audio bit rate: 1536 kbps
- size: 420 MB

The WinDV capture is about 1/8 the size of the kworld capture. Does this mean it is a compressed AVI? Or is it just using different sample rates? Sort of like WAV files (16 bit x 44.1 KHz vs 24 bit x 96 KHz)? I wouldn't call the 16x44.1 (CD-quality) WAVs "compressed"; they're just less dense. Is that the same case with AVIs?

AVI is a container not a format. You were capturing uncompressed with your K-World device. The camcorder is hardware compressing to DV format in the camera and then streaming in DV format to IEEE-1394 (aka Firewire,I-Link). DV format uses 4:1:1 sampling and ~5x intraframe compression resulting in a 25Mb/s bit rate. Audio is uncompressed PCM @ 48K sample rate.

Many other types of compression CODEC can be used in an AVI container. The extension "AVI" tells you nothing about the codec being used. Gspot is a good program to ID the video and audio codecs being used. The term DV "raw" describes a file format used by Mac and some other equipment. RAW just means the data is saved to a file as it comes in over the Firewire port. AVI means audio video interleave. DV data is interleaved to AVI before writing to disk. (More detail at MSDN if you want to understand interleave).

DV uses ITU-Rec601 digital levels, frame sizes and frame rates. This makes DV interoperable with most digital broadcast formats (e.g. ATSC, DVB, Digital Betacam) and DVD MPeg2 which also references ITU-Rec601.
http://en.wikipedia.org/wiki/CCIR_601

[edDV]

Okay, I see... So since DV-AVI (which I'm using) is already compressed, I shouldn't bother expecting to get it any smaller, right? Makes sense.

Okay, so now for an update:

I'm suspecting that my capturing isn't as high-quality as it could be. My current captures look much better than with my old analog capture device, but the colors still seem faded and the resolution is blurry when viewed at full screen (1280x1024). I don't think it's DV-AVI's compression that is doing this; it seems to be more of a hardware sensitivity issue (either the analog outputs on my Hi8 camera or the analog-to-digital converters in the DV camera are weak).
...

For archive, you should keep DV as DV. Next best is MPeg2 in DVD format but that is a large step down.

As for quality assessment, you have much to learn about monitoring an NLE like Vegas. First rule of thumb, monitor from a TV not a computer monitor. Vegas (also Premiere, FCP, etc,) support monitoring the timeline from the IEEE-1394 port. You connect your TV through the camcorder. It helps to have a TV with "user" settings because you will need a different "brightness" setting to monitor a DV stream. This all gets back to that 0.0 vs. 7.5 IRE issue discussed in your other thread. The picture you get off the timeline should be identical to the picture you see while dubbing "pass through" or when playing MiniDV tape. The "quality" is set by the camcorder's analog to digital conversion. DV bit rates are more than enough for analog NTSC source.

[tripecac]

You can't use a computer monitor to accurately gauge the quality of the capture ... your resulting DVD should be fairly close to the original camera footage.

Yeah, I did a test like you said, and I see exactly that! (see https://forum.videohelp.com/topic353032.html for details) So, it seems like what I'm capturing now is actually pretty accurate. It just looks weird on the monitor.

Gspot is a good program to ID the video and audio codecs being used.

Thanks! This is starting to make more sense now...

kworld capture stats:
- OpenDML (AVI v2.0)
- Video: 1.75 GB (97.57%)
- Audio: 44.6 MB (2.43%)
- AVI Overhead: 99.5 KB (0.01%)
- Codec: YUY2 (YUV packed 4:2:2)
- Audio: PCM Audio - 96000Hz 3072 kb/s tot (2 chnls)

Firewire/WinDV capture stats:
- DV Type 2 AVI
- OpenDML (AVI v2.0)
- DV Audio/Video: 21.0 MB (93.96%)
- AVI Copy of DV Audio: 1.12 MB (5.02%)
- AVI Overhead: 235 KB (1.03%)
- Codec: dvsd (DVC/DV Video)
- Audio: PCM Audio - 48000Hz 1536 kb/s tot (2 chnls)

For archive, you should keep DV as DV.

Okay, this means I'm back to choosing between DVD, HDD, and MiniDV. At this point I think I am favoring HDD since it's the easiest and takes up the least physical space.

monitoring the timeline from the IEEE-1394 port.

Does this mean Vegas can send the video back to the camcorder, which then displays it on the TV? Sounds like a great idea! I will try to figure out how to do it...

UPDATE: I got the external monitoring working!!! This is seriously cool! Now I just need to find a permanent place for the TV...

Thanks so much guys! This forum (along w/ my testing) is teaching me more about video in the past couple days than I learned in the past couple years.

[thecoalman]

Yeah, I did a test like you said, and I see exactly that! (see https://forum.videohelp.com/topic353032.html for details) So, it seems like what I'm capturing now is actually pretty accurate. It just looks weird on the monitor.

The big difference is your monitor is progessive and a TV is interlaced. The footage you have is interlaced too which looks the best played back on a interlaced TV. when it's played back on monitor the software player you are using needs to deinterlace on the fly and results vary by software player.