RAID 5 Configuration - I'm So Torn!

[rbautch]

I need about 1-1.5TB of storage space, and of course all my data is critical so I want full redundancy. My PC has an ASUS P4P800SE mobo, with a p4 3.2GHz processor. I do all my captures with DV firewire. After settling on RAID 5, I'm having trouble deciding between these three configurations:

1) Building a new server just for RAID and connecting it to my PC's with Gbit Ethernet.

2) Adding a PCI-X raid controller card to my existing PC.

3) Swapping out my mobo with one that has on-board raid 5.

With option 1, I'm concerned that my mobo is really capable of achieving gigabit speeds (unsure what bus the ethernet controller is using).

With option 2, I'm concerned that I won't get enough speed to support firewire video transfers. I know PCI-X can achieve 1.0GB/s speeds, but I heard that gets slowed down considerably if it's connected to a standard PCI slot.

With option 2 and 3, I'm concerned my PSU can't handle the load from 6 hard drives.

I don't mind spending a few bucks to get a robust system. Please help!!

[edDV]

I need about 1-1.5TB of storage space, and of course all my data is critical so I want full redundancy. My PC has an ASUS P4P800SE mobo, with a p4 3.2GHz processor. I do all my captures with DV firewire. After settling on RAID 5, I'm having trouble deciding between these three configurations:

1) Building a new server just for RAID and connecting it to my PC's with Gbit Ethernet.

2) Adding a PCI-X raid controller card to my existing PC.

3) Swapping out my mobo with one that has on-board raid 5.

With option 1, I'm concerned that my mobo is really capable of achieving gigabit speeds (unsure what bus the ethernet controller is using).

With option 2, I'm concerned that I won't get enough speed to support firewire video transfers. I know PCI-X can achieve 1.0GB/s speeds, but I heard that gets slowed down considerably if it's connected to a standard PCI slot.

With option 2 and 3, I'm concerned my PSU can't handle the load from 6 hard drives.

I don't mind spending a few bucks to get a robust system. Please help!!

Lets start with assumptions. Why do you figure you need RAID?

What software are you using? Are you using a real-time hardware path?

For redundancy of DV data, just keep a tape backup.

[rbautch]

The biggest reason is that I'm tired of spreading my projects out over multiple disk drives, always hunting for the drive that has enough space for my current project. I'd like one large "virtual drive" where I can efficiently orgainize my projects.

Most of my projects are for customers and I can't afford losing data, so I want full redundancy. I also have my entire DVD collection ripped onto my hard drives (I serve them to tvs from there), and I'd hate to have to re-rip them if I lost a drive.

I'm using Scenalyzer to capture, Vegas to edit/render, and DVDLab to author.

What is a "real time hardware path"?

[rallynavvie]

I actually went through a similar ordeal as I was filling space with my video work. It was all customer data so redundancy and security were paramount. However you need to be more specific to the application. What are you using that needs the bandwidth or striping? The only way I can even think of that you'd exceed a single hard drive's I/O would be capturing from multiple sources, as in probably 4 or more in DV or 3 in HD. But from the rig you're driving it doesn't sound like that's what you're doing. The only other time you may need that sort of I/O performance is when using AfterEffects or something laying down a lot of effects and scrubbing back and forth between the same chunk of video. If that's the case set up a small "scratch" array in RAID 0 between a couple of small disks. I did this with two 18GB 15k SCSI drives but you'd have to spend a little more for the hardware.

I applaud your efforts with redundancy however. I solved this by using simple RAID 1 between two identical drives. I'd buy a pair of 200GB of 250GB drives and mirror them from my onboard controller. Then when they were full I'd take inventory of them, label them, pull them both, one going to my closet and one going to my bank box for safe keeping. This way I had archived all my data in two places. If you're willing to spend the money and are doing this for hire then this would be a great solution, it has served me well.

I did use a file server for ongoing projects so that I wouldn't have to swap hard drives in when I needed to work over an old project again. I had a few repeat customers that I used this with. My file server used to be a dual Athlon MP machine but now it's my new Xeon machine. I moved up to GbE just so that I could move large files over the network faster. I would never attempt to work on files remotely though, there's just too many wierd things that can happen with your network to attempt doing that, even if it were isolated from the internet. You can probably get away with an old machine for a file server though, just make sure its PCI bus can support GbE speeds.

[edDV]

I aggree with rallynavvie. I make sure all projects are archived on hard disk (+DVD for graphics and audio elements+DVD copy masters) and DV tape (save all useful clips and saved edit master).

I don't do off premises backup copies although I should now that media is getting cheaper. I should be at least separating the HDD and DV tapes.

I still swap drives for projects. That way I can keep all the temp files in place and simply continue from where I left off. I used to use RAID to get bandwidth but don't need it for 98% of what I do here. I maintain an old networked WinNT machine with SCSI RAID, ATA RAID and Pinnacle DC30plus (MJPEG) for reviving old MJEG projects or for uncompressed capture (mostly animation).

RAID is a hassle. I like partitions sized to the project that are easy to defragment. External file servers have additional issues and problems that we can discuss. To do it right, you need to spend the $$$$ for SDI.

[edDV]

What is a "real time hardware path"?

For Vegas, that would be the Blackmagic Decklink
http://www.blackmagic-design.com/products/sd/
for connecting to SDI servers and external decks.

Premiere Pro and AVID have many realtime environments that require expensive RAID or SDI RAID servers.
http://www.videoguys.com/realtime.htm

[rbautch]

Thanks for the insight. I'm especially interested in your comments about working on files over a network. My I/O needs are straightforward, and the most intensive operations would be firewire captures (which is about 500mb/s, IIRC) and video rendering. If I pursued the server option, I'd be looking to use it for more than just storage, but also as a real time I/O device. In other words, I'd want all my files on the server, and none on the local PC. I guess my question now becomes: can I do these operations effectively over a GigE network in real time?

Instead of a separate server, if I use a PCI or PCI-X Raid controller in my main PC, would the PCI bus speed limit my ability to do real time DVI captures via fire wire? I'm not sure what the PCI bus speed of my mobo is. The spec sheet lists "Maximum Bus Speed" as 800MHz, but I'm not sure that applies to the PCI bus.

Keep in mind that I'm looking for about 1TB of space, so I'll likely use four 300gig drives.

[edsmith77]

With this hit and miss approach to solving your needs you will most likely end up with a less than adequate solution.
What you need is a regimented SE approach to requirements definition with cost and prioritization factored in.
With this data in hand the path forward is usually very clear.

To start on a piece of paper write down in sequence order your absolute "must" have needs. Follow that with the "want" to have needs. That followed by the "nice' to have needs. Then rearrange your list into a sequence of implementation steps. If you do not have all of the "MUST" have's at the top of the list followed by the "WANT" to's and lastly the "NICE" to's.
Then go back to step one and start again. Because you do not have the correct set of MUST requriements.

After completing the above, then next to each step write down what it will take to make that step happen (hw/sw). Then price each one. Then add up the list from the top until you run out of money. Then there is your answer.

[edDV]

...

"the most intensive operations would be firewire captures (which is about 500mb/s, IIRC)"

What is this?
DV transfer is a stream (not file transfer under OS control) of 28-35Mb/s. Are you using IEEE-1394 for some other purpose?

For reliable performance it is best to do DV transfers to and from a local drive other than the OS drive. This is important to avoid frame drops. Once you have all your streams in files, then you are into normal network system analysis as edsmith77 describes.

You are taking on a network engineering task. You may be up to it but are your customers willing to pay you to develop this network?

[rallynavvie]

Yeah, there's no way you're pushing the limits of FireWire or ATA bandwidth. As I said you need to be capturing from several source simulatneously in order to exceed what a single internal hard drive can handle. I don't recommend doing this over a network though. I only use GbE for moving my files back to my main workstation from my file server to work with, not work with them directly from the file server. GbE is fast, but it still just uses a network protocol and is a lot more likely to have issues than would a hard drive.