Lots of dropped frames when using RAID - Why?

[Malchiah]

This post was blank by accident. The intended post follows.

[Malchiah]

Ok... my initial posts were appearing blank. Here is my problem:

I'm trying to capture DV from my Sony VX2100 via firewire using Premiere Pro. When I capture to my system drive (120gb IDE drive) I get NO dropped frames. But when I capture to my two brand new Western Digital 250gb SATA drives set up in RAID, I get dropped frames almost immediately after I begin!

What could be causing this? Could it be that the RAID controller onboard my Asus P4PE motherboard can't handle this? Is there a better RAID controller out there that works well with capturing and editing DV?

(I've tried both RAID-0 and RADI-1 and I get the same results.)

I'm currently running:
Windows XP SP2, newly reinstalled to eliminate clutter
P4 2.4ghz
1gb RAM
Geforce FX 5700 video card
WD 120gb system drive
2x WD 250gb SATA drives in RAID
Onboard Firewire

Thanks.
- Joel Morgan

[trevlac]

You don't provide enough information.

Here's a wild a$$ guess. You are using a Highpoint raid card with raid 5. The write speed is dog slow, so you can't capture without dropped frames. Worse yet, the raid 5 checksum is calculated by your cpu, so this is putting a further load on the machine.

How'd I do?

[lordsmurf]

RAID was made for servers, not video capture.

[Cornucopia]

RAID was made for servers, not video capture.

WTF? RAID is MANDATORY for video capture if you're doing multi-stream uncompressed SD, or ANY mildly or uncompressed HD material!

Remember, there are different RAID levels : 0, 1, 5, 10 etc.

Video people (often) use raid level 0, which Stripes the discs. Each sector has 1/2 written to disc 1 and 1/2 written to disc2 (or read from). This allows fast access even with slower drives. Make sure you have good quality drives, though, because Raid Level 0 has NO fault tolerance and you would have both drives screwed if even 1 of them fails.

My guess is the poster was using the wrong raid level.

Scott

[edDV]

The real question is why use RAID, unless using older HDD technology or attempting uncompressed capture.

DV capture does not need RAID 0 with a modern disk controller and HDD (i.e. ATA 66 or better).

Exception is support of high end multi-stream realtime hardware that does need RAID 0 to support 3-5 simultaneous DV or SDI streams.

[Cornucopia]

DV doesn't "need" RAID, but it can help. I've got a RAID system that I primarily use for DV and have never had even 1 dropped frame in 4 years.
Plus, it makes file copying be almost 2x as fast.

Scott

[canadateck]

I always thought if anyone is Deep into Digital Video of any kind, they would have a RAID. I have always used RAID.

[edDV]

The key issue is no longer sustained data transfer rate. A single ATA-100 controller-HDD pair can routinely reach 40-55MB/s sustained transfer in isolation. More important is separating the video drive and controller from OS activity*. A second EIDE or SCSI disk controller and HDD will achieve isolation from OS processes and moderate background tasks.

This separation depends on PCI bus mastering technology that allows data transfer to occur separate from CPU intervention. In other words, the IEEE-1394 interface and the second disk controller can do their thing while the OS and the main drive operate largely in separation.


* It's important that all "scratch disk" and "capture buffers" be placed on the video drive, not the OS drive.

[Malchiah]

Thanks for your replies even though my initial posts were blank. I did actually write a message, but my posts were appearing empty and I couldn't edit them. (Others apparently have had this problem recently.)

Anyway, please read the edited post above.

[Malchiah]

As noted in my edited post above, I tried both RAID-0 and RAID-1 with the same results.

And edDV, the odd thing is that I actually get BETTER performance capturing to my system drive (which is IDE) than the RAID. That defies logic. That means, if I get good performance on my system drive, I should get even better performance from the RAID. Something is obviously wrong.

[canadateck]

If you are Tranfering DV footage,I would say that there is somthing else wrong,you Transfer DV not Capture, RAID is IMO the best for Capture,Not really needed for Transfer ,but would be better. Did you just install Sp1? was it working before you instaled it?

[lordsmurf]

I routinely (and have for years) captured MPEG-2 as high as I want (up to 15Mb/s even), and uncompressed (or even codecs) AVI. Just use 7200rpm and follow standard drop-frame prevention rules.

RAID is for servers, and was created for redundant backups. This RAID 0 nonsense often just causes more trouble than it's worth, and strains the disks in use (shorter lifespan, and you kill TWO at once, not just one). The fragmented nature of the tech is also NOT VERY GOOD with large video files.

Again, not for video. There are some special video servers that use variations of RAID, but we're not talking things done at home with consumer equipment.

[canadateck]

Again, not for video. There are some special video servers that use variations of RAID, but we're not talking things done at home with consumer equipment.

Not everyone Knows what a RAID is or how to set one up. I have never heard RAID is not For Video,If anything else Always the oposite.

I routinely (and have for years) captured MPEG-2 as high as I want (up to 15Mb/s even), and uncompressed (or even codecs) AVI. Just use 7200rpm and follow standard drop-frame prevention rules.

I can say the same Using a RAID.

[canadateck]

This RAID 0 nonsense often just causes more trouble than it's worth, and strains the disks in use (shorter lifespan, and you kill TWO at once, not just one)..

This is a very Bold Statment ,We can say the same about Over-clocking,But in the End "Faster Is Better". What happens when somthing fails ,well that is a different story.

[Malchiah]

In my particular case, it doesn't matter whether RAID is good or not. It simply is unusable. (Dropped frames immediately after I begin transferring.)

I know people who use RAID all the time for video and they don't have problems. My old school still uses PC's with a few large hard drives set up in RAID-0.

My guess is that the onboard RAID controller I'm using is a crappy one. If anyone has had a good RAID experience, I'd like some suggestions as to which one I should get.

Canadateck, you're right, I did mean transfer, not capture. Also, the drives are both new and I reinstalled Windows when I got them, so I don't know if they worked before the service pack upgrade.

[edDV]

As noted in my edited post above, I tried both RAID-0 and RAID-1 with the same results.

And edDV, the odd thing is that I actually get BETTER performance capturing to my system drive (which is IDE) than the RAID. That defies logic. That means, if I get good performance on my system drive, I should get even better performance from the RAID. Something is obviously wrong.

Well whether it is needed or not, it should be working better than that. Is this the Promise RAID chipset? I have 2 of those here that currently are set to be EIDE3 and EIDE4 with multiple drives attached.

I also have an older system that has the Promise FastTrak-100 ATA RAID controller running a pair of 40GB ATA-100 7200RPM Quantums in RAID 0. Everything works fine on that system and has for 4 years now. Using the Canopus sustained transfer test, this RAID produces 52 MB/s sustained, more than enough for a 9MB/sec MJPEG DC-30plus single stream card or for DV.

A new single 200GB ATA133 drive on another system also gets 50MB/s sustained transfer using the same test. The rate is the same for either a ATA-100 or ATA-133 disk controller. A friends 7200RPM SATA drive runs about the same speed.

Realtime video compositing cards require 3-5 sustained and synchonized streams. In those systems RAID provides more reliable multi-stream performance over the entire disk capacity. For single DV streams this just isn't a factor.

I get into my RAID setup with Ctl-F during boot. It shows my RAID as "Functional".

I can't tell you what is wrong with your system. The Promise RAID chipset does work. I'd suggest you configure without RAID as EIDE3 and EIDE4 and start editing.

[proxyx99]

If using RAID would allow you to do what you can't do without it, perhaps... but all ATA66 and up drives in IDE setup will easily handle the DV transfer so why bother. Why go through the pains of setup, plus loss of space. Just to say "I've got RAID"? Who cares...?

[FulciLives]

If using RAID would allow you to do what you can't do without it, perhaps... but all ATA66 and up drives in IDE setup will easily handle the DV transfer so why bother. Why go through the pains of setup, plus loss of space. Just to say "I've got RAID"? Who cares...?

A bit harsh but so true.

You simply DO NOT need RAID for DV "capture" these days.

So if I was in your position (the originator of this thread) I would totally give up on the whole RAID thing.

Unless you got a BIG cockroach problem

- John "FulciLives" Coleman

P.S.
Just use one of your RAID drives as your second HDD and dump your DV "capture" there instead of the main system drive (i.e., the drive that boots with Windows installed on it).

Also I have heard that some people (and this is very rare) do experience A/V sync issues with DV "capture". When that happens using ScenalyzerLive as your DV "capture" application seems to solve such A/V sync issues. I had read (don't know how true this is) that ScenalyzerLive uses it's own "built-in" routine for doing what it does whereas most if not all other DV "capture" applications source some work out to the Windows operating system and that is the reason why some people experience A/V sync problems when NOT using ScenalyzerLive. Granted that explanation might be BS but I have read many threads where people claimed that ScenalyzerLive was the only DV "capture" application that worked for them (in terms of A/V sync) so I wanted to bring it up because this seems to be a little known "secret".

[Zisguy1]

This RAID 0 nonsense often just causes more trouble than it's worth, and strains the disks in use (shorter lifespan, and you kill TWO at once, not just one)..

This is a very Bold Statment ,We can say the same about Over-clocking,But in the End "Faster Is Better".

Your statement seems just as bold, maybe even bolder. "Faster is better?" So you'd go with a very fast PC that crashes numerous times per day because of overheating, or too-fast RAM access, or maybe dragging the PCI bus out of spec? Have fun...

[Jester700]

This RAID 0 nonsense often just causes more trouble than it's worth, and strains the disks in use (shorter lifespan, and you kill TWO at once, not just one)..

This is a very Bold Statment ,We can say the same about Over-clocking,But in the End "Faster Is Better".

Your statement seems just as bold, maybe even bolder. "Faster is better?" So you'd go with a very fast PC that crashes numerous times per day because of overheating, or too-fast RAM access, or maybe dragging the PCI bus out of spec? Have fun...

Well, there are levels to everything. I've OCed my PCs since the 486 days, and have had very litte trouble, basically because before doing anything important I test the setup very well, hammer it for hours and check temps, etc. Video encoding is great for this, BTW. And I'm conservative - I'm not interested in the top speed I can get. I'm interested in a decent increase over stock speed while still stable.

With some OCs, the only thing out of spec is the CPU; and remember that these chips come off the same lines as faster marked ones.

[canadateck]

This RAID 0 nonsense often just causes more trouble than it's worth, and strains the disks in use (shorter lifespan, and you kill TWO at once, not just one)..

This is a very Bold Statment ,We can say the same about Over-clocking,But in the End "Faster Is Better".

Your statement seems just as bold, maybe even bolder. "Faster is better?" So you'd go with a very fast PC that crashes numerous times per day because of overheating, or too-fast RAM access, or maybe dragging the PCI bus out of spec? Have fun...

No I wouldnt,I have never Overclocked my CPU, RAID is meant to be a RAID, A 1200 CPU is menat to Be a 1200 CPU.

Saying a RAID 0 is nonsense well thats just nonsense.

[edDV]

RAID 0 still has a place. Just that place is not single DV stream transfers. I keep it going for old time's sake with my MJPEG capture board.

[Malchiah]

The Promise RAID chipset does work. I'd suggest you configure without RAID as EIDE3 and EIDE4 and start editing.

Yes, it's the Promise FastTrak RAID controller. The only problem here is that it's SATA, not IDE. For some reason, when Asus put the SATA option on the P4PE, they made it so it's only usable in RAID. I can't run a single SATA drive, as far as I can tell.

In the bios, there is an option for SATA/RAID Enabled or Disabled, and that's it. I tried Enabling it, then booting into Windows without RAIDing the drives, but they are not recognized until I either Mirror or Stripe them. I flashed the bios with the latest version in case I was missing some menu options in the bios, but no luck there. There doesn't seem to be an option to use SATA without RAID.

[Malchiah]

Just use one of your RAID drives as your second HDD and dump your DV "capture" there instead of the main system drive (i.e., the drive that boots with Windows installed on it).

The RAID drives are SATA. The Asus P4PE motherboard doesn't seem to allow me to use them independently. That does sound strange, but I have never found the options in the bios to allow that.

Like I said before, I am getting better performance capturing to my system drive. What I am doing now is transferring my DV to the system drive, then moving the files over to my RAID drives for storage. I have about 10 hours of DV footage to input and not enough drive space without the RAID.

The only hassle with that is re-linking the media files with the clips in Premiere once I've moved their location.

[edDV]

Unless you have plans for a high end multi-stream realtime card, I'd get a separate (non RAID) PCI SATA controller and forget the Asus ports.

[gadgetguy]

I feel your pain, for I too cannot capture to my SATA drive. In fact I had to try several 3rd party drivers to be able to use it for reliable storage. The only thing I have been able to determine is that, although the SATA interface is faster, there seem to be alot more read errors so the end result is slower throughput. Too slow for video capture/transfer which is time sensitive. I now have a 60 gig ATA100 that is dedicated for capture/processing and I only use the SATA for storage.

IMHO the original question is a little off base because it's not actually RAID that is causing the problem, but rather the implimentation of the SATA interface. I say implimentation, because the SATA spec is solid for any use (video or otherwise), but as we all know, different manufacturers offer a wide range of implimentations that "follow" spec.

[Malchiah]

IMHO the original question is a little off base because it's not actually RAID that is causing the problem, but rather the implimentation of the SATA interface.

Thanks. I will try a PCI SATA card and see if the drive performs better. That may at least narrow down whether the slow performance (dropped frames) is due to the SATA interface or the RAID. (Gadgetguy sounds familiar. You wouldn't be someone well-known, would you?)

Has anyone out there used stand-alone SATA drives for video capture or DV transfer? Have you had any problems?

[lordsmurf]

SATA is overrated too. Next gen, but not really "better" for most things.

A good IDE 7200rpm drive, regardless of location, as long as it is NOT the OS drive, is fine for MPEG, AVI, even DV transfer. Video is not that demanding on the drives themselves.

Fragmentation is a large concern for big files.

Analogy:
You're using "SUPER UNLEADED" gas for no reason, for some kind of perceived false ideal that "bigger is better". But the sad fact is your "car" is upchucking on it, and would have been fine if you had just given it "REGULAR UNLEADED" as suggested in the owner's manual.

[edDV]

I've helped a friend install a Promise SATA controller for his WD 10,000 RPM Raptor. It wasn't much faster than my 7,200 RPM ATA-133 200GB Hitachi. His scored 61MB/s, mine 55MB/s which is more than fast enough for me.

http://www.promise.com/marketing/datasheet/file/SATA150_DS_101703.pdf

BTW: Sustained transfer is better at the beginning of the drive (inside) so best to place capture partition there.