This is a follow up to my thread: https://forum.videohelp.com/threads/395144-Please-explain-the-advantages-of-a-NAS-for-...rs#post2566747
There are some members who consistently promote RAID for home use and I don't understand why I or many (most?) home users would need it. Please advise why a RAID would be an advantage for most home use. I'm especially concerned about claims that RAID is an alternative to traditional backup.
FYI, I'm using PCs since 1986 and remember when RAID (Redundant Array of Inexpensive Disks back then as I recall) was first available as an option for home use. Striped RAID was necessary for heavy data use (video capturing in particular) and Mirror RAID made sense because hhd transfer speeds via IDE were much slower.
To the best of my knowledge, no matter the type of RAID system, traditional RAID, ZFS, UnRAID, etc. There are only two RAID modes, Striped and Mirror and every RAID configuration is a variant of those two modes.
As always, please correct any info that I may have wrong.
Striped RAID, where the writing of incoming data is split between two or more drives for a speed increase is unnecessary since the introduction of SSDs. With the exception of extremely heavy data high speed data transfers such as 4K, 8K or beyond video capture, there's to home user application that can't be achieved with a single SSD.
Mirrored RAID, where the incoming data is written to the two or more hdds (I remember somebody actually did a floppy disc RAID!) simultaneously. Some RAID systems can perform a parity check as the data is written. However, this "instant backup" has a major flaw. If the data is corrupted during the transfer, permanently deleted or overwritten, the data is completely lost just it would be on a single single. This is a critical reason why RAID is not a backup solution. With a traditional backup, the user has the ability to restore a known good copy of the lost data.
Parity and rebuilding. This is the one area where I see RAID has an possible advantage, though not necessarily required for home use. RAID 5, 6 and 10 (1+0) use a parity drive(s) that allows rebuilding (slow to very slow) of a failed drive(s). Data from the remaining drives can be restored to a replacement up to the moment the drive(s) faild. But again, any corrupted, deleted or overwritten files are still permanently lost.
Viruses and ransomware. IMO, a hugely critical factor against RAID, especially without an offline backup. Any storage, local or networked, even Cloud, can be attacked by a virus or ransomware. I worked for two companies, one large and one small that where the data drives (on a local dedicated server for the large company) that were hit with ransomware. At the large company, the shared drive was attacked and everything prior the the previous night's backup was lost (I kept my current work files on my work PC, so wasn't affected). At the small company, six (6) months of accounting data was lost and had to be reentered.
Cost, especially for large multi drive collections. As I've stated before, my video collection is fairly large (10's of terabytes and multiple hdds) and I have a main setup and an exact backup of everything on a second set of hdds that will never be connected to the main PC (I'll walk the main drives to the secondary setup for transfer/file sync). I consider my original DVDs and Blu-Rays as my second set of backups. To create a mirrored RAID (single mirror drives + one for parity) configuration would cost thousands of dollars for little to no benefit over what I currently have.
Verification, file sync and parity. When copying files, I use Teracopy with verify on. It adds about 50% additional time for the transfer, but Teracopy will give me a list of files that failed verification during the copy process. For after copy filesyncing, I use ViceVersa (the pro version that adds some file exception options. I use it to compare the files side by side and keep copying and deleting the files until ViceVersa says the source and target folders/drives are identical. I could turn on file verification to generate a SHA hashfile to doublecheck, but usually don't bother since Teracopy already verified the copy/move. Something that I haven't been doing, but will for my most critical data/document files is use a Par program like Parchive to create a .par files that will allow me to recover corrupted files files.
So, back to the top question. What the advantage of RAID for me or other home users?
+ Reply to Thread
Results 1 to 30 of 45
LOL if you have a working set of drives and a backup then you already have a RAID 1 array, just not in a box or that updates the backup automatically.
my 3 nas boxes are behind a firewall and not allowed outside contact from the internet. they could possibly be attacked through one on the networked computers but in the 7 or so years they have been up no one has hacked into any computer here. maybe just lucky, i don't know.
i just like the convenience of having everything always accessible to all devices(computers, phones, tablets, tvs, bluray players, fire tv boxes, chromecasts, etc.) on my local network, and not losing any data if one drive dies. hasn't happened yet, but i have a spare matching drive for each nas just in case.--
"a lot of people are better dead" - prisoner KSC2-303
Raid is more about speed and/or data availability.
Raid 0 is speed. If your application can not saturate a single device then you have no need for it. The major disadvantage is that the failure of one device means a loss of all data.
Raid 1 trades write performance for hardware redundancy. There is no write advantage as the same data needs to be written to all devices but reads can be satisfied by any device. Failure of one device causes no data loss and the remaining device can continue to read and write data while rebuilding the replacement device. Reading is somewhat slower as there are fewer devices to respond.
Raid 10 is striped mirrors. It tries to give you the speed advantage of raid 0 with the availability advantage of raid 1 at the cost of double the devices, no
parity is involved.
Raid 5 & 6 try to balance speed, availability, and costs. Writes are still somewhat slow as parity needs calculated but is spread across all the devices. Reads are fast as any device can satisfy the request. Availability is high because one device failure doesn't mean the loss of all data but it does mean reads are slower because the data needs reconstructed. The other advantage is fewer devices are required than in a mirrored setup.
Viruses and ransomware have no bearing on the use of raid, any device attached to the system is vulnerable. Raid is not a backup system though it can save your data if hardware fails before you get it backed up.
Most of the advantages that you would get from raid would be satisfied by an ssd, they are significantly more reliable than spinning disks so mirrored setups are rarely needed and their speed makes striped arrays are mostly unneeded. Parity raid probably has the best use case for home, more space and protection against hardware failure.
So here is one scenario where raid would save the day. You've invited all of your friends over for a Star Wars marathon, all 9 episodes. You've spent hours copying the files to your HTPC, taking 50% longer so you can verify the copy is good. The popcorn is popped and every one is ready for the show to begin. You hit play on your remote and you get an error message - failed to read disk in drive D. Now you get to tell everyone the will be a delay while you copy the everything from your backup to a new drive, hopefully it wasn't the os drive also. If your movies had been on a raid 5 array, you may have gotten an notice that it had degraded. You would hit pause, go and replace the failed drive, return and hit play and the marathon would continue.
Raid won't protect against the user file deletion, virus attack, or bad controller.
It will provide in all other cases, instant copies of the data to 2 drives or more.
Because drive failure, not controller failure, occurs far more often, this eliminates the biggest issue facing home users.
Then, like any single drive user, an offline backup to a drive kept offline and unused, on an occasional basis provides the backup in case of main drive/raid virus attack, user deletion, etc.
Again, these assume you the user aren't mistaking deleting files often, your pc isn't being subject to virus issues often, etc.
For the majority of home users, 1 main drive backed up to 1 external backup drive isn't enough because they often lose the main drive data before having a chance to backup due to drive failure.
Raid prevents that with dual drives.
Now, if you really want to go crazy, you'd have to have a backup drive setup to be write-only (no deletion), with file history mirroring, with multiple drives to prevent serious attack failures and such, but that begins to be far more than necessary for home users and more complex and costly.
In the end, depends on the user and data.
How much complexity and cost?
Dual drive raid to single external backup drive is usually sufficient to retain the basics like taxes, office documents, etc.
For home video collectors that go crazy with 10+ TB of rips, well....that's another story depending on how sure you want to be.
Might as well start looking into mirrored cloud backups or off-site nas mirroring since any fire would nuke all on-site copies.... Not to mention primary mirrored raid backed up to mirrored raid external drives.
At that point, you're looking at windows server clusters because it has all that built in - continuous mirroring, offsite, automatic healing and new drive restoration, etc.
Other than if (which I wonʻt do anymore for security) I were to mount my backup drive(s) (in reality my main drive(s) since theyʻre connected to my laptop for playback) to my main PC, theyʻre not logical units at all. One set of drives is a backup and are inaccessible to the main PC (all three of my PCs are not networked to each other).
As I stated. RAID in any configuration is not a proper backup. Mirror RAID 1, 5, 6 and 10 (1+0) does mirror your data at the moment you write it, but again, permanently delete, move, overwrite or format one drive, and the rest of the drives, no matter how many there are in the mirror are in EXACTLY the same state as the original, GONE.
Your network accessible NAS(s) have nothing to do with RAID, other than your NAS may bet setup with RAID and the question posted.
Last edited by lingyi; 20th Jan 2020 at 02:16. Reason: Duplicate paragraph
Recovery from a hardware failure is a possibility, depending on what failed. One or two (in RAID 6 or 10 (1+0) hdd failures. Yes, you can probably completely recover the data. Controller error that corrupted the RAID, possibly complete loss of all the data in the RAID. Linus Media Group (Linus Tech Tips) went through a major scare when their Petabyte Project (1PB of RAID storage), went down due to a controller error while they were performing a backup of the array. Tens or hundreds of terabytes of data was potentially lost.
As for your scenario. about the movie night, if the start time of the showing was time critical, you could just as well use the backup for showing and copy it to the drive you were planning to use for the showing after. If something happens to the backup during or after the showing, you have the secondary backup on or offsite on another drive or the original discs.
If the showing is not time critical, rebuilding a RAID takes longer than straight copying.
For copy verification, while RAID may alert you of an error in hardware, but so would Teracopy with verify on during the copy process.
Regarding replacement of the hdd in the RAID. Since Mirror RAID requires at least two hdds, with the spare that means having three drives onsite. With three onsite drives, I could have the original, a primary backup and a secondary backup not attached to a device that may have had a hardware failure that may corrupted the mirror drive also. [And may corrupt the replacement drive also].
Last edited by lingyi; 19th Jan 2020 at 18:21.
weird argument. true, raid isn't a backup, per se. raid backs up the data on the array. nothing more or less. your money, your call. mine is on nas boxes, not multiple time consuming individual hard drive backups. i put a file on a nas and it's going to be there even if a drive drops out without any work from me other than swapping out a drive to restore the array.--
"a lot of people are better dead" - prisoner KSC2-303
RAID is not needed for home use today.
Extra speed would only be needed if you are capturing uncompressed HD. Very unlikely anyone at home would do that, and I don't think it is done very often commercially.
Redundancy and "hot backup" is sometimes needed for servers, but not for home use. Just backup your files regularly, or use a background backup app that backs up files as you work and save.
Yes, I forgot to add this to my OP. IMO, the best thing that RAID does is protect itself, with data integrity a secondary function. Built in software and for server/enterprise use, hardware redundancies (I think some server grade RAID hardware even has hot swappable RAID controllers). If I did need a RAID, it would be in a standalone enclosure (NAS capable or not) to keep it isolated from hardware failures on my PC.
As I remember, when RAID first came to the home market (at a cost far, far beyond anything today), it was only available in a dedicated SCSI enclosure, could only use SCSI drives and needed a dedicated add in controller. IDE RAID was never really a thing, not only because SCSI was much faster, but motherboard IDE connectors were (I believe) limited to four (4), eight devices total, but since IDE defaults to the speed of the slowest device, so all the drives had to be identical, ideally down to the model.
At the risk of being pedantic, RAID 0 is not really RAID because there is no redundancy, the term RAID 0 is a misnomer.
But I agree, for most people RAID, in any form is no longer needed, especially with the advent of NVMeS.
Even a basic 6.5K Alexa pushes 732MB/second for a single track of video.
Multiple tracks are often used in films, so even a Samsung evo 970 m.2 can be overwhelmed. Sure, 980 is coming, but far cheaper to simply raid stripe a handful of cheaper ssds together to create an even faster raid drive than any ssd in existence for a reasonable price.
I use raid 1 + 0 double parity and striping. speed is the reason and a semi-backup system. if you have the access to it why not use it. I use enterprise hdds (sas) with a raid card that was 30 bucks. I made my nas boxes from computers and the system works great. the only raid arrays I wouldn't use are 1 and jbod no backups on them
b) Even if we did, 1 single program broadcast in 8K does not necessitate striped SSDs, unless you were planning on capturing the stream in full 8K using a lossless format or lightly compressed codec such as ProRes.
An 8K broadcast would require sufficient bandwidth to stream it and an 8K monitor to view it on, something I doubt more than a handful of people in the world currently have.
Even with 4K, most U.S. based content providers do not broadcast in 4K, I don't think there's any TV show in the U.S. that is recorded in 4K and even services wuch as Net Flix only use 8-16 mb/s for their 4K content.
On a system with the cheapest SSDs i could buy, one for reading and one for writing, 16GB DDR3, a 4790 based Xeon and a GTX1050 I can edit and work with most 4K and 6K sources, though jpeg2000 and some ProRes variants are a beast to decode and severely bottleneck this system.
Don't waste your time with RAID, unless you have a roach problem. Just pick up one Intel 720P for reading and one for writing and call it a day.
4K, 8K, and RAID are all pretty much a waste of time and money for the average home consumer.
I already addressed RAID, and sophisticles is correct that there is virtually no broadcast or streaming content for 4K. What's more, on a typical 55"-65" display, you can't really see any difference between HD and 4K when viewed at a ten foot distance. Even with a large screen, the improvement is minimal.
Like 3D, curved screens, and other innovations of the past twenty years, 4K has not exactly taken the world by storm, especially when you compare any of them to the true consumer electronic "hits" of the past forty years, like videotape, CD audio, DVD, satellite video delivery, audio streaming, and video streaming.
I use it because its there, I stream all over my house multiple places. I do not have a problem getting anything across my gigabit home network nor any gaming that occurs through lan. I can also stream outside my network without problems (barring the connection where im at). I doubt my sas drives cost anywhere near 2tb drives from anywhere else as a matter of fact they were quite less cost than hdds that have a continual use rating at most stores/ internet shops.
The only thing worse is suggesting RAID-0 or JBOD for non-scratch/temp use.
However, this "instant backup" has a major flaw. If the data is corrupted during the transfer, permanently deleted or overwritten, the data is completely lost just it would be on a single single. This is a critical reason why RAID is not a backup solution. With a traditional backup, the user has the ability to restore a known good copy of the lost data.
It's here, been here, arrived.
It's already a mastering minimum for many - E.g. Netflix requires all submissions to be 4K source or better, 4K delivery at minimum. Superbowl is 100% 4k. Everything Amazon Studio makes has been 4K since 2013.
Every top smartphone from Apple to LG to Sony to Huawei to Samsung has been able to record in 4K for years.
The main reason the USA hasn't seen that much 4k include:
1. The slowest broadband speeds of a 1st world country.
2. The TV broadcast standard ATSC 3.0 was just approved in 2019, years after other countries. https://www.digitaltrends.com/home-theater/atsc-3-0-ota-broadcast-standard-4k-dolby-atmos/
If it takes a decade like hdtv to be adopted, we might not even see 8k broadcasts until 2030 in the usa (already started in Japan last year).
3. Thus, no 4K atsc 3.0 tuner TVs out yet - you have 4K TVs receiving 2K broadcasts due to older tuners out today.
4. No widespread 4K uhd bluray acceptance.
Lots of Americans still on old DVD players.
SATA hard drives are the only "affordable" way to build a 16TB mirrored raid cheaply today. SSD drives would practically quadruple+ the cost and drive bays required.
Just for my media and non-media files alone requires at least 11TB of storage.
12tb hdd $320~
2tb ssd $200~ x 6 = $1200~
So, single set of 12tb worth of ssd with no raid, no backup.
Or 4 12TB hdd in dual raid 0, each pair backed up to the other for the same price.
4K as an acquisition media (i.e., using a 4K video camera) is a totally different conversation. While there are some issues about editing 4K, that can be solved by using proxies. The advantages of having extra resolution, in order to crop and zoom in post, is one of many advantages.
So, 4K for acquisition: yes. 4K for watching on a typical living room LCD: no.
i don't know about you guys but i prefer nas boxes over individual hard drive backups as they are neat and tidy and i can put them in the garage, out of sight and no noise. drive backups have to be plugged in, files copied, and stored somewhere.... i have eSATA so it's not a really big deal here but still a pita. storage well that's another story. LOL
[Attachment 51573 - Click to enlarge]--
"a lot of people are better dead" - prisoner KSC2-303
Keeping and maintaining my non-RAID, non-NAS collection isn't a pain at all!
[Attachment 51582 - Click to enlarge]
The five drives on top are spares. And the backups (connected to my main PC) are four bay enclosures instead of the eight bay ones connected to my laptop. Noise and heat aren't an issue because I have to keep my A/C on whenever I'm home. Can't open the windows because of noise, heat and odors and don't want natural light because it affects viewing.
Okay, so what we have so far is the only advantage that RAID has for home is that in mirror mode, it can offer near instantaneous pseudo-backup for file writes (at a slight speed loss) and the possibility of data recovery with a parity drive to rebuild the array in the event of an hdd failure. Seems like little gain for at least 2X the cost of a single drive setup.
So the way I understand it and have always understood it, please correct me if I'm wrong or missing something. Home RAID Pros and Cons are:
- Automatic mirroring (with mirror RAID configurations) of writes at a slight speed lose due to having to write to two drives simultaneously, or in the case of ZFS, write the buffered data after the write action is completed to the drives.
- Possibility of data recovery IF the failure is due to hdd failure and not device (enclosure, MB) failure.
- Offers some level of file transfer verification on controllers/systems that support it.
[LIST][*]Costs at least 2X more than a single drive setup since you require at least one additional drive for true backup.[*]Offers zero advantage over a single drive for accidental file deletion or overwrite and accidental formatting or partitioning. (1)[*]Offers zero additional protection against viruses and ransomware both, five, ten drives in a mirror or striped configuration will be corrupted/locked simultaneously. (2)[*]Takes longer to rebuild in the event of a drive failure than a restore from a separate backup drive. (3)[*]The pseudo-backup can easily duplicated with a few clicks, copy/paste the file to another drive or partition at the same time you start the write operation and can start/schedule a backup immediately to a backup drive to keep a third copy of mission critical data. Using programs like Teracopy and ViceVersa you can verify that the file transfer went well as both allow the generation of SHA HASH, on the fly for Teracopy, after the fact for ViceVersa.
(1) I posted about a month ago about my latest brainfart moment. Quick formatted a new 12TB that I had just transferred 9TB of files to from 4 & 8TB drives. I didn't properly rename the new drive and formatted it instead of the 8TB. 9TB of files gone in an instant! Sighed but didn't panic because fortunately the drive I just formatted was the spare backup to the backup. Started over and 18 hours later, back to as if nothing happened!
(2) My personal strategy that I just realized I should use is to never let my main PC see my backup drive. I'll carry the main drive another PC for backup/file sync. Not 100% certain the backup drive won't get hit by a virus or ransomware, but far less likely to be hit if anything on the main PC doesn't know it exists.
(3) It was stated that you can still use the drives during a rebuild and while that's true, it's not recommended due to potential data corruption and adds to the rebuild time. Since RAID was designed (and used in professional operations) for always always available access, RAID controllers usually allow dialing in how much time is available for the rebuild. Here's a good website that useful for calculating rebuild and file transfer times https://www.memset.com/support/resources/raid-calculator/. As I stated above. 9TB data transfer with verification with Teracopy took ~18 hours at ~120-130MB/s.
These two articles from OnTrack, know for their data recovery software and services are a great read about RAID rebuilds: https://www.ontrack.com/blog/2013/02/11/raid-vs-drive-rebuilds/ and https://www.ontrack.com/blog/?s=raid+rebuild
From Part 1. Note the "B" word!
How can you prevent this from happening and what can you do if this happens to you? The best way to prevent data loss is to create sound backups. Test them often to ensure that if you have a drive failure, your backups will help you to recover from a failed RAID rebuild. In the event the RAID array goes into a degraded mode, stop all activity on the volume and take a backup immediately to prevent data loss if a second drive fails and takes down the entire array. If you are unable to take a backup, then clone or image all of the disks before rebuilding the array. These images will preserve the data on the disks in the event the rebuild fails, allowing for a full recovery of critical data.
I love watching the background of Asian variety shows because that's often where and when the cast and guest are most natural. One of my favorite moments is when a celebrity guest made a sour face because the other participants in a game beat her to answer a question, but was smiling when she knew she was on camera.
Today, with a fast internet connection, you can stream live changes to an Azure, Backblaze, Carbonite, etc cloud storage account to achieve the same live data protection as RAID mirrored.
As for separate, offline backups, RAID never was designed to handle those situations where the data is legitimately deleted by the user, or not by virus. That's a command issued to the working drives - raid only cares about the health of drives.
Only ways around the virus or user error are offline, occasionally backups (E.g. One drive per day, rotated backups over the week + monthly drive backup - if you're trying to be super cautious), or build out a continuous, revisioned storage system that stands independent of your pc (thus, only accepting commands for data writes and reads, and hopefully ignores viruses on the network).
Old days was a simply ftp account, write only, no directory listing, on a seperare storage device - can only add, never delete data from remote.
Today, you've got Windows Server clusters and more.
Even nas can auto backup to local nas and cloud.
A way to minimize costs.
Two drives for the maximum storage space needed with spare room.
One internal, live. One external, offline backup.
Then, a single ssd drive of sufficient space to hold temporary expected data long enough between occasional backups to external drive. After backup, move data to the internal drive.
The much higher reliability of the ssd will greatly reduce chances of non-backed up data loss between backups, while you still benefit from only spending money on 2 huge hdds for all your data.
(Both big drives are vulnerable to viruses, but let's assume you've got a clean system. If not, and you often get infected, a continuous file revisioning backup system is needed.)
Idea others have of multiple ejectable drives in a tall server case works fine, too. If you've got the money, nothing like a rack of 10 drives =D He he.. SATA docks, shuckable WD external red drives, slickdeals.net sales....