The (In)Famous Backblaze - Hard Drive Reliability Review for 2015

[lingyi]

Just remembered to check and Backblaze has posted their Hard Drive Reliability Review for 2015 here: https://www.backblaze.com/blog/hard-drive-reliability-q4-2015/

For those who aren't familiar with what this is, Cloud Service provider Backblaze posts quarterly reports about their experience (including reliability / failure rates) using desktop hard drives in their Cloud servers.

I highly recommend at least skimming through the entire blog post before posting "See...XXX manufacturer is best" or "XXX is the worst". This is ONE company's experience in an environment far better (cooling, regulated power) and worse (24/7 heavy data use) than anything that would likely be found at home.

It's not perfect, but AFAIK, this is only report and data of this scale available to the general public.

[SameSelf]

Backblaze's results are garbage. They deploy the drives in custom rackmount units that house something like 60 drives side by side. This is a very rough use case for drives due to high temperatures, vibration, etc. Ordinarily enterprise class drives equipped with special vibration dampeners, etc. are deployed in these environments e.g. database servers. But BB's data, presumably, is not accessed nearly as much as a db server since they are a cloud storage provider. So what BB is really doing is making the case that consumer drives can be deployed where enterprise drives are normally used, thus saving money. Their crusade of taking this data and translating it into which drive is more reliable for a consumer is misguided because these drives were never designed for such an environment.

[smitbret]

Backblaze's results are garbage. They deploy the drives in custom rackmount units that house something like 60 drives side by side. This is a very rough use case for drives due to high temperatures, vibration, etc. Ordinarily enterprise class drives equipped with special vibration dampeners, etc. are deployed in these environments e.g. database servers. But BB's data, presumably, is not accessed nearly as much as a db server since they are a cloud storage provider. So what BB is really doing is making the case that consumer drives can be deployed where enterprise drives are normally used, thus saving money. Their crusade of taking this data and translating it into which drive is more reliable for a consumer is misguided because these drives were never designed for such an environment.

Where/when did Backblaze ever mention enterprise drives? I don't get the Backblaze data bashing that whines about using Desktop drives in an enterprise environment. Backblaze has simply stated that they feel that it is a better value to use consumer grade drives than it is to use enterprise grade. They don't hide that. They never claimed that Desktop was any better or worse than Enterprise. They simply said, "Hey, we are using thousands of consumer grade drives. You might find some of our reliability data useful in determining which ones might be more reliable for you." For average Joe and I the data is a large sample stress test.

While the results may not directly correspond to drive reliability in home use it is certainly not garbage. Anyone that buys a HDD based solely on Backblaze data results is lazy but there are certainly some takeaways that are of great value. At this point it is pretty common knowledge in the tech world that the 3TB Seagate drives are simply not reliable. Are there people that are using them successfully? Yes, but there are tons of other horror stories out there. Backblaze was one of the first ones to the table with hard data showing that 3TB Barracudas just aren't durable and generally unreliable. I would never buy one and I wouldn't recommend that anyone else should, either. On the flip side of that is the consistently good performance of HGST drives across all sizes.

Finally, if you want to defend "Enterprise-class" design so vehemently, try to defend the NAS ready WD Red drives that are supposedly built for 24/7 use but still fail at a consistently higher rate than anything from HGST, occasionally worse than any of the Seagate desktop drives and surprisingly no better than their own WD Green drives. What can we take away from that? Not worth the money. I won't spend for one. I feel just as good spending less for a Toshiba that is just as reliable or the same amount for an HGST that is obviously more durable in an "extreme" environment.

If I load up a dozen each of Chevy half-tons, Ford half-tons and Dodge half-tons with 3 tons of dirt and drive it 20 miles across a bumpy gravel road and back and repeat it eight times a day for 3 years I will probably get a pretty good idea of the build quality of each truck. Were any of them made for this purpose? Nope, but if the bed separates from the body on seven of the Fords after doing this for 6 months but the Dodge and Chevy have only lost one or two then it doesn't take a rocket scientist to surmise that Dodge and Chevy might be building something with a little more durability and that if I plan on any kind of rugged use the Ford might not be my choice.

[SameSelf]

tldr, 500 words to my mere 100. U mad bro?

[hello_hello]

Backblaze's results are garbage. They deploy the drives in custom rackmount units that house something like 60 drives side by side. This is a very rough use case for drives due to high temperatures, vibration, etc. Ordinarily enterprise class drives equipped with special vibration dampeners, etc. are deployed in these environments e.g. database servers. But BB's data, presumably, is not accessed nearly as much as a db server since they are a cloud storage provider. So what BB is really doing is making the case that consumer drives can be deployed where enterprise drives are normally used, thus saving money. Their crusade of taking this data and translating it into which drive is more reliable for a consumer is misguided because these drives were never designed for such an environment.

You didn't read the study Google released several years ago? They don't use enterprise drives either.

While I'd agree with the point on subjecting the drives to a lot of vibration (assuming that's the case), Google concluded the amount of work a drive does has no correlation to it's lifespan, which probably makes sense given the platters are spinning whether or not the drive's reading/writing data. They also concluded that temperature has little correlation to lifespan unless it's consistently very high (over 55 degrees, from memory).

I'd really like to see some data showing "enterprise" is more than a marketing term, or enterprise means more than "charge more for an extended warranty".

tldr, 500 words to my mere 100. U mad bro?

10 words to someone's 500. Are you ignoring everything that doesn't suit you again?

[smitbret]

tldr, 500 words to my mere 100. U mad bro?

I brought a lot more to the conversation than you did.

[SameSelf]

Someone feels like a unique snowflake

[usually_quiet]

Just remembered to check and Backblaze has posted their Hard Drive Reliability Review for 2015 here: https://www.backblaze.com/blog/hard-drive-reliability-q4-2015/

For those who aren't familiar with what this is, Cloud Service provider Backblaze posts quarterly reports about their experience (including reliability / failure rates) using desktop hard drives in their Cloud servers.

I highly recommend at least skimming through the entire blog post before posting "See...XXX manufacturer is best" or "XXX is the worst". This is ONE company's experience in an environment far better (cooling, regulated power) and worse (24/7 heavy data use) than anything that would likely be found at home.

It's not perfect, but AFAIK, this is only report and data of this scale available to the general public.

I always find the Backblaze reports to be interesting. Thanks for posting the link.

[lingyi]

Backblaze's results are garbage. They deploy the drives in custom rackmount units that house something like 60 drives side by side. This is a very rough use case for drives due to high temperatures, vibration, etc. Ordinarily enterprise class drives equipped with special vibration dampeners, etc. are deployed in these environments e.g. database servers. But BB's data, presumably, is not accessed nearly as much as a db server since they are a cloud storage provider. So what BB is really doing is making the case that consumer drives can be deployed where enterprise drives are normally used, thus saving money. Their crusade of taking this data and translating it into which drive is more reliable for a consumer is misguided because these drives were never designed for such an environment.

Your points (and every other point or question I've seen posted here regarding Backblaze's reports) have been answered in the blog or comments. "From Hard Drive Stats – FAQ - February 2015 https://www.backblaze.com/blog/hard-drive-stats-faq/

Q: You use a variety of storage pods, doesn’t vibration matter?
A: While we do use different storage pods, the hard drives that we use are spread out throughout most of them. Typically our hard drives don’t experience a high level of vibration, but it is more than one might typically find inside a single hard-drive enclosure such as an external hard drive. Of course, some external hard drives are carried around in backpacks, dropped on floors, and such.

Q: Your drives are running 24/7/365 – is that detrimental to their longevity?
A: In our datacenter Storage Pods are “up” unless they need to be taken down for maintenance or there is a power outage. This happens infrequently as noted in our Smart Stats post when we talked about “Power Cycles”. This begs the age-old question: is it better for your hard drive to turn off your computer when you’re not using it or keep it running? Our vote is to keep them running.

Q: Why do you use consumer drives?
A: The redundancy built into our data storage model with RAID 6, along with our own data integrity checking, and our SMART stats monitoring, ensures that the data we store will be safe and available regardless of the drive models used. This allows us to purchase drives that have a lifecycle cost that is the least expensive when considering the initial cost, warranty, failure rate, product availability, etc. In short, Enterprise and Consumer drives both deliver the same reliability in our environment, so we choose to the drives that have the lowest lifecycle cost.

Q: Do your hard drive stats really matter for a consumer trying to make a purchase?
A: That depends. If you view our data center as a “worst-case” scenario for hard drives, then yes, you can surmise that any hard drive that a consumer would buy, would perform at a slightly better rate than our hard drives. You could also conclude that the drives that “survive” in our data center are really good.
If you don’t view it as a stress-test, then no, the data is merely an interesting results summary from a company with a lot of spinning hard drives.

Q: What about temperature?
A: Our hard drives are relatively cool, and don’t tend to overheat. We’ve looked at this in-depth and you can read about our findings on our “Hard Drive Temperature – Does it Matter?” blog post!"

There are additional links to the pages referenced in the excerpt above that can be accessed when viewing the actual blog post.

Note: to anyone who may be wondering...I have no affiliation or interest in Backblaze other than a interest and appreciation of the information they post. I highly recommend to everyone that they search for "hard drive reliability" on the blog to read much more about their continuing experiences with desktop HDDs.

[lingyi]

I always find the Backblaze reports to be interesting. Thanks for posting the link.

Thank you for your post and interest.

[SameSelf]

If you want to drink BB's marketing Kool-Aid, be my guest. Just realize they are trying to convince you to trust them with your data, so bias is present. But let me let you in on a little secret. Data managers around the world are quietly realizing that everything ever said about hdds was wrong (including RAID). The only thing that can be reasonably said with any certainty at the moment is that spinning rust is convenient. But one man's convenience is another man's liability. So, take that for what it's worth.

[hello_hello]

If you want to drink BB's marketing Kool-Aid, be my guest. Just realize they are trying to convince you to trust them with your data, so bias is present.

So what's their interest in offering data related to hard drive failures at all then? That sounds like SameSelf Kool-Aid to me.

But let me let you in on a little secret. Data managers around the world are quietly realizing that everything ever said about hdds was wrong (including RAID).

I don't think the fact that you simply make stuff up is much of a secret any more.

The only thing that can be reasonably said with any certainty at the moment is that spinning rust is convenient. But one man's convenience is another man's liability. So, take that for what it's worth.

What is it worth? I don't see how it means anything. What are you trying to say?

[lordsmurf]

It's really nothing new.

Hitachi good (but noisy).
Seagate good.
Western Digital not as good.

1.5/3tb = bad
2tb/4tb = good

That experience mirrors my own.

[Beck38]

My experience (I'll keep it to this century), both on desktop but primarily (as far as the number of drives running 24/7 in raid nas, 44 drives total there) has been top-notch for hgst (no helium yet but up to 6TB sata), very good to excellent for wd black; other wd (red/green/others) ok but nothing to write home about. Seagate? Close to pitifull, they are banned from my systems. Any review of their operation that puts them anywhere above that I'd be really concerned about their honesty.

[lingyi]

It's really nothing new.

Hitachi good (but noisy).
Seagate good.
Western Digital not as good.

1.5/3tb = bad
2tb/4tb = good

That experience mirrors my own.

LMAO! Such wonderfully delicate (i.e. non-offensive) phrasing in your post.

This from the usually highly opinionated (and deservedly so, given his experience) little blue man!

[lingyi]

If you want to drink BB's marketing Kool-Aid, be my guest..

It may be "marketing Kool-Aid", but it's brilliant and open "marketing Kool-Aid" that exposes the flaws, trials and costs of protecting a client's (and by extension, one's own personal) data.

But let me let you in on a little secret. Data managers around the world are quietly realizing that everything ever said about hdds was wrong (including RAID).

Please post links to the info that "Data managers around the world are quietly realizing..." as I (at least) would love to read empirical data rather than personal opinion.

The only thing that can be reasonably said with any certainty at the moment is that spinning rust is convenient. But one man's convenience is another man's liability. So, take that for what it's worth.

I take "convenience"= lower cost, something Backblaze (and I personally) acknowledge as the reason for using desktop drives. If there is any large (>2TB+) alternative to the "convenience" = lower cost of magnetic HHDs for storage, I (and I'm sure others) would love to hear it.

Edit: Speaking for myself, I count "convenience" = lower cost as being able to restore 2TB of data from one HDD to another in 6-8 hours without any intervention (i.e. optical disc switching).

[lingyi]

Came across this website, presentation and paper (yes, through the Backblaze blog <grin>) about SSD use and performance in Google Data Centers

https://www.usenix.org/conference/fast16/technical-sessions/presentation/schroeder

"Flash Reliability in Production: The Expected and the Unexpected
Authors:

Bianca Schroeder, University of Toronto; Raghav Lagisetty and Arif Merchant, Google, Inc.
Open Access Content

USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.
Schroeder PDF
View the slides
BibTeX
Abstract:

As solid state drives based on flash technology are becoming a staple for persistent data storage in data centers, it is important to understand their reliability characteristics. While there is a large body of work based on experiments with individual flash chips in a controlled lab environment under synthetic workloads, there is a dearth of information on their behavior in the field. This paper provides a large-scale field study covering many millions of drive days, ten different drive models, different flash technologies (MLC, eMLC, SLC) over 6 years of production use in Google’s data centers. We study a wide range of reliability characteristics and come to a number of unexpected conclusions. For example, raw bit error rates (RBER) grow at a much slower rate with wear-out than the exponential rate commonly assumed and, more importantly, they are not predictive of uncorrectable errors or other error modes. The widely used metric UBER (uncorrectable bit error rate) is not a meaningful metric, since we see no correlation between the number of reads and the number of uncorrectable errors. We see no evidence that higher-end SLC drives are more reliable than MLC drives within typical drive lifetimes. Comparing with traditional hard disk drives, flash drives have a significantly lower replacement rate in the field, however, they have a higher rate of uncorrectable errors."

Most of the paper went completely over my head, but as I believe published objective data is far more valuable than personal opinion and experiences, here it is!

On a loosely related topic, do a Google news search for "jedec temperatures and data retention". Almost exactly a year ago, a paper stating that SSDs MAY lose data under extreme heat conditions and non-use was re-examined. My take-away from the articles I read: Yes, it's POSSIBLE for SSD data to be lost under unlikely extreme heat conditions and SSDs are not intended to be nor used as archival media.

Edit: It appears the JEDEC paper apparently requires a paid subscription to download. Website here: http://www.jedec.org

[usually_quiet]

Researchers from CMU and Facebook did another flash memory failure study at Facebook, where they use large capacity enterprise SSDs. There is a synopsis here: http://www.zdnet.com/article/facebooks-ssd-experience/ with a link to the study itself. It also shows SSDs are not immune to data loss.