How to check file integrity of data burned to optical discs

[lingyi]

A hash is no better than a a byte-by-byte file compare. Generating one is only useful the next time(s) you want to check the file(s) -- you won't need to read both the original and the copy, just check the hash of the copy.

I understand and agree that a file compare is fine for most situations, such as when I use Teracopy to copy/move files with verification on. No errors, I'm golden.

But a saved HASH can be a godsend in the scenario I've given above. Where the DVD is used to do a restore to a failed drive. Without a HASH to verify the files being copied to the new drive are exactly the same, you're copying the unknown to your new drive. I find this particularly interesting because of the number of posters on Reddit who treat optical discs, particularly M-Disc, "It lasts for 1000 years!". IMO, a disc that lasts one day with a verified copy of a file is better than one that lasts 1000 years with a corrupted file.

It seems the issue isn't easily or definitely solved as shown by a quick Google search. Just when I think I read about something that works, someone points out limitations. If I find my spindle of blank DVDs, I may give some of the other programs a try as they sound promising. But it would be just for kicks.

Heck, I if find something that does works well, I may actually tell gamey about it! But I have to be 110% sure of how and why it works before opening up that box!

[lordsmurf]

I explained that the only way he could do that would be to copy the files from his discs and perform a compare against known good files not on his corrupted hard drive.

Sometimes he asks good stuff, and I try to help, but I admit to "tuning out" at times, or skimming.
But reading this here, I thought of something...

Usually corrupt files get more corrupt. So even if the burned files are corrupt, there are good odds that size/hash mismatch will reveal bad files. Neither is good, both are bad. It's also safe to assume adjacent files are bad, usually logically more files in the same folder. So the whole folder is then suspect.

The ability to visually map the drive can help, and certain defrag programs can help show that (but do not defrag!). But simply launching a defrag program can piss off a bad drive, so beware.

I've done all of that, for HDD data recovery needs.

Then he dropped the bombshell. He not only deleted the files on the bad drive, but he's unable to get them again! Back and forth with him asking if he could compare the four disc copies he has against each other and my explaining that all that would do is prove that all four copies have the same possibly bad data. *SIGH*

He must've mixed up the ziplock bags. Plain bread in one, peanut butter and tuna on disc in the other.

. IMO, a disc that lasts one day with a verified copy of a file is better than one that lasts 1000 years with a corrupted file.

But I have to be 110% sure of how and why it works before opening up that box!

Schrödinger's cat. The data is both good and bad until verified.

maybe lordsmurf, lordsmurf, lordsmurf (LOL) will show up

[Attachment 55490 - Click to enlarge]

[lingyi]

Seeeeeeee...lordsmurf IS Beetlejuice!

"He must've mixed up the ziplock bags. Plain bread in one, peanuts and tuna on disc in the other."

Nahhh...it's those darn AliExpress bags that draw in moisture!

Seriously though, this thread gave me new respect for gamey. Well, not new, just some as I seriously wonder about those at who talk about how optical discs are so great for backups. As you said, Schrödinger's cat, especially the absurdity of it, exacerbated by the difficulty of being able to verify data on the disc.

[johnmeyer]

Seriously though, this thread gave me new respect for gamey. Well, not new, just some as I seriously wonder about those at who talk about how optical discs are so great for backups. As you said, Schrödinger's cat, especially the absurdity of it, exacerbated by the difficulty of being able to verify data on the disc.

It is NOT difficult to verify the data; it is, in fact, trivial, as you have been shown by multiple people in this thread. You just refuse to accept the obvious.

DVD and CDs are, and continue to be, the best option for long-term storage. No one reading this thread should be misled by someone who doesn't know what he is talking about.

Name one other media storage option that will last longer, based on actual scientific accelerated age testing. By contrast, I have provided, on many occasions, links to CD and DVD accelerated aging tests that indicate that the data should be readable in 100 years, or more. The data on a disk drive may also last that long, although the mechanics may bind up. I've had to open up disk drives to free the disk that had frozen. Theoretically you are supposed to do this in a clean room, but if you're desperate you can do it without the clean room, and get the disk to work again.

[Cornucopia]

You can use ISOBuster to read & create MD5 hashes of written discs (both optical and HDD/SSD).

BTW, "accellerated aging tests" are still just an extrapolated approximation, but have been pretty reliable. 100 years? I don't know. But in 3 years I will have in my CD collection 12 discs which I will have bought 40 years earlier when getting my initial trial of CD technology. ALL of them still sound as pristine as they did the day I got them. I've gone through about 5 or more players in that time period, though!

Of course, there's a big difference between stamped (pre-recorded) disc technology and quality vs. burned (dye or phasechange) (re-) recordables. Certainly if lingyi is putting them onto the discs himself, they would be of the burned category, so the longevity is not as assured.


Scott

[johnmeyer]

The aging tests I used to link to were for burned discs (i.e., using dye). There was a pretty big difference in outcomes depending on the dye and manufacturing techniques used.

You can actually track the detorioration of any discs you own using DVD Speed. It will report on the total number of errors corrected. As time goes on, that will likely go up. However, if after a decade you can't see much deterioration after a decade, then the disc will probably outlast you.

[lordsmurf]

DVD and CDs are, and continue to be, the best option for long-term storage.

DVDs? Sure, maybe, those can be. If the media is good. (Hint: gold is crap, MDisc is suspect.) The real problem will be readers and players.

CDs? No! No, no, no no no! The discs are really fragile. Simply rubbing it too hard with a fingernail can screw the disc. Unlike the plastic sandwich that is DVD, the CD layer is easily exposed. It only has a thin coating of goo between the data and oxygen.

Name one other media storage option that will last longer, based on actual scientific accelerated age testing.

"Advanced aging tests" are largely BS beyond about 50 years of extrapolation.

Optical discs really won't last any longer than film, magnetic tape, or even paper. The polycarbonate layer, for example, will tint (usually yellow) over time. That can turn so-so readability into non-readability really quickly. Some of my oldest CD-R has slight tinting, especially anything that was aged in the car trunk CD player.

Furthermore, lots of data will mostly be unusable in a few decades anyway, even if "good". I have lots of data from the 80s and 90s that's essentially worthless, no real way to access it. Files that can't be opened, disks/discs/tapes without drives (or expensive to acquire). I actually still have a box full of punch cards (a few may be missing now, jumbled up).

I've done a lot -- lot, lot, lot lot lot! -- of optical media tests in the past 20 years. Those cherry-picked lab longevity tests don't bamboozle me whatsover. I know better. Real-world and lab just is not the same whatsoever. (Psst... that why media companies come to me. Sorry, NDAs.)

Theoretically you are supposed to do this in a clean room, but if you're desperate you can do it without the clean room, and get the disk to work again.

Yep. Not good, but it can work. Been there, done that. Data recovery for $1k+ not needed, backups available, but proof of concept.

[johnmeyer]

The only legacy of a long-forgotten US vice president was the line "nattering nabobs of negativism." It sure applies to that last post. It is all negative without one constructive idea of what would be better.

The idea that accelerated aging is useless is utter nonsense. As an engineer, we did this all the time at Hewlett-Packard, where I worked at their main test and measurement facility back in the early 1970s. Obviously you can't wait 100 years to know if your predictions are correct, but you CAN wait a few years and check whether what was predicted is starting to come true. Also, if you have studied science and apply scientific principles, such tests are not "BS".

Optical discs really won't last any longer than film, magnetic tape, or even paper.

Gosh, I never said it would last longer, but if it lasted just as long, that is all we want. I have linked many times to movie film I transferred from 1928. It was in as good shape when I transferred it ten years ago as when it was taken.

I have transferred generation one audio magnetic tape that my father recorded in 1948 just before I was born. It is 70+ years old and is not shedding, and the audio is pristine.

And I have no idea what you mean by "even paper." Our entire recorded history is on various forms of paper, and those have survived centuries, and some have even lasted millenia. Newsprint and cheap paper don't last, but even half decent paper will last for centuries. As one example, I scanned my grandfather's diary from when he was an army chaplain in WWI, back in 1917. The pages were a little yellow and a little brittle, but still very much intact.

Finally, you say you have data from the 1980s that you can't access. Is that because the media has failed, or because you no longer have hardware to read the media? I suspect the latter. If you really needed to get to the data, you could still find hardware. I still have QIC tape drives and all sort of other tape drives, and Zip disk drives, etc. All of them still work.

This is another reason to stick with DVDs as an archival medium. Because of DVD's success, there will be hardware to read it for decades to come. As one example, the CD was introduced in 1982 and is now more or less obsolete, but any hardware that reads round shiny objects can still play them, almost forty years later.

[lingyi]

I'll add ISOBuster to my try someday list. TY!

As for media longevity, I've given up worrying about it years ago. My strategy is to is replace and re-backup often. I only backup to hard drives and replace them soon after their warranty is over, replacing them with cheaper and larger drives, then move my main drives to additional backup status. Not cheap, but effective.

[johnmeyer]

As for media longevity, I've given up worrying about it years ago. My strategy is to is replace and re-backup often. I only backup to hard drives and replace them soon after their warranty is over, replacing them with cheaper and larger drives, then move my main drives to additional backup status.

I do this as well, although I do it mostly just to make the data more easily available without having to haul out some ancient piece of hardware. I have some data that started out on 5-1/4" floppies and went through about five different formats before ending up on one of my SATA bare drives.

The oldest digital format media I have are the "popsicle stick" magnetic cards for my HP-67 programmable calculator that HP gave to me when I left for graduate school in 1976. It holds 224 "lines" of code. I still use it almost every day. Decades ago I wrote a summary of my experience: HP-35 Unfair Advantage (plus HP-67 story)

[lordsmurf]

The only legacy of a long-forgotten US vice president was the line "nattering nabobs of negativism." It sure applies to that last post. It is all negative without one constructive idea of what would be better.

The opposition to overly glowing optimism (orgasmic, even) isn't necessarily negativity -- it's realism.

I'm a realist.

DVD media -- a quality DVD media -- is indeed one of our best options. But the same can be true of film, paper, and magnetic media. DVD is just "a" option, not the "best" option, and certainly not the "only" (or "only good") option.

But I'm forward looking. I don't just see what's in front of my face: the disc. I think in terms of the data "on" the disc.

The idea that accelerated aging is useless is utter nonsense. As an engineer, we did this all the time at Hewlett-Packard, where I worked at their main test and measurement facility back in the early 1970s. Obviously you can't wait 100 years to know if your predictions are correct, but you CAN wait a few years and check whether what was predicted is starting to come true.

And when does that happen with optical media marketing? Essentially never. One and done. Very often, buried in docs, it's scaled back later. I've seen "100" years be clawed back to 40 a few times now.

And I have no idea what you mean by "even paper."

Paper is usually ephemera. Hence "even".

Finally, you say you have data from the 1980s that you can't access. Is that because the media has failed, or because you no longer have hardware to read the media? I suspect the latter. If you really needed to get to the data, you could still find hardware.

Example = programs that will not run on modern hardware, and the old (not even legacy, just plain damned old) hardware needed to run the software is scarce to non-existent.

I know you can "still find" many things. But it often costs a shit-ton of money, and even major corporations and governments wrestle with "is it worth it".

Another example = a single roll of GAF film is nearly impossible to develop, toxic chemicals, with only one lab in the entire world that can do it -- and it costs about $200 last I checked. $200 is cheap, too, when it comes to data recovery of old formats.

Betamax is a popular widespread format that becomes harder and more expensive with each passing half-decade. VHS has started to move in that direction, and in 10-20 more years won't be a pretty situation. I'm sounding the alarm on this now, and have been in contact with serious hardware developers to attempt to fend this off, kick the can down the road a few more decades.

Because of DVD's success, there will be hardware to read it for decades to come.

Pfft. I really, really doubt that. We've already heavily shifted to a non-optical worldwide society. Let's talk in another 10 years, see where we're at.

Idea for exception = if optical media manufacturers also get into optical drive manufacturing. Then we may have many decades.

As one example, the CD was introduced in 1982 and is now more or less obsolete, but any hardware that reads round shiny objects can still play them, almost forty years later.

Due to backwards compatibility of optical media, period. But again, we're move towards being a disc-less society.

What the means is this: If you're worried about data retention, and longevity, you'd better grab a nice stash of optical drives, and the gear to power them. Right now, there are many "dead" DVD recorders with no potential replacement opticals. You must cannibalize a working deck. However, also realize that failed drives are one of the more common fail points for the gear. Combine this with the fact that many HDDs marry the recorder hardware (not portable to another unit), and you'll find yourself cannibalizing a good unit (likely not cheap, either) to recover the HDD. That's just one example, but put that on a macro scale.

"Cloud" is next thing, but bandwidth mostly sucks, and security of data is still a concern. (Noting that "cloud" is just hard drives in a datacenter, hopefully redundant.)

When you want to talk about data longevity, the discussion isn't just about blank media, but the contents. Not just integrity, but usability.

Sure, we have documents/whatever going back centuries, but the language is forgotten. So WTF good is it? Not everything has a Rosette Stone.

Are you seeing my macro point yet?

You must look beyond the shiny flat spinny thing.

long-forgotten US vice president

The crooked VP of a crooked Nixon? Not likely. That SOB will be in history books forever.

[abolibibelot]

Recently found this, thought it would be a worthwhile addition to this discussion.

http://stahlworks.com/dev/index.php?tool=md5list

If you're storing backups on DVD or USB stick, you may turn them into self-verifying archives by adding an md5 list, sfk.exe, and a short batch file in it's root directory. For example, create a batch file "90verify.bat" that contains two commands:

91tools\sfk md5check 91tools\md5sum.txt
pause

Then create a directory 91tools, copy sfk.exe into it, and run

sfk md5gento=91tools\md5sum.txt

Test the whole stuff once before burning, by running 90verify.bat. Then burn everything onto DVD. Insert the DVD into another PC, double-click on 90verify.bat. This way you verify that your DVD can be read also on another machine, which is an important test for your data's safety.

This sfk tool (a 2.4MB .exe file) is tremendously versatile and can also perform comparison, search and replace, data extraction, and other stuff I haven't yet explored.



Besides, as I'm catching up with the newer replies...

@johnmeyer

DVD and CDs are, and continue to be, the best option for long-term storage. No one reading this thread should be misled by someone who doesn't know what he is talking about. Name one other media storage option that will last longer, based on actual scientific accelerated age testing. By contrast, I have provided, on many occasions, links to CD and DVD accelerated aging tests that indicate that the data should be readable in 100 years, or more.

Can't agree with that at all. I have had Verbatim discs which became totally unreadable after a year or so. Some can last much longer, but it's totally unpredictable. (I've also had discs with a lousy brand which were still readable last time I checked, and which therefore vastly outlasted the Verbatims.)

The data on a disk drive may also last that long, although the mechanics may bind up. I've had to open up disk drives to free the disk that had frozen. Theoretically you are supposed to do this in a clean room, but if you're desperate you can do it without the clean room, and get the disk to work again.

This could be done relatively safely with older drives, but with current extra-high-density models it has become extremely hasardous to attempt such dirty DIY tricks. I've asked at forum.hddguru.com what approximate capacity they would consider the absolute limit to attempt an open case repair of a HDD in regular dusty environment with reasonably good odds of success, the reply was 80GB.

@lordsmurf

DVDs? Sure, maybe, those can be. If the media is good. (Hint: gold is crap, MDisc is suspect.) The real problem will be readers and players.
CDs? No! No, no, no no no! The discs are really fragile. Simply rubbing it too hard with a fingernail can screw the disc. Unlike the plastic sandwich that is DVD, the CD layer is easily exposed. It only has a thin coating of goo between the data and oxygen.

Discs in general are supposed to be treated with extra care, so damage due to mechanical causes should be accidental and rare. (Back in 2012, the neighbour's cat, a huge ca. 20lbs ginger cat named Scar, jumped on my desk and made a bunch of DVDs fly onto the hard floor, several were kaputt as a result.) As for damage due to chemical causes, again, I've had DVDs becoming totally unreadable after about a year, while many CDs were still fine more than 10 years after being burned. (I haven't checked them much since I transfered their contents to HDD. Interestingly, when I did that tedious operation once and for all in 2009, the discs which happened to have severe readability issues were mostly among those stored in individual “crystal” cases, while those stored in spindles of 50 were almost all fine — another counter-intuitive observation. Among those in spindles, only some which were at the top of the pile and thus more exposed to sunlight were damaged ; it may have been avoided by storing the spindles in a closed compartment. Also, I sometimes dried clothes by hanging them in the same room where the discs were stored — don't do dat.)
And I have had CD-R with “gold” dye (Kodak brand, among the very first CDs I burned) stay perfectly readable while supposedly high-grade Verbatim AZO got f**ked-up pretty quick.

"Cloud" is next thing, but bandwidth mostly sucks, and security of data is still a concern. (Noting that "cloud" is just hard drives in a datacenter, hopefully redundant.)

Which rises the question, has there yet been a report about data integrity issues among prominent online backup services ? Like, people checking their data some years after the initial upload and noticing some level of corruption. Even few cases of minor corruption would be very problematic.

[johnmeyer]

@ abolibibelot
So what do you recommend as the best solution for long-term storage of photos and videos?

[Nilzzon]

Something like Quickpar http://www.quickpar.org.uk/index.htm

[abolibibelot]

So what do you recommend as the best solution for long-term storage of photos and videos?

I'm not sure, not much is made with “long-term” in mind these days as you might have noticed... (Even poor old Arecibo which was built at a time when people and governments still cared about long-lasting and far-reaching endeavours.) But for the most important stuff a combination of 2/3 HDDs (ideally 1 backup at home updated regularly and 1 in a remote location, updated less often) and an online backup plan seems the most weasonable. I've been willing to subscribe to Backblaze or sumptin' for years but I still have a lousy ADSL upload bandwitdth, which I constantly max out for file sharing purposes, so it would be next to useless for the time being.

[lingyi]

+1

The longest proven backup media is LTO at up to 30+ years, IF stored under proper conditions, low light, cool, dry, consistent temps. And even tape is subject to the same rules as any media that can fail at any time, for any reason, without any notice.

Long term storage as stated above is a strategy, not a medium. Multiple backups on multiple different media with at least one copy offsite in case of local loss, stored in good conditions, checked, verified and transferred to new media every few years. PAR and RAR recovery files are good as an additional layer of potential data recovery, but is just part of the strategy. Over the next 20-30 years, you may end up with10 copies of your data, but that's 10x more likely you'll have recoverable data.

It's not cheap, but still cheaper than having to pay for data recovery which isn't 100% guaranteed.