Archiving files

[TheZEDMan]

Sorry about reactivating an old post, but I think I have some good ideas to contribute (from a Computer Science background), since we are also talking about archiving files on a computer.

One other tips to consider, even more than software obsolescence, is ensuring that you have more than one copy and some extra redundancy.

SOFTWARE OBSOLESCENCE
-------------------------------
Software obsolescence is not too much of a problem, in general, because of emulation. Through emulation, you can run almost any program from any machine -- even programs and software that are older than me (and I'm almost 30 at the time of this writing). This does not mean that software for decoding your programs can become obsolete, but it is very unlikely that it should become completely obsolete anytime soon (if you use emulation). Be sure to keep copies of the software for decoding and playing your video (or at least for converting it to uncompressed images/video).


BIT ROT
---------
One problem, that is rarely discussed is "bit rot". Bit rot is the gradual decay of digital information -- over time ones and zeroes can change due to random events, such as electro-magnetic interference from the external environment, or slow radioactive decay (NOTE: for those who haven't taken physics, this is nothing to be alarmed about -- almost EVERYTHING in the environment is under constant radioactive decay, just the process is really REALLY slow and low-scale -- in-fact we can use radioactive decay information to estimate the life of trees and other objects in archeology; however, I digress).

Unlike analogue data, digital data often has checksums and error correction to help mitigate bitrot; however, over a long period of time, even the checksums and error correction data may become corrupt. Therefore, if you are storing information for a long time, be sure to use programs to generate MD5 and SHA sums of all of your files. These can be useful to detect whether a file has changed since you made the sum, as the MD5 and SHA sums will not match when you check your files in the future.

Be sure to also use redundancy programs like QuickPAR or MultiPAR, and Dvdisaster. These programs can generate error correction files which may be used to fix problems which arise from bitrot.

Finally, be sure to duplicate your backups (at least 3+ copies of everything) and have more than one drive for playing back your data. This will help safeguard your data against both hardware and software failure. Also, having more than one playback drive will protect you somewhat from obsolescence (however, remember that over time hardware may fail).


MASTER COPIES vs. WORKING COPIES
--------------------------------------------
Always be sure to make a working copy of any media you wish to access frequently. This will allow you to safely watch your video on the working copy without affecting the longevity of the master copy.


STORAGE MEDIA
-------------------
For media, I would recommend tapes and CDs/DVDs for storage (best to use BOTH!)

TAPES: I am in the process of archiving my old Betamax tapes to DVDs (@2 hour SP). The oldest tape, from 1985, recorded on a high quality TDK tape at BetaII, plays beautifully (except for some darker colouring which may have been due to the camcorder, it looks like it was shot very recently). Another tape from 1987, recorded on a Polaroid supercolor tape at BetaIII, has some slight tracking issues near the end (which may be the VCR), and looks to be lesser quality; however, it is still quite watchable.

CDs: Some of my oldest CDs from 1999/2000, recorded on regular Verbatim discs (which are a medium quality disc) at 2x on a generic 4x CD burner, with sharpie marker all over, are still playable. This is not the ideal situation for archiving discs, but they have still stood up quite well to the test of time. Also note that CDs have a better chance at holding up to the test of time because their pits/lands are larger (less dense) than DVDs; thus, the same area defect on a CD (whether due to damage or time) will affect less of the CD than it would the DVD.

In contrast, for long-term storage I would NOT recommend FLASHDRIVES, as I have seen unpredictable results. The oldest flashdrive I had was from 2000. It failed horribly with only medium use after 5 years. I have also had many flashdrives work on some computers but not others. In addition, they are very easy to loose. The only advantage of flashdrives is no moving parts, so they are a very good solution for transporting your data from point A to point B. If you plan on transporting data frequently, then consider using them, but be sure to keep your data backed up on another medium.

For HARD DRIVES, I would say MAYBE. Hard drives can be useful for storage, but they have a good risk of failing after about 6-7 years. Sometimes you can get really lucky, and they will last longer (I do have a 40MB hard drive from 1986/1988 which still worked in 2001 and may still work); but, conversely, you may also be very unlucky and have them fail within months of purchase (one drive I bought in 2002 died on me within 2 months). If you use internal hard drives, all of these problems can be mitigated to a large degree with RAID6 (or even RAID5) and ZFS. In contrast, for the external hard drive, you may choose not to use these, because you will likely want your drive to work on more than just your computer. So, for external hard drives, be sure to migrate your data to a new drive every 5 years or so. Also, never have only one copy on one hard drive.


STORAGE
-----------
Always follow the manufacturers recommendations for storage, but the most common recommendation is low humidity and low temperature. Probably storing them in a cool space in your house with low humidity should do you well enough for most cases (unless you are industry -- in which case you can afford temperature/humidity controlled rooms). You can also use some silica desiccant packets if you have them, BUT BEWARE! Always make sure there is good separation from the media, as silica desiccant can gel if it absorbs too much water. Thus, you may put the desiccant packets in the same large box as the tapes or CDs, but NEVER put the packets in the individual CD cases or tape cases!

Also, if you are using magnetic media, you may consider electromagnetic shield storage boxes. I have not used these, and do not know their true time-tested efficacy, but it is a worthwhile mention.

To put things into perspective, however, I have successfully stored many 25-30 year old Betamax tapes in a cardboard box in my house. To this day (in 2014), the tapes still playback quite well. Thus, even average good storage may be adequate.


FORMAT CONSIDERATIONS
-------------------------------
Perhaps the most important factor to consider when choosing a format for storing your data is: will the format playback well if damaged? While we can do everything to slow down and correct most of the damage and degradation of media over time, we cannot ensure that the media will never degrade. Therefore, choosing a format which is very resilient to damage may be important. For instance, no matter how damaged a vinyl record becomes you can always play it back (and even if record players become obsolete, you can still make your own cheap DIY record player with a needle, some tape and a paper cone).

Even if it is a highly compressed low-quality duplication which survives (or even the original VHS tape), and even if it survives in a very damaged state, it is still better than having nothing survive the test of time.


CONCLUSION
---------------
Always be redundant and keep multiple copies of anything you want to save!

Although my post may sound excessive (because I am on the 3rd page, which eliminates most of the context), my purpose is to discuss as many thoughts as possible, so you can consider all of the possibilities and come up with your own storage plan (applicable to your needs -- consumer, "prosumer", or industry).

Remember, the cost of an extra VHS player, a higher quality DVD, a better quality capture card, or extra hard drives may seem like a lot when you buy them at the time, but they will all seem almost insignificant in hindsight. In contrast, if you loose your data, you will always regret not spending that tiny amount of extra money (even if it wasn't so tiny), especially since re-shooting your video or regenerating your data can be anywhere from expensive to impossible!

Best luck with archiving, and let us know how things go in 7+ years!

[redwudz]

TheZEDMan, In the future, please start a new thread instead of adding on to a extremely old thread. I've moved your post to a new thread.

Moderator redwudz

[Baldrick]

Spammer?

[TheZEDMan]

TheZEDMan, In the future, please start a new thread instead of adding on to a extremely old thread. I've moved your post to a new thread.

Moderator redwudz

Thank you. I will do so in the future.

Spammer?

Nope, I'm new to here, but I'm not a spammer.

[Cornucopia]

Most of what you are saying is good stuff (which has already been said here, just not combined into one single post). However, I would amend your working HDD RAID suggestions...

For true end-to-end media security, the RAID system must be capable of successfully rebuilding itself back to a fully restored state. Unfortunately, it has been shown that as harddrives have gotten much larger, the possibility of additional drives getting corrupted WHILE REBUILDING IN THEIR VULNERABLE/REDUCED STATE (and thereby ruining the restoration) has increased exponentially on RAID 5 and even on RAID 6 setups. Even though there is less overall available space on a RAID 10, aka 1+0 (or 0+1) system, it has proven to be less vulnerable to this kind of double-whammy (I can attest to this firsthand, both the bad Raid5/6 and the good Raid10 occurrences).

Scott