What Sort of Time Frame is Realistic for Large Scale Data Storage?

Paul Querna wrote an interesting post back in June about “forever storage” — data storage that could potentially be stable for civilization-spanning eras of time,

Not everyone will believe we can keep growing technology at the pace we have, nor that we might be able to stop death and diseases in our generation, but I do believe we are in the age where information created and stored today, could survive forever.

There are small technical challenges, like how would you write to media intended to last thousands of years, where would you store it all, and how would you pass on access to this data to whomever you desire, but I think they are all solvable.

If you can store your body in cryogenic storage for thousands of years, why can’t you store your data; Not just for yourself, but for your descendants.

One of the interesting questions here is just how well previous civilizations have done at information preservation. On the one hand, I can still read the full text of Sophocles’ Oedipus the King even though it was written about 2,400 years ago. On the other hand, even though he was one of the most famous and successful Athenian playwrights, only 7 of the 123 plays he wrote survives in complete form.

It would be interesting to estimate what percentage of written data generated by civilizations prior to the 15th century survived to be readable today. I’d be surprised if more than 5 percent of such material survived, and suspect something like 0.5% is excessively optimistic (if anyone knows of any published estimates of long-term information survival, please send me a link or reference).

Add to that we’re relying primarily on magnetic-based form factor-based hard drives for large scale storage which is technology that has been around for just 32 years now.

One possibility is to use something like the Rosetta disc or some of the physical, non-dye based optical solutions which should last a very long time, though at exorbitant prices (there’s no way I’m putting my 25tb or so personal data archive on them).

One of the commenters to Querna’s post highlights the data storage of memristors which are capable of storing data at much higher densities than existing hard drives and are nonvolatile (sort of like flash memory today). Moreover, Stanley Williams — who first invented the memristor — has said that the lifespan of memristors could be for periods far longer than mere millenia. Now all we need are for memristors to become cheap and widely available!

And don’t forget that even with memristors or something like it, at the moment your data is stuck in this lonely gravity well in some third-rate planetary system that is vulnerable to a number of potential catastrophes. What we need are autonomous, reproducing memristor-bots that can spread that data archive throughout the universe (now that’s cloud-based computing).

Synology 1511+

Engadget notes the Synology 1511+ has been upgraded so it will accommodate the 3tb drives that are starting to show up. This is a 5-bay NAS that goes for about $900-$950 without disks. The nice thing about the Synology is that you can buy a 5-bay storage expander for about $500, and the 1511+ can support up to two of those. So, potentially, this thing can store 45tb for about $4,800.

Oh yeah, I’m getting one of these.

Hard Drive Transporter

So, I have a lot of hard drives lying around that I occasionally move from office to office and need effective way to store them in the meantime. I’ve tried a couple of different solutions, none of which worked very well, until running across Silicone Forensics’ Hard Drive Transporter. As you can see, on the outside this is a softshell case that has a window on the back for inserting a label:


On the inside, there’s a foam insert that snugly fits and protects a 3.5″ hard drive. Alternative inserts are available for 2.5″ and 1.8″ drives as well. These things can be had for $7.95/each, so they’re also a relatively cheap solution for moving or storing hard drives so that they have some protection.

SSDs – Write Endurance Myths

I happened to be reading a comment on another website the other day where the commenter claimed SSDs would never be viable alternatives to traditional magnetic hard drives until manufacturers solved the “100,000 rewrite cycle issue.” StorageSearch.com’s SSD Myths and Legends – “write endurance” attempts to put this claim to bed once and for all.

The main point is that most of the SSD drives being sold today have write endurances ratings in the 1 to 5 million rewrite cycle range. The 100,000 limit was the maximum available in the late 1990s, but has long since been superseded.

The upshot of this is that even using extremely data intensive applications that write and rewrite large amounts of data over and over again to and SSD, the user is looking at decades of such intensive use of an SSD before it fails.

This is why typically, SSDs released today have MTBF ratings equivalent to those of magnetic hard drives (not that there aren’t issue with MTBF as a guide to hard drive reliability, but rather that there is nothing about recent SSDs that will make them fail more frequently than their magnetic bretheren).