Tag: Data Storage

FreeFileSync

Update – November 11, 2014

Until recently I regularly used and wholeheartedly endorsed FreeFileSync. Recently, however, the developers have bundled the software with browser hijacking malware that installs itself when you attempt to install the software. As such, I no longer use or recommend FreeFileSync.

 

 

FreeFileSync is a free and open source tool for syncing directories and files. I use it primarily to mirror my main personal data drive — which clocks in at about 3 million files in 1.1 terabytes —  to a local backup.

In the past, I’ve actually paid for commercial sync tools and this blows them all away. It tears through the compare and sync very quickly, and is extremely configurable if you want to go beyond simple mirroring.

I rely on this daily, and the best praise I can give it is that I just hit the Synchronize button and forget about it.

 

 

What Sort of Time Frame is Realistic for Large Scale Data Storage?

Paul Querna wrote an interesting post back in June about “forever storage” — data storage that could potentially be stable for civilization-spanning eras of time,

Not everyone will believe we can keep growing technology at the pace we have, nor that we might be able to stop death and diseases in our generation, but I do believe we are in the age where information created and stored today, could survive forever.

…

There are small technical challenges, like how would you write to media intended to last thousands of years, where would you store it all, and how would you pass on access to this data to whomever you desire, but I think they are all solvable.

If you can store your body in cryogenic storage for thousands of years, why can’t you store your data; Not just for yourself, but for your descendants.

One of the interesting questions here is just how well previous civilizations have done at information preservation. On the one hand, I can still read the full text of Sophocles’ Oedipus the King even though it was written about 2,400 years ago. On the other hand, even though he was one of the most famous and successful Athenian playwrights, only 7 of the 123 plays he wrote survives in complete form.

It would be interesting to estimate what percentage of written data generated by civilizations prior to the 15th century survived to be readable today. I’d be surprised if more than 5 percent of such material survived, and suspect something like 0.5% is excessively optimistic (if anyone knows of any published estimates of long-term information survival, please send me a link or reference).

Add to that we’re relying primarily on magnetic-based form factor-based hard drives for large scale storage which is technology that has been around for just 32 years now.

One possibility is to use something like the Rosetta disc or some of the physical, non-dye based optical solutions which should last a very long time, though at exorbitant prices (there’s no way I’m putting my 25tb or so personal data archive on them).

One of the commenters to Querna’s post highlights the data storage of memristors which are capable of storing data at much higher densities than existing hard drives and are nonvolatile (sort of like flash memory today). Moreover, Stanley Williams — who first invented the memristor — has said that the lifespan of memristors could be for periods far longer than mere millenia. Now all we need are for memristors to become cheap and widely available!

And don’t forget that even with memristors or something like it, at the moment your data is stuck in this lonely gravity well in some third-rate planetary system that is vulnerable to a number of potential catastrophes. What we need are autonomous, reproducing memristor-bots that can spread that data archive throughout the universe (now that’s cloud-based computing).

The M-Arc DVD System

First heard about Millenniata’s M-Arc archival DVD system when the Long Now blog mentioned that the product was actually shipping.

Millenniata claims its M-Arc DVDs are backwards-compatible with existing DVD technologies, but rather than using a laser to heat up a photosensitive dye, the M-Arc uses a mechanical process to make scratches in a physical layer that M-Arc claims will last potentially for centuries if stored properly. According to a brief summary on the manufacturer’s site, the M-Arc:

  • Preserves data for centuries with physical changes in data layer
  • Constructed with rock-hard materials known to last for centuries
  • Backwards-compatible on all standard DVD drives
  • Functions like a standard DVD with a capacity of 4.7 GB
  • Exclusively written by the M-Writer™ Drive

    • The Millenniata site doesn’t list any prices, but Long Now reports $1,700 for the writer and $16-$25 per 4.7gb disc depending on the quantity.

    Star Wars Flash Drives

    Since there can never be enough Star Wars merchandise (apparently), Star Wars flash drives seemed the obvious way for Lucascorp to go. Along with Yoda, a Boba Fett, C-3PO, Darth Vader and others will be available in October 2009. Unfortunately there was a bug in the Han Solo version — it turned out you had to insert the Greedo flash drive first before it would work.

    Yoda Flash Drive

    My Kind of Long Term Data Storage

    Via the Long Now blog, I ran across this article describing researchers’ efforts to create a data storage system to last a thousand years or more. Rather than rely on optical or magnetic media, both of which can be corrupted fairly easily over decades — much less hundreds of years — of storage, Japanese researchers proposed a system of stacked wafers composed of mask ROM,

    Thus, the researchers proposed the idea of saving data on the mask ROM with electron-beam direct-writing technology, stocking the wafers and packaging them with SiO2 to form a “slate.” When a wafer (reader) for reading data is attached to the slate, it becomes possible to supply power and communicate signals by wireless.

    If four 15-inch wafers made by using 45nm CMOS technology are stacked, the memory capacity will be 2.5 Tbits.

    As long as humidity levels are kept low, the proposed device would have a lifespan of hundreds of years.

    The Long Now blog adds a nice twist on how to solve a problem inherent with any such device — a thousand years from now, how will our descendants know how to access the data?

    If someone finds this disk 1,000 years from now, how will they know how to access the information?   We think a microetched instruction manual might do very nicely.

    Nice. That’s how I plan to backup my World of Warcraft videos.

    How Will We Store (and Find) All That Porn?

    I really wish there was a video online somewhere of Rose White’s presentation on data storage at 25C3, The Infinite Library: Storage and Access of Pornographic Information.

    Of course, it has always been a pain to store pornography — and so we have the cultural trope of a stash of magazines “under the mattress” or in a box hidden in the closet. But as the sex industry shifts toward digital publication at every level, we might imagine that mere storage will become a problem of the past, or, at least, a problem related to legacy materials (books, magazines, videos, comic books, photographs, etc.). Cheap, massive storage media means no more problem, right?

    Well, reviewers of porn find that they quickly amass more material than they will ever have time to peruse; librarians who need to provide access to controversial and poorly cataloged material end up overwhelmed; even casual collectors of pornography still need some way to keep track of what they have.

    Toward that end, I am doing preliminary research on how people store and access their digital pornography collections. In my early interviews, I have already encountered a fascinating mix of responses; one person has said they store their porn “in the cloud,” while another explained his detailed system for hiding digital porn files from his partner.