Is Data Generation Outstripping Our Ability to Store It?

The Associated Press reports on a new estimate of how much data is produced worldwide every year that is significantly higher than previous estimates. It is so high, in fact, that the author of the study speculates whether or not there is actually enough storage space deployed worldwide to capture all of that data.

According to the AP, an IDC report estimates that last year about 161 billion gigabytes worth of data were generated worldwide. This is a major increase over the past best estimate by researchers at the University of California-Berkeley which estimated a global worldwide production of 5 billion gigabytes worth of data every year.

Is there enough room to store all this? Not according to the IDC,

Perhaps most noteworthy is that the supply of data technically outstrips the supply of places to put it.

IDC estimates that the world had 185 exabytes of storage available last year and will have 601 exabytes in 2010. But the amount of stuff generated is expected to jump from 161 exabytes last year to 988 exabytes (closing in on 1 zettabyte) in 2010.

“If you had a run on the bank, you’d be in trouble,” Gantz said. “If everybody stored every digital bit, there wouldn’t be enough room.”

Storing 161 exabytes should be easy. Imagine if we just want to store a copy of all 161 exabytes within the United States. That would require just a single 500gb hard drive per man, woman and child alive today. At a consumer-level cost of 40 cents per gigabyte of storage, that would only be about $65 billion in a country with a GDP of $12.5 trillion.

Is there currently 161 exabytes of storage out there? Maybe. Gartner reports that in 2005, there were 380 million hard drives shipped worldwide. If you add up all the hard drives currently out there in some machine or other around the world it’s possible the total might come closer to 161 exabytes than IDC thinks. Of course, that’s just for one year’s worth of data, and considering most of those hard drives ships were significantly smaller than 500gb … it’s unlikely there’s enough hard drive storage for that amount of data.

But, of course, hard drives are not our only method of storing data. Much of the data that IDC measures is stored on digital tape, and other media that mostly stay in the original format or is only temporarily on a hard drive.

For my part, I’m adding about 1TB a month to my personal data stash, mostly in the form of videos and pictures burned to DVD and stored in my basement. Currently it costs me about $100 to store 1 TB of data on DVD, so once hard drive space crosses the 1TB/$100 threshold (1TB/$50 if I want to be have RAID-level data security), it’ll be cheaper for me to simply build large disk arrays and store the data live rather than offline.

And that, my friends, will be a beautiful day that will occur almost certainly before the end of the decade at the current rate that hard drive costs continues to fall.

Source:

Tech Researchers Calculate Digital Info. Brian Bergstein, Associated Press, March 6, 2007.

One thought on “Is Data Generation Outstripping Our Ability to Store It?”

Leave a Reply to Hayden Dunavant  Cancel reply