Digital Data's Future—You Ain't Seen Nothin' Yet! - Industry Trend or Event

Computer Technology Review, Oct, 2000 by Fred Moore

The new millennium has promised to formally usher in the Information Age. The question of how much digital data exists now comes too frequently to simply leave it as one of academic interest only. As we move step-by-step closer to delivering an information utility, the need to at least get a handle on the world's digital storage profile mounts.

Assessing the amount of digital data in the world is no easy task. Though no one actually counts each byte of digital data created or knows its life span, best estimates are available and now indicate that as much as two-thirds of the world's data is being "born digital," meaning that its original occurrence was in a digital format generated by a computer. By the year 2005, it is projected that as much as 18 percent of the known data in the world will be captured in machine-readable (computer recognizable) digital format.

By 2005, nearly 82 percent of the world's data will remain on paper, microfiche, graphs, charts, various films or other non-machine-readable formats. Much of this data has existed in its non-digital form since its creation. The degree of digitizing non-digital data is relatively insignificant in the overall digital data progression. The costs of this conversion are very large and converting from nondigital to digital often involves additional application or workflow conversions, driving costs even higher.

Growth rates for digital data vary considerably. We observe traditional applications growing at 45-60% per year with some Internet, e-commerce and B2B based applications growing in excess of 100% annually. Recent and well-publicized growth rates from excite.com, who added 45TB of disk storage in less than two years and mail.com, who add approximately 27TB of disk storage every 45 days are prime examples of the upcoming storage demand profile of our future. Do these businesses back up their data, creating even more digital storage demand? Do they send additional copies to branch or regional offices or business partners? How often do they delete or retire obsolete data? As an industry in general, less data is being deleted (look at email for example), archives last longer, more and various types of backup copies are created, and applications such as e-mail replicate the same digital date for numerous users at numerous locations.

Accumulating data indefinitely without implementing archival and retirement policies can turn storage management into waste management. In addition, we see the costs of managing storage continue to range from three to ten times the cost of storage hardware. With hardware prices falling at 35-40% annually and management costs, which include an increasingly short supply of storage trained people along with software, rising at over 15% per year, the notion of not managing storage and simply buying more hardware is sometimes being mentioned as the easiest way out.

The digital data storage technologies map into a storage technology hierarchy consisting of magnetic disk and several removable media products that are making mass storage, data archiving and electronic data vaulting affordable realities for many businesses. Approximately 10 percent of the digital data in the world resides on magnetic disk storage and an estimated 90 percent of digital storage resides on removable storage. The amount of digital data on tape now ranges from four to fifteen times the amount of data stored on disk. The percentage of data stored on a technology does not represent the size of the technology's storage market however.

The disk storage market totals approximately $34 billion annually while the tape drive, library, and magnetic tape media markets total between $6-7 billion annually. The amount of digital storage is being driven up beyond historical levels by many new storage-intensive applications such as the Internet, e-commerce, and B2B, while data is being retained for longer time periods as attempting to manage very large storage pools becomes more difficult and costly. One page of text requires 2,400 bytes of storage. An 8-in.x 10-in. color photo will use about 38MB to store the digitized image. A standard x-ray takes 12MB to be stored digitally. A typical hour of telephone recording requires 22MB of storage. Fingerprints typically require 2MB per fingerprint set with an average of 4.2 fingerprints per arrest. Digital security beyond fingerprinting using retinal scans, blood types, DNA, and other biometric measures remains in its infancy but storage demand for this application is projected to grow exponentially.

Automated libraries architectures using magnetic tape, possibly small form-factor magnetic disks, the CD-DVD and possibly other emerging storage media are becoming the storage foundation containing most of the mass-storage demand growth though they will not account for the biggest piece of the hardware revenues. Making this possible are the unprecedented tape capacities now being delivered. New and emerging digital applications will continue to fuel a period of explosive growth for storage well into the next century. Terabyte-plus databases for a variety of new applications, data warehouses, electronic voice and in particular video mail attachments will drive storage requirements for "mail" applications beyond anything we've seen before. A ten-second video attachment is one thing, a two hour video attachment will be another. We will soon describe storage in new terms such as zettabytes (1021 or 1,000 exabytes), and yottabytes (1024 or 1,000 zettabytes).