The incredible shrinking petabyte: how compression technologies are helping store more data in less space

Computer Technology Review, June, 2004 by Richard Gadomski

The Internet brought with it a round-the-clock business environment where Web traffic, e-mail and e-commerce transactions drive data creation, storage and business continuity requirements that continuously change. Traditionally, large organizations dealt with the data deluge by simply buying massive storage libraries, filling up large data warehouses with wall-to-wall archival solutions to hold and manage it all.

But just as technology advancement and dropping manufacturing costs eventually brought the power of refrigerator-sized computers of the 1950s to desktop PCs of the 1980s, storage hardware and media are rapidly shrinking and becoming ubiquitous to every business and trade. The ability to store large volumes of data in a smaller area is a growing priority for companies looking to improve their cost of doing business and take up less space on their data room floor. And the creation of smaller and smaller storage hardware is being matched by innovations in the media needed to store that data.

Today, innovations in data compression and coating technologies are enabling more information to reside on a smaller surface. As a result, it will soon be possible to fit a terabyte worth of information--an equivalent to 500 million text pages generated from 50,000 trees--onto a cartridge that fits in the paln of your hand.

Data storage has a long and wonderful history filled with tremendous innovations that have kept up with and enabled the burgeoning information age. And while today's data warehouses may be cumbersome, the quest to reduce the size of those data stores is in fact as old as the computing industry itself.

Tubes to Tape: The History of Storing Data

It took nearly 3 million years for humans to figure out that etching animals and good hunting sites onto stone for preservation and information-sharing was a valuable--and life-prolonging--tool. Thankfully, data storage technology and compression is evolving considerably faster since the birth of modern information technology just 50 years ago.

Before the 1950s, large rooms filled with vacuum tubes and wires were used to perform even the simplest mathematical calculations. Information was kept on paper, in books at libraries or--if you were really high tech--punch cards. Personal computers, hard drives and digital data storage were the stuff of science fiction novels and Saturday matinee movies.

The baby boom in the United States in the 1950s forced the government to create a better way to track the numbers and needs of a rapidly growing population. The result was a burgeoning U.S. census bureau and Social Security administration that became the catalysts for innovation to find a better way to store data. In 1951, two UNIVAC I computers were delivered to the U.S. Census Bureau that together weighed 32,000 pounds and contained over 10,000 vacuum tubes. The UNIVAC's tape storage device called UNISERVO I held 1,000 words with 11 digits. UNIVAC was state-of-the-art, and handled a whopping 1,000 calculations per second at a cost of $159,000 for the first mass-production unit. To give you some perspective, that would be nearly $1.5M in 2004 dollars. That was some investment for a machine quite different from today's toaster-sized computers capable of 2 billion calculations per second--and cost less than $1,000.

As refrigerator-sized computers began to proliferate, a series of enhancements and firsts for tape storage quickly followed, to help reduce and improve efficiency of the space that computing power took up on the data center floor.

In 1986, an IBM compression development known as Improved Data Recording Capability (IDRC) provided a significant improvement in data compression for tape of at least 2X. IBM's Enterprise Systems Connection Architecture (ESCON) channel interface, delivered in 1987, enabled tape storage to be managed across a network instead of at the central server. Tape subsystems could then be deployed for better data management and across several kilometers--a distance previously unheard of.

In 1991, the 36-track drive was introduced, which used a new extended-length chromium dioxide tape and provided 800MB of storage. With the IDRC, capacity was expanded to over 2.4GB--the highest data capacity available at that time.

As data storage has evolved, other types of storage have followed magnetic tape storage and have led to the development of everything from real-time storage servers to the floppy disk and zip drives of the average PC. CDs changed the world of audio recording while DVDs are currently doing the same thing for video data storage.

Higher Densities, Smaller Footprints

All of the great digital storage innovations over the past 50 years owe their development to new ways of compressing data onto smaller and smaller formats. Compression requires some form of mathematical formula, or algorithm, to reduce redundant strings of data and uses a code to create patterns of data that fit onto a smaller space. Not all compression technologies are equal, however, and some recent advances are allowing for unprecedented amounts of data to be compressed to smaller sizes than was ever before thought possible.