Are you prepared for long-term data preservation? - first in/first out

Computer Technology Review, Oct, 2003 by Richard Harada

A lot has been written lately about the need to store and archive important data to meet today's mandates of various government regulations, including the Sarbanes-Oxley Act, SEC Rule 17a-4, HIPAA, and a myriad of others. With more and more data being stored electronically, how does a company keep this information safe and accessible for years to come when the technology required to store the information seems to change every few years?

The digital world allows greater accessibility both for the consumer and business alike. Wouldn't it be great if the local cable company could provide any movie or documentary ever produced, to any subscriber, at any time of the day or night? And what if the data and photos from space exploration could be used to forecast future events? Digital media is an excellent method for storing music, films, news videos and data in a very accessible form; traditional archival media (carving in stone, writing on paper, scanning to microfilm) is no longer suitable for accessing and sharing the vast amount of audio, video and textual information that is being created, nor is it easy to manipulate its content to create new content.

The works of past masters such as DaVinci, Mozart and Shakespeare have survived for more than 400 years, and likely will survive the next 400. But what are the chances that the works of composers and writers of the twenty-first century will even be remembered, much less available, 20 years after their creation? Information committed to stone or paper or microfilm is readable by the human eye, which likely will not change much in the future. But the same might not be the case of disk drives, memory and data formats used to store digital information. There isn't a technological "Rosetta Stone" to help the archeologists of future centuries decipher this information.

And even if someone could read the logs, how would they process the vast amount of information? Indeed, this isn't just a problem for our future generations. A great number of historical records from the past 20 to 40 years have been lost, resulting in the need to recreate data rather than access the data that already existed. Archivists seek to have the technology that takes better measures to store information that can be used now and for years to come.

* The State of New York and several other states lost 1960s-era inventories of land use and toxic disposal sites (i.e., the Love Canal).

* Written data from NASA's 1976 Viking mission to Mars has been lost. This data could have been used for current and future explorations of Mars.

Satellite data recorded in the 1970s, which was to be used to identify ecological trends in South America's Amazon Basin, is now lost forever.

* In 1992, in an act of hatred, the National Library of Yugoslavia in Sarajevo was burned, destroying tens of thousands of rare Islamic texts and more than 3 million other volumes, in an attempt to delete the complete history of an entire culture.

* In 1999, the U.S. National Archives reported that it didn't save more than 40,000 internal e-mail messages, prompting a review of its data storage policies.

* All corporate records of the Pennsylvania Railroad were inadvertently deleted.

Large amounts of historically significant information are now being created digitally. Vast quantities of data are being gathered on Earth's environment, measuring the effects of global warming, deforestation, ozone depletion and other potential threats to life on the planet. Scientists are in the business of charging ahead into new frontiers, and they need new technologies to get there, but they typically do not spend enough energy worrying about preserving what they've collected in the past. Much of this priceless, irreplaceable information is in jeopardy of being "lost" or has not been collected.

Fully 75% of all government transactions are now being conducted electronically, and the volume of data being generated by the government is doubling every year. For our democracy to endure, it is absolutely essential to preserve the records of these transactions: records of births and deaths, land ownership, treaties, maps, laws, patents, architectural diagrams, etc.

There hasn't been any one digital storage technology used to meet the challenge of "forever" data preservation. However, data tape has been used for the past 50 years and continues to be an important part of the preservation solution. The key to a successful data migration is to estimate the amount of data to be stored, the frequency of its access, and to use a tape technology with long-term data retention characteristics and published compatibility roadmaps that will survive the introduction of new storage technologies. Today's professional tape technologies offer the opportunity to bridge the chasm between what we want to store today and what we want to access at a later date.

Given the access criteria, capacity, cost, scalability and reliability requirements that archivists face today, tape continues to be the best choice.