Archival data has a new mission: Critical; it's not what it used to be

Computer Technology Review, Feb, 2003 by Fred Moore

4) Destroy/delete cycle -- historically at the end of seven or more years (P<=.001)

Note: P is the probability that the data, file or object will be accessed during the various lifecycle stages.

Though the first two categories of the data lifecycle remain similar to the past, the last two components are changing. The third component, the archive cycle, is now extending indefinitely and often well past the traditional seven-year window. Less data is being deleted and more data is being kept for longer periods of time.

What does this mean to the storage industry? Digital archives are quickly defining new requirements for storage and its management.

Key data requirements for digital archive management:

* Retention/destruction management

* Audit provisions for tracking and reporting

* Long-term data preservation

* Compliance management for legal issues

* Authentication

* High availability for data and devices

* Advanced search and access capability with unique naming conventions (taxonomy)

* An industrial class HSM (Hierarchical Storage Management) for tiered SLAs

* Renewed use of WORM (Write-Once-Read-Many) functionality as certain data must never be changed

* The large-scale storage requirements mandate low-cost storage, TCO becomes a key consideration as data lifecycles increase

The time of viewing archival storage as the final stage of existence for data is passing. In some cases the value and utility of data is actually increasing as data ages even if the accesses to that data decline. Surprisingly, archive data is possibly becoming the fastest growing segment of the storage industry in terms of storage demand. What a surprise! Many of today's storage intensive applications are instantly creating fixed content and archive data. Applications including voice, text, graphic images, audio, HDTV, 3-D graphics, and movies all create the demand for archival data preservation. New and emerging digital applications will continue to fuel many years of explosive growth for storage as terabyte-plus data-warehouses, VCR to HDTV quality movies, the possible digital cinema, electronic voice and video-mail, digital security systems, and digital photography all will drive major changes in the way we view archival storage. Approximately 10 percent of the digital data produced in the world resides on magnetic disk storage, and an estimated 90 percent of digital storage resides on removable storage media such as tape, optical (CD, DVD) or small-diameter removable disks.

Given this, the storage industry is beginning to view archival data as a much more meaningful class of storage. Though low cost is important, the new requirements for preserving data and making it accessible on a broad scale are real. Again we visit the need and value of a tiered storage hierarchy (and HSM functionality) that differentiates between performance, capacity, retrieval capability and now data protection and security. Rigid disk drives, magnetic tape drives, optical disks, flexible drives and flash memory will all play bigger roles for storing fixed content for a wide class of users. The unique combination of complex objects, along with different availability and bandwidth requirements for archival data pose several new challenges for the storage management industry. The sheer size of fixed content and archive files changes the rules for moving data from place to place as transmission times are surpassing current architectural limits quickly.