Digital preservation looks forward

Information Outlook, Sept, 2002 by Amy Friedlander

A TEAM LED BY HAL R. VARIAN AND PETER LYMAN AT THE UNIVERSITY of California at Berkeley recently used data from the late 1990s to estimate that "the world's total yearly production of print, film, optical and magnetic content would require roughly 1.5 billion gigabytes of storage. This is the equivalent of 250 megabytes per person for each man, woman and child on earth" (www.sims.berkeley.edu/research/projects/how-much-info/). Although these numbers sound somewhat terrifying, 250 MB, says Victor R. McCrary of the National Institute of Standards (NIST), represents about five percent of the storage capacity of a recordable DVD-5 disc. "But for preservation, the other important aspect, besides capturing the data on the media, is the space to store the media. A three-foot shelf holds 90 CDs in jewel cases; with DVD-5, this only takes up 5.2 inches."

Saving the data does not mean that our children will be able to find what they may be looking for in that sea of bits, nor does it mean they will even be able to display it. Storage media degrades, and files can no longer be read because the hardware and software environments that supported them have vanished. In December 2000, the Library of Congress (LC) successfully requested a special appropriation of $100 million (later rescinded to $99.8 million) from the U.S. Congress, entitled the National Digital Information Infrastructure and Preservation Program (NDIIPP) (PL 106-554). This initiative sets out to evolve a national strategy for the long-term preservation of digital content in collaboration with representatives of other federal, research, library and business organizations. Not all libraries have the same priorities; indeed, weeding a collection may be more important than preserving it for some libraries. But any library or archive that maintains an online catalog probably has at least one important r esource to protect.

The Program

LC's efforts to digitize and distribute materials began with the American Memory program in 1990. Originally an educational CD-ROM project, the popular program, based on materials in the library's collections, was folded into an equally popular Web-based initiative. In 1998, senior managers, including Associate Librarian for Strategic Initiatives Laura E. Campbell, who leads NDIIPP; Associate Librarian for Library Services Winston Tabb; and Register of Copyrights Marybeth Peters, began to formulate a strategy for managing digital materials. A report by the Computer Science and Telecommunications Board of the National Research Council, commissioned by Librarian of Congress James H. Billington, supported meeting the nationwide challenges of preserving electronic resources, in particular, the so-called "born digital" materials--those that are created or distributed primarily, if not exclusively, in digital form, for which there is no analog equivalent and which are particularly vulnerable to complete or partial loss.

NDIIPP funds are to be released in stages--$5 million was immediately authorized to support planning; $20 million is to be made available after the submission of the plan, (which is scheduled for delivery to Congress later this year); and the final $75 million will be contingent upon raising as much as $75 million in matching funds. The legislation directed LC to develop the plan in consultation with concerned industries, other major federal libraries and research institutions and not-for-profit organizations, including the Council on Library and Information Resources (CLIR), OCLC and others. Since spring 2001, LC has undertaken a planning process that involves four dimensions: stakeholder meetings from which to listen and learn; collaborative research with other federal agencies; conceptual framework for organizing relationships and technologies; and scenario planning to surface tacit assumptions and to explore unusual options and alternatives. A 27-member advisory board has been organized with representativ es from a cross-section of industries, concerned federal agencies (Department of Commerce, NIST, National Science Foundation (NSF), Office of Science and Technology Policy), not-for-profit institutions and major libraries, with representatives from the National Library of Medicine (NLM), National Archives and Records Administration (NARA), National Agricultural Library (NAL) and the British Library.

Consultations with representatives from publishing, entertainment, broadcasting, libraries and non-profit organizations in a series of three invitation-only sessions in November 2001, and a second set of more focused scenario planning workshops in the winter and spring of 2002, elicited a sense of what was happening outside of the traditional library and archival communities. There was surprising (and welcome) support across industries, where, for example, the recording industry has already made progress in thinking about the technical issues.

A number of themes surfaced across the sessions. Digital assets are degrading at a rapid rate; proprietary and/or obsolete formats inhibit the ability to recover or reuse these valuable materials (known as the "playback" problem). Some sort of decentralized or distributed solution around a common core of standards or best practices is desired, but several technical experts warned planners not to underestimate the complexities of the different formats. Managing copyrighted materials is challenging but intellectual property protection is not the only or even the most difficult challenge. A balance between the economic rights of the rights holders and the legitimate interests of the public, particularly to support education and scholarship, would be required. Copyright in the digital world is a thorny topic. LC has commissioned a white paper, which discusses the key issues and will be available via the NDIIPP Web site, launched last spring (www.digitalpreservation.gov).