Evaluating Digital Libraries: A Longitudinal and Multifaceted View

Library Trends, Fall, 2000 by Gary Marchionini

ABSTRACT

THE PERSEUS DIGITAL LIBRARY (PDL) IS ONE OF THE primary digital resources for the humanities. Under continuous development since 1987, the project has included an ongoing evaluation component that aims to understand the effects of access to digitized source materials in the humanities. A summary of the PDL genesis and current status is given and the multifaceted and longitudinal evaluation effort is described. A brief synthesis of results is provided and reflections on the evaluation along with recommendations for DL evaluation are given.

INTRODUCTION

Digital libraries marry the missions, techniques, and cultures of physical libraries with the capabilities and cultures of computing and telecommunications. Evaluating digital libraries is a bit like judging how successful is a marriage. Much depends on how successful the partners are as individuals as well as the emergent conditions made possible by the union. All three entities--the two individuals and the gestalt union--are of course influenced by their context as well. The difficulties arise from the complexity of mutually self-adapting systems interacting in a rich environment. Metrics for success for component parts of a complex system may be distinct from the metrics for success of the marriage (e.g., success for an individual partner is typically necessary but not sufficient to ensure success for the marriage).

Digital libraries (DLs) are extensions and augmentations of physical libraries (Marchionini & Fox, 1999). As extensions, we might evaluate the individual partners using existing techniques and metrics. Assessing the impacts of libraries on the lives of patrons and the larger social milieu are the ultimate goals of evaluation, but the practical difficulties of assessing such complex and varied impacts cause us to measure the effectiveness and efficiencies of library operations and services as surrogates for these impacts. Metrics such as circulation, collection size and growth rate, patron visits, reference questions answered, patron satisfaction, and financial stability may be used to assess physical library performance in this regard. Clearly, these are metrics that may be points of departure for evaluating digital libraries, but they are not sufficient to characterize the new rapidly emerging entity. Evaluation criteria for digital technologies can also be useful. For example, metrics such as response time, storage capacity, transfer rate, user satisfaction, and cost per operation may be useful in assessing technological components but may not be sufficient to characterize DL performance, let alone impact. As extensions of physical libraries and digital technologies, these metrics are good starting points, but we must look further to consider the effects of DLs as augmentations that provide new services, products, and capabilities.

In assessing new services and products, it is difficult to distinguish novelty effects (both positive and negative) from long-term effects. More importantly, new services and products typically create new effects that cannot be predicted until an "installed base" of practice takes root. Additionally, some of these unanticipated effects are due to the new services and products, and some are due to the marriage of existing services and products to the new ones. It seems certain that assessing these effects will not happen in "Internet time." The effects of DLs will emerge over time as physical libraries, DLs, and people mutually adapt and mature; the problem of evaluation for DLs is thus one of assessing complex adaptive systems.

The goal of this discussion is to provide a view of an important DL that has been evolving for more than a decade. Over this time, both the Perseus Digital Library (PDL) (Crane, n.d.-a) and the related evaluation effort have evolved, guided by central missions to provide and understand the effects of broad access to digitized source materials in the humanities. From the beginning of the evaluation effort in 1987, the primary evaluation aim was to address the impact of this project on users and the humanities community. This article will provide a reflective summary of results for this particular DL, discuss the methodological approaches taken to understand the evolving DL, and argue for multifaceted and longitudinal assessments of DLs in general. The article first describes the genesis, evolution, and current status of the Perseus Digital Library; provides a perspective on evaluation as a research and problem solving endeavor; summarizes how this perspective was applied to the evaluation of the Perseus DL over a twelve-year-period and what outcomes have emerged; and finally provides some reflections and recommendations for DL evaluation.

THE PERSEUS DIGITAL LIBRARY: ACCESS TO PRIMARY RESOURCES

As stated on its Web page: "The Perseus Project is an evolving digital library of resources for the study of the ancient world and beyond." The mission statement reads: "Our primary goal is to bring a wide range of source materials to as large an audience as possible" (Crane, n.d.-b). These themes of evolution and wide-scale access to source materials have been constants from the earliest days of the project. A small team of classicists led by Gregory Crane began planning in 1985, and several small grants supported a number of prototypes that led to a large grant from the Annenberg/CPB Project to begin building the "hypertext" in 1989. One component of the plan was an external evaluation effort that has continued until the present. The initial plan was to digitize as many ancient Greek texts and English translations as possible; gather or create images, maps, and video objects related to locations and artifacts; and build tools for searching and manipulating these materials (see Crane, 1988, and Crane & Mylonas, 1988, for early articulations of the Perseus vision). The Apple HyperCard platform was selected since it offered the best hyperlinking and multimedia capabilities in the late 1980s. One objective was to create a CD-ROM package that contained many of the primary readings and resources that students taking classics courses would need and to make this package available for the cost of one or two textbooks. In addition to university students, the Perseus team expected that classical scholars would find the corpus and tools helpful to their research and would also contribute new translations, interpretations, and tools. Because the funding aimed to apply new technologies to improve learning and teaching, the project was characterized as an interactive curriculum.