Introduction - criteria for assessment of digital library services

Library Trends, Fall, 2000 by Thomas A. Peters

Noting that evaluation has not been a conspicuous activity in many DL projects and programs, Saracevic concentrates on clarifying the; definitions and possible taxonomies related to DL evaluation programs. Saracevic warns that evaluation at these early stages of DL development and evolution could have dangerous stifling effects. Among other things, Saracevic suggests that the conceptual state of the art of DL evaluation may be insufficiently developed at this point in time to be widely practiced. He advocates a systems approach to evaluation, defining evaluation as an appraisal of the performance or functioning of all or part of a system in relation to some articulated objectives. Clear specifications regarding construct, context, criteria, measures, and methodology are required for any evaluation of digital libraries. Any DL evaluation project or program must specify clearly what elements (e.g., collections, access, services, costs, etc.) are being evaluated. The levels of a DL evaluation can range from a broad social level through the interface to the engineering, processing, and content supporting the system. Saracevic notes that, to date, evaluations of digital libraries have not been conducted on more than one level. Criteria from the evaluation of traditional libraries, traditional information retrieval systems, and traditional HCI studies can and must be adapted for DL evaluation projects. Uniformity across DL systems and persistence over time may serve as additional evaluation criteria. Saracevic offers multiple perspectives for conceptualizing the structure and purpose of DL service assessment projects and programs.

Seadle provides an anthropological perspective on the National Gallery of the SpokenWord, an NSF-funded DLI-2 project centered at Michigan State University. He argues that any attempt to assess the worth and outcomes of a DL program must be preceded by an attempt to understand the people (including their "micro-cultures") involved, ranging from the co-principal investigators of the grant-funded project to the real and anticipated end-users. A micro-culture denotes units of shared meaning as small as academic professions, university departments, and interest groups. Most DL users have links to, and are molded and influenced by, various micro-cultures. Seadle notes that the meaning and goals of a DL often are elusive. The texts emanating from a DL project, such as the original grant proposal, often do not fully reveal the goals and meaning. Seadle suggests that the methods and mode of cultural anthropology can provide the intellectual foundation upon which informed choices can be made concerning the sample population, survey design, and focus group selection for the assessment component of the project. Seadle concludes, among other things, that the imprecisions of meaning lie at the core of the evaluation issue regarding DL services. The different micro-cultures involved in this DL initiative make different assumptions about the composition, needs, and desires of the end-user population. The introduction of cultural anthropology to the DL assessment process is a beneficial message for librarians.