Excellence in evaluation: early landmarks at the National Library of Medicine

Library Trends, Spring, 2008 by Barbara A. Rapp

ABSTRACT

F. Wilfrid Lancaster has earned a reputation for greatness in the evaluation of information storage and retrieval systems. Many of his extensive contributions stem from his early experience with the National Library of Medicine (NLM) MEDLARS system. His evaluation of the MEDLARS Demand Search Service in 1966 and 1967 was an important landmark as one of the earliest evaluations of a computer-based retrieval system and as the first application of recall and precision measures in a large, operational database setting. In 1971, his evaluation of the MEDLARS AIM-TWX system was an important study of early online systems and their direct use by end users. This paper summarizes Lancaster's two major evaluations of the MEDLARS system, including the information environment at the time and their impact in the field of information science. Examples of Lancaster's other evaluation work with information retrieval systems are provided, followed by discussion of the textbooks that grew out of his evaluation experience and expertise. The article closes with comments from current and former NLM staff regarding Lancaster's time at NLM or his influence on their own career.

INTRODUCTION

F. Wilfrid Lancaster established himself as a giant in the evaluation of information storage and retrieval systems early in his career, and his reputation for greatness in this arena stands today.

Many of Lancaster's extensive contributions stem from his experience with the National Library of Medicine (NLM) MEDLARS system. As one of the earliest evaluations of a computer-based retrieval system, his evaluation of the MEDLARS Demand Search Service in 1966 and 1967 was widely regarded as an important landmark, earning praise as the "beau ideal" in the Annual Review of Information Science and Technology (ARIST) (Brandhorst & Eckert, 1972). A few years later, his evaluation of the MEDLARS AIM-TWX system in 1971 was an important study of early online systems and their direct use by end users.

Lancaster undertook these evaluations in an environment of innovation and rapid change--in computing, in information retrieval applications, in information science research, and in information system evaluation.

In this paper, I first summarize Lancaster's two major evaluations of the MEDLARS system and discuss their impact at NLM and more generally in the field of information science. Next, I provide examples of his other evaluation work with information retrieval systems and discuss the books that grew out of his evaluation experience and expertise, the books that instructed so many of us about information systems--their design, analysis, and evaluation. The article closes with comments from current and former NLM staff regarding Lancaster's time at NLM or his influence on their own career.

EVALUATION OF THE MEDLARS DEMAND SEARCH SERVICE

Lancaster is widely associated with the MEDLARS evaluation--but what was it and why was it so important?

MEDLARS stands for MEDical Literature Analysis and Retrieval System and was developed to computerize the production of Index Medicus, a major printed index to the biomedical literature produced by NLM. The computer-based searching component was called the Demand Search Service. When launched in March 1964, there was no other publicly available, fully operational electronic storage and retrieval system of its magnitude in existence (Miles, 1982).

At the time of Lancaster's evaluation, the MEDLARS database contained about 800,000 bibliographic records from January 1964 forward, growing at the rate of about 200,000 records annually. Articles were indexed from a set of 2,400 journals, using the hierarchically organized MeSH controlled vocabulary that consisted of about 7,000 "fairly conventional pre-coordinate type subject headings" (Lancaster, 1968a). This was an offline, batch search system. Search requests were submitted in writing to NLM staff, who created and entered the search strategies. The searches were then run sequentially against the database tapes.

The Information and Evaluation Environment at the Time

Appreciation of the importance of the evaluation, and why it was influential for NLM and the information retrieval field, may be helped by providing a sense of the information environment at the time and the visibility of NLM's initiative to provide computer access to bibliographic data.

The use of computers for bibliographic retrieval systems was in its infancy, and many of the extant systems were experimental or small in nature. In their comprehensive history of online information services, Bourne and Hahn (2003) credit MEDLARS as "one of the earliest large-scale online retrieval operations," and describe an environment of tremendous increase in medical research publications and need for more efficient methods of information retrieval. In a recent historical paper on the development of the MEDLARS system, Dee (2007) characterized the environment by saying, "NLM's accomplishments regarding MEDLARS were cutting edge, placing the library at the forefront of incorporating mechanization and technologies into medical information systems" (p. 416). Dee also noted "enthusiastic public interest" in MEDLARS, citing coverage in the Wall Street Journal and other newspapers. In his comprehensive history of the NLM, Miles (1982) summed it up by saying "On the whole the system was one of the largest and most successful library automation projects. Its success marked a milestone in the evolution of modern libraries."