From the Editors: external peer Review at HSR

Health Services Research, Oct, 2004 by Ann Barry Flood

In 2003, we initiated a series of editorials entitled "From the Editors" that were designed to make our policies and procedures as transparent as possible. To date, we have written about our special issues and special sections, our transformation to an electronic peer review process, and the rates of and time-to-make different types of editorial decisions. (1) The perspective on our peer review process taken to date has emphasized information most important to our authors. In this editorial we examine our process from the perspective of the reviewer and address four areas: (1) the HSR process for selecting external reviewers, (2) what editors want to know from reviewers, (3) HSR's experience in using reviews in our new internet-based system, and (4) examples of how reviews can impact the content of manuscripts.

External peer review is integral to the process of evaluating the scientific design and evidence-base and of ensuring the integrity of our field. It helps ensure that published manuscripts have undergone careful and open examination of the theoretical background and models used; that the methods and data sources and findings have withstood critical scrutiny; that the conclusions and implications have been evaluated regarding their importance and validity; and that the manuscript has been judged for its overall worthiness of publication. Key to this effort is a small army of potential reviewers (HSR has about 2500 in its database) and members of the Editorial Board who agree to bear an unusually large load as well as other duties. The official payments for reviewing consist of an annual "thank you" and "thanks to anonymous reviewers" from some grateful authors. The principal reward, however, is simply knowing they have contributed to the process to ensure quality of research. Other incentives include advising some colleagues [although anonymously], the potential to see cutting-edge work before its publication, and receiving feedback from the review process itself. We are indebted to all our reviewers for their time and effort, and this editorial is dedicated to them. And, while we hope that experienced reviewers will find this editorial edifying, it is particularly designed for young investigators and scientists-in-training. We invite you all to join our army. (2)

OUR PROCESS FOR SELECTING EXTERNAL REVIEWERS

In order to maximize the contributions from external reviewers, either of the two co-Editors-in-Chief (EIC) first screens all manuscripts to determine their potential suitability for HSR, both in terms of content and quality. The EIC also judges whether any disclosed conflicts of interest or prior dissemination preclude publication in HSR. Overall, the EICs reject about 30o/0 of HSR manuscripts without external review at this step.

If manuscripts pass this initial screening, the EIC then assigns the manuscript to one of the six Senior Associate Editors (SAE), on the basis of their expertise and overall workload. At the same time, the EIC suggests several external reviewers and may comment on the types of expertise that may be appropriate to draw upon. We intend to select reviewers who provide a balance of the areas of expertise most pertinent for reviewing the manuscript, and so the EIC may create two lists to draw upon.

The SAE, after reviewing the manuscript and the EIC's suggestions about reviewers, makes the actual selection of reviewers, often using reviewers who are not on the original EIC list. The list may include up to 10 or more potential reviewers with appropriate expertise, because the SAE needs to invite reviewers until two agree to review. Occasionally, a potential reviewer will provide an alternative name if he/she cannot review.

There are several criteria and sources the editorial staff (i.e., the EICs and SAEs) draw upon in selecting potential external reviewers for a manuscript, including: (1) the names submitted by the authors as potential reviewers or reviewers to avoid [the latter is always respected; we don't necessarily use the former, but always find it helpful]; (2) authors of selected references used in the paper; (3) an examination of names we have in our reviewer database or who are otherwise known to us; and (4) a keyword search using our automated system. MEDLINE searches are used as a tool to help ensure we do not include reviewers who have co-authored recent papers with the authors or are at the same institution. MEDLINE can also identify authors of papers on similar topics and help us examine the suitability of the publications of potential reviewers.

HOW OUR ELECTRONIC PEER REVIEW SYSTEM WORKS FOR REVIEWING

We started using an internet-based peer review administration system in June of 2003. This system allows us to track key steps, including submission of manuscripts and their revisions as well as e-mail communications about the manuscript, reviews, and recommended decisions at various levels.

Focusing on the electronic process as it relates to reviewers, this system permits the editorial staff to look at the brief biographical information and key words entered by each reviewer; also recorded for each person is the number of manuscripts currently under review, the reviewer's average time to return reviews, whether they are on our Editorial Board, and whether they have recorded a "vacation message" requesting that they not be selected for a given period of time. The EICs or SAEs can add comments too; here, we typically remark when the reviews were unusually excellent or problematic.