Methods used to streamline the CAHPS® Hospital Survey

Health Services Research, Dec, 2005 by San Keller, A. James O'Malley, Ron D. Hays, Rebecca A. Matthew, Alan M. Zaslavsky, Kimberly A. Hepner, Paul D. Cleary

There currently exists no universally accepted method of determining and reporting patient assessments of hospital care (Caste et al. 2005). The CAHPS[R] hospital survey was designed to provide consumers with comparative information about hospital performance regionally and nationally, as well as provide hospitals with a national benchmarking database that could be used to set performance goals and evaluate progress toward those goals (Goldstein et al. 2005). The conceptual framework of the survey drew from the domains of quality health care proposed in the Institute of Medicine's (2001) (IOM) report Crossing the Quality Chasm: A New Health System for the 21st Century: (1) respect for patients' values; (2) attention to patients' preferences and expressed needs; (3) coordination and integration of care; (4) patient information, communication and education; (5) physical comfort; (6) emotional support; (7) involvement of family and friends; (8) transition and continuity of care; and (9) access to care.

The development of items for these nine dimensions is detailed in Levine, Fowler, and Brown (2005), but will be briefly summarized here. A large pool of candidate item concepts relevant to the nine IOM quality domains was identified based on content included in the seven hospital surveys submitted for consideration in response to a Federal Register call for contributions (Goldstein et al. 2005). Questions were drafted to address the candidate items by following CAHPS survey design principles (Goldstein et al. 2005), including the requirement that items refer to observable behaviors or features of the environment (i.e. how often something is done or whether it is present) and do not refer to events for which the patient is not a knowledgeable informant (e.g. appropriate use of diagnostic procedures). The pool of drafted questions was tested for comprehensibility and content validity by following cognitive testing methodologies (systematic, in-person interviews) with former hospital patients as detailed by Levine, Fowler, and Brown (2005). Items that were ambiguous or confusing to interviewees, were not interpreted as intended, or did not refer to interviewees' direct experiences were modified or deleted. This process identified serious problems with 70 percent of the candidate items and eliminated all of the items from two of the IOM domains: those dealing with coordination of care and the involvement of family and friends. The final field test survey contained 33 items that referred to seven of the IOM dimensions of quality: respect for patients' values; attention to patients' preferences, and expressed needs; patient information, communication, and education; physical comfort; emotional support; transition and continuity of care; and access to care.

The motivation to shorten the pilot test questionnaire came from the CAHPS design principal to incorporate stakeholder input throughout the survey development process. During electronic and in-person meetings and in response to a Federal Register call for comments on the pilot test instrument, stakeholders emphasized the need for brevity. They required that the survey be as short as possible in order to reduce administration costs and to allow room for users to add customized content (e.g. additional questions specific to their particular hospital system). In response, we sought to reduce the length of the survey by half. In this article, we present the analytic process by which we determined how to shorten the pilot-test version of the CAHPS Hospital Survey. This process required a careful balancing of three considerations: (1) the statistical properties of the item and composite scores; (2) the importance of item and composite content to patients; and (3) representation of IOM domains.

CAHPS survey design principles require an integration of quantitative and qualitative data in order to avoid problems associated with relying on one source of information to the exclusion of the other. For example, it is not unusual to find questionnaires published in the peer-reviewed literature that were developed according to qualitative methods but not evaluated statistically for the reliability or validity of their item or composite scores. This is a risky method because regardless of how appropriate the question content appears, the data provided by the responses to the questions will have limited utility if the variance in responses is severely restricted or if the data do not indicate differences in health care quality. On the other hand, if one were to choose items for a questionnaire based solely on the properties of the data they provide (e.g. whether the responses discriminate among units of interest) with no regard for content, the resulting tool could include a small number of conceptually unrelated questions and the data could lack validity for stakeholders. It is unlikely that such a questionnaire would enjoy widespread use regardless of how precisely the data described differences in quality of care. Fortunately, the statistical properties of the questionnaire item responses and the importance of the item content, theoretically and to stakeholders, often provide the same guidance with regard to which subset of items are the best to select. In this article, we describe how standard psychometric methods and focus group methodology were used to identify the best subset of the 33 report items fielded in the version of the CAHPS Hospital Survey fielded in a three-state pilot test (described below).