Improving quality assessment through multilevel modeling: the case of nursing home compare

Health Services Research, June, 2007 by Greg Arling, Teresa Lewis, Robert L. Kane, Christine Mueller, Shannon Flood

Many health care quality assessment systems rely on data from patient-level outcomes in order to draw inference about provider performance. Providers having higher rates of adverse outcomes (e.g., infections, pressure sores, or mortality) are assumed to be delivering poorer quality care. Despite the hierarchical nature of these data (e.g., patients nested within health care facilities), there have been relatively few efforts to model health care quality explicitly from a multilevel perspective. Ignoring the multilevel nature of these data can lead to erroneous inferences about care quality. We apply multilevel modeling to a health care quality assessment system--Medicare's Nursing Home Compare--to show how it contrasts with conventional methods and leads to better estimates of care quality.

The Nursing Home Compare website is designed to provide consumers with quality-related information on the nation's approximately 17,000 nursing facilities to help them make an informed choice about nursing home care. The nursing home quality measures (QMs) are a major component of this system in addition to results from the facility's licensure and certification report. The 12 chronic care QMs reported on this website are expressed as facility rates of the incidence or prevalence of potentially problematic care processes (e.g., restraints or catheters) or untoward outcomes (e.g., mobility decline or urinary tract infections). The Nursing Home Compare reporting system compares each facility's QM rate to the state and national averages. Facilities with higher than average QM rates are presumed to offer poorer quality care.

The QMs could be of great benefit to consumers in helping to identify the best quality facilities, as well as to providers who could compare themselves against peers and identify areas for quality improvement. However, the current QM rates have fundamental problems (Mor, Angelelli et al. 2003; Mor, Berg et al. 2003; Zimmerman 2003; Arling et al. 2005). First, risk adjustment is used sparingly; by giving only minimal attention to differences in resident acuity or risk among facilities, the QMs may generate unfair or misleading facility comparisons. Facilities with residents at higher risk of poor outcomes (i.e., more medically unstable, functionally dependent, or cognitively impaired) are at a disadvantage when compared with facilities taking care of lower risk residents. Second, the QMs fail to adequately deal with estimation error. OM rates are presented in Nursing Home Compare with no information about their accuracy or precision. For example, no consideration is given to facility size even though QM estimates for small facilities are less precise and are more likely to take on extreme values than are estimates for large facilities. The General Accounting Office (2002) criticized the Nursing Home Compare QMs along these lines.

This study proposes a better method for estimating facility QM rates. We develop and test multilevel models that produce risk-adjusted empirical Bayes (EB) estimated QM rates with credibility intervals (CIs). Our objectives are to (1) describe the multilevel modeling approach and why it is particularly well-suited to this application; (2) describe a set of multilevel models and produce new fully risk-adjusted EB QM estimates and CIs; (3) assess outcomes from these models--change in facility QM rate, rank, and likelihood of being flagged as having quality problems; (4) draw conclusions about effectiveness of the current versus new QM rates in minimizing estimation error and discriminating between facilities in the quality of their care; and (5) consider the practical implications of this approach.

QUALITY MEASURES (QMs)

Twelve chronic care QMs were selected by the National Quality Forum from among a pool of nursing quality indicators (QIs) developed by researchers at the Center for Health Systems Research and Analysis (CHSRA) at the University of Wisconsin (Zimmerman et al. 1995) and by other researchers primarily at Abt Associates and Brown University (Morris et al. 2003). Table 1 lists the 12 QMs and their definitions. For the resident, the QM is a binary variable defined by the presence or absence of the care process or outcome. The facility QM rate is a proportion based on the number of residents experiencing a care process or outcome divided by the number of residents at risk. All residents in the facility may be at risk (no exclusions), or the QM may be calculated for only a subset of residents who meet certain conditions. Rates are calculated and reported each calendar quarter.

RISK ADJUSTMENT

Risk adjustment has received considerable attention in the general health services research literature (Iezzoni 2004; Stafford et al. 2004; Blumenthal et al. 2005) and more specifically with nursing home QIs (Arling et al. 1997; Mukamel 1997; Kidder et al. 2002). On the one hand, there is general agreement that differences in resident acuity or risk between providers should be taken into account. On the other hand, risk adjustment if not appropriately carried out might "let providers off the hook" by adjusting away true differences in quality or setting lower levels of acceptable performance for certain resident groups. After much debate, members of the National Quality Forum took a conservative approach by selecting QMs for Nursing Home Compare that were only minimally adjusted (National Quality Forum 2002).