Study reveals benchmarking flaws of many report cards, quality rankings: medical centers with high transfer rates are at a disadvantage

HealthCare Benchmarks and Quality Improvement, August, 2003

Hospital rankings and report cards are growing in number and importance, but a new University of Michigan study suggests these measures may be inaccurate if they don't take into account the high number of very sick patients that large hospitals receive as transfers from other hospitals.

This study, which focused on medical intensive care unit (MICU) patients, was as much about benchmarking as it was about the MICU, says Andrew L. Rosenberg, MD, assistant professor of anesthesiology and internal medicine at the University of Michigan Health System (UMHS) in Ann Arbor, and lead author of the study.

"The idea of this study was to try to quantify something that most physicians intuitively know: Transfer patients are sicker," says Rosenberg. "However, this is difficult to quantify because the type of precise data needed are often lacking; they are expensive and hard to get at. In fact, much of [the quality rating] benchmarking deals with administrative databases, not clinical databases."

The UMHS study results were published in the June 3, 2003, issue of the Annals of Internal Medicine, in an article titled, "Accepting critically ill transfer patients: Adverse effect on a referral center's outcome and benchmark measures."

"We used a very detailed clinical database [APACHE III for Acute Physiology and Chronic Health Evaluation]," Rosenberg notes.

The study examined 4,579 consecutive admissions for 4,208 patients from Jan. 1, 1994, to April 1, 1998. A full 25% were transfer patients. Its measurements were MICU length of stay, hospital length of stay, MICU readmission, and hospital mortality rates. "We reasoned, why not study the place [MICU] where the most valid benchmarking tools are used?" says Rosenberg. "If we still can't adjust for the ICU, how can we possibly do it at another level?"

Even using tools to account for differences in diagnoses, severity of illness, and other predictors of outcome, the transferred patients had 38% longer ICU stays and 41% longer hospital stays compared with patients who were admitted to the ICU directly, and were twice as likely to die in the hospital. As a result, the authors report, a hospital that gets 25% of its ICU patients as transfers from other hospitals would show an extra 14 deaths for every 1,000 admissions, as compared with a hospital that accepts no ICU transfer patients and provides exactly the same quality of care. This seemingly small 1.4% difference would be enough to drive down the hospital's score. The transferred patients also had higher Acute Physiology Scores at admission and discharge than did directly admitted patients, and they were more likely to have complex problems, such as severe infections and upper gastrointestinal bleeding.

Study makes valid point, experts say

Benchmarking and health care experts contacted by HBQI generally agree that Rosenberg and his research team make a valid point.

"If someone is not adjusting for severity or acuity, it's back to the proverbial 'apples and oranges,'" says Robert G. Gift, MS, president of Systems Management Associates in Omaha. "You are not looking at data that are truly comparable."

"The distinction between administrative data and clinical data is a good point," adds Sharon Lau, a consultant with Medical Management Planning in Los Angeles. "Clinical data are very difficult to get."

Lau finds the whole issue of transfer patients interesting. Early in her career, she was the administrative staffer for a neonatal intensive care unit. "This was an issue every day," she recalls. "Because we did not have a maternity unit, every patient we had was transferred in. Of course, we'd get the sickest of the sickest, and no inborns to balance that out. If our morbidity and mortality had been measured [against other hospitals], it would have been horrible, but if you had a sick newborn, ours was the place it would have been sent."

"I think what he [Rosenberg] says is technically and scientifically valid," says Philip A. Newbold, MBA, chief executive officer of Memorial Hospital and Health System in South Bend, IN.

What this means, says Rosenberg, is that when "Top Hospital" rankings are issued, some of the best hospitals may not be included. Yet many consumers use these rankings to determine which hospitals deliver the best care.

"It's the referral centers, the big urban systems, that are often the centers of last resort and are mandated and built to take care of highly complex patient cases," he notes. "But if the benchmarks don't account for that, these centers are at risk for being compared negatively to other hospitals."

This is not a case of sour grapes on the part of UMHS, Rosenberg says, because UMHS often does well in such rankings. "But do you really think those 100 hospitals are the only 'best' hospitals in the country? A lot of them are relatively small community hospitals. If you are really ill in Atlanta, you'll go to Emory. In Cleveland, it's the Cleveland Clinic. But generally, those are not the hospitals that show up on those lists."