Study reveals benchmarking flaws of many report cards, quality rankings: medical centers with high transfer rates are at a disadvantage

HealthCare Benchmarks and Quality Improvement, August, 2003

"Some of the methodology is proprietary," says Rosenberg. "Take APACHE III, for example. In order to do their study, they created a company and used the company to collect the data. That became a big controversy, because then your model is in a 'black box.' They counter by saying, 'This our business.' I can understand both sides, but if you want the best quality measures, you want to know how they do it."

Lau agrees. "This is a real issue; we're dealing with it now with the Joint Commission [on Accreditation of Healthcare Organizations] on pediatrics core measures," she says. "The children's hospitals with which we work have requested a waiver because the core measures are not at all appropriate for pediatrics."

Lau says her firm is working with the Joint Commission and several major pediatric groups that lobby for child health care to develop good core measures. "But if you go to risk-adjust for conditions like asthma, all risk-adjustment methods are proprietary," she complains. "So, either everybody buys the same thing, or you can't risk-adjust."

Not all standards are in a black box, notes Newbold. "With Baldrige, you are required to share your data," he says. "Press-Ganey has an annual conference, and everyone shares what they do. There are lots of forums where the information is almost in the public domain, and that's where we want to get it, so everyone can get better at what they do."

Data-gathering presents challenges

Another challenge, says Rosenberg, is simply gathering all the data you need. "There's a huge amount of information that's not available because it's not collected, and we have six people who do nothing but collect data."

For example, he notes, there are aspects of quality that have to do with the effect of teaching programs, the number and quality of nurses, ancillary support, the census, how busy a hospital is, the volume of cases a hospital has, and so on.

"Then there are things having to do with the patient himself that we are just starting to get at," Rosenberg says. "We think of transfer patients as those who failed to respond to therapy, but what was the quality and intensity of care at the caring hospital? We're not studying psychological factors at all. Physiologic reserve is another variable. If you're young and healthy, you have a better chance of getting better. Kids, especially, have such reserves, while 90-year-olds do not, but we don't have measures for this. These are all part of risk-adjusting patients in order to benchmark."

"It would be wonderful if there were some kind of way to score patients at admission, so that if they were very sick, you'd get 'credit' for it, while the others take the cream," says Lau. "It's the same with kids; adult hospitals hold onto the tonsillectomies and send out cystic fibrosis.

"But the bottom-line message is, don't avoid benchmarking; get together with these other hospitals. Try to join collaboratives, get together with your associations, and come up with standardized methods to monitor specific issues that need to be benchmarked. Line up as close as you can, and look for improvement."