Benchmarking and safety: natural fit if you know what to do with data: misreading results, using wrong benchmarks is formula for failure

HealthCare Benchmarks and Quality Improvement, May, 2004

Given the steady drumbeat for improving patient safety from diverse corners of the QI world, it's only logical for quality professionals to use all the tools at their disposal--and that includes benchmarking.

However, experts warn, while benchmarking can prove extremely valuable in your efforts to boost patient safety, those efforts can be for naught if you aren't careful about your decisions concerning what to benchmark, what your goals are, and how you interpret your data.

"People respond better when they have a goal, and physicians are notoriously good at goal seeking," notes Stephen Lawless, MD, MBA, chief knowledge officer for Wilmington, DE-based Nemours. "If you do not give them something to go for, what's the impetus to change? The real question is: What do you benchmark against--the overall average or an idealized goal--and what should that goal be?"

"Are safety and benchmarking a fit? Yes, absolutely," says Ann Nakamoto, JD, MSN, a quality improvement manager with Children's Regional Medical Center in Seattle. "More so because we now look at health care on a national basis; and given that we're trying to learn from each other and share learning, I think it's critical to lift the level of patient safety."

"From a consultant's standpoint, absolutely," says Sharon Lau, a consultant with Medical Management Planning (MMP) in Los Angeles. "If you don't have a comparison group, how do you know you're doing it right? You can know your own internal trends and if you're getting better, but you've still got to have some kind of mark out there in the world to know if you're in the ballpark."

But not everyone is sure. "I'm strongly in the maybe camp. I think there's potential value, but I have real reservations based on what's currently available," notes Matthew Scanlon, MD, assistant professor of pediatric critical care at the Medical College of Wisconsin and patient safety officer at Children's Hospital of Wisconsin, Milwaukee.

The challenge in benchmarking for safety is not so much the benchmarking process itself as it is the comparative tools available, observers say. "You can benchmark anything in patient safety as long as you can measure it," Lau says. "The difficulty comes in finding an appropriate measuring scale. How you classify some patient safety issues can be challenging."

In the case of errors, for example, "we are very lucky the Institute for Safe Medical Practices has a national rating scale we have been using for years," she notes.

"I know that some areas have benchmarks in place, like in infection control, the NGCPR [National Group on Cardiopulmonary Resuscitation], and several others, including MMP," adds Nakamoto.

"All these groups move toward developing and further enhancing databases working in that direction. The Joint Commission [on Accreditation of Healthcare Organizations], ORYX, and CMS' [the Centers for Medicare & Medicaid] core measures are moving on a national basis to identify benchmarks and to establish a common language on how to boost patient safety," she says.

Nakamoto adds a word of caution, however. "I believe that as we all move our efforts toward achieving this goal, we need to find our common definitions. NGCPR, for the medication groups nationally, for example, has classifications of injuries including close calls. I don't see the other medical events having something similar to that, so we haven't yet quantified these things on an agreed-upon basis."

Scanlon also presents a mixed picture. "When you look at benchmarking, the first question you ask is why are you doing this--for improvement or accountability--and, of course, how will those data be used? I think there's a lot of value for improvement of patient safety, and those of us who are seriously interested in improving our organization would have value from a peer group to compare ourselves to--but right now that's not possible."

Why is that? "Because of legal ramifications, discovery issues," he adds. "Are you opening yourself up to legal issues if you show a certain error rate?"

Even good benchmarks can present problems, Scanlon continues. "AHRQ's [the Agency for Healthcare Research and Quality] quality indicators theoretically could be benchmarked against, but most people don't have the sophistication to be able to compare apples to apples. Also, there are various versions of software available, and some people have been publishing papers using data that are outdated. If those data points are used to benchmark around, it could be problematic."

In addition, a number of databases do not adjust for severity of illness, he explains. "A lot of administrative screening databases use ICD-9 codes," he observes. "The problem with attribution of those is this: If you are a center that gets a lot of referrals, and the center that sends a patient to you contributed to the error but didn't document it, you get credit for it even though you inherited it."

Data: The devil's in the details

Even if you have decent benchmarks available, the way you approach the task and interpret the data can have a significant impact on your end results, experts agree.