Don't let obsession with numbers take your eyes off your goals: percentile rankings can be moving target; focus on PI instead

HealthCare Benchmarks and Quality Improvement, March, 2005

Key Points

* The proliferation of comparative databases doesn't mean rankings have gained in significance.

* Good benchmarking requires achieving a balance that incorporates quality, cost, and speed.

* Measure success by the positive changes you have made and whether you're reaching your goals.

At first glance, the proliferation of comparative databases for benchmarking activities may be a good thing, and of course, it always helps to have more data, but benchmarking experts warn there can be too much of a good thing--especially if it causes you to lose perspective.

"I'm finding that there's an obsession with the desire to attain a certain percentile level of performance, without regard to what that might mean for the organization," says Robert G. Gift, vice president for strategic planning and business development at Memorial Health System in Chattanooga, TN.

"With the proliferation of all of the different comparative databases, it seems to me people are pursuing being at a certain percentile without really understanding: a) that we have to change our underlying processes that produce those results; and b) that if you push on one performance metric to the exclusion of all others, it will create some organizational dissonance as well as performance dysfunctions. You must keep things in context," he adds.

"He's absolutely right on," says Sharon Lau, a consultant with Medical Management Planning in Los Angeles. "The thing I find I need to stress when talking to people about benchmarking is that it can't be done in vacuum."

Lau describes the benchmarking process as a three-legged stool, incorporating cost/productivity/financial; quality, which includes outcomes and customer service; and speed of service--how long it takes to get something done.

To Philip A. Newbold, MBA, chief executive officer for Memorial Hospital and Health System in South Bend, IN, it's a case of "deja vu all over again," harkening back to the early days of quality improvement, when American industries first sought to duplicate the efficiency of Japanese companies and began adopting the philosophies of W. Edwards Deming, et al.

"As it was then, this is the key point: Are you really serious about change and transformation, or do you just want to play with the toys?" he challenges. "You must confront what it really means to execute and perform and to change culture, and to make sure that happens every day, all the time."

Gift concurs. "It seems that there is a disconnect--or a lack of understanding--that exists between attaining a level of performance and the changes required in the processes that produce that performance," he says.

The consequences of imbalance

Exactly what can happen to your organization if you take your eyes of the QI 'ball'? "If we push on the productivity performance without focusing on patient satisfaction, it may be that we get the productivity we want--on the two patients we have left," Gift warns.

He offers what he calls his favorite example: Your organization has an operational database that reports worked hours per departmental work load unit--i.e., per procedure, per patient day, per test. Now further suppose that the organization has decided to focus on achieving the top 25th percentile for all of the departments across the board.

"Here's what happens: There are nuances in the data that people may not fully understand," Gift explains. "If I'm in the materials management department and I am pressured to achieve at least the 75th percentile, it may be that some other facilities [in the comparative group] may in fact exclude from their departmental performance a key performance, such as sterile processing.

"The flip side also may be true: As director of the OR, I am pushed to the top 25th percentile, whereas other facilities may not do sterile processing or central supply," he notes.

"If you look at materials management and just go after that productivity, you are going to skew your balance--and benchmarking is a balance," Lau adds.

"Sure, you want to be more productive, but if it takes you 12 hours to get a g-tube up to a unit, your speed may be shot to hell. Or you can get it up in two seconds and not meet your other goals. If you tweak one piece [of the three-legged stool], it will impact the other two," she adds.

What you should be looking for, Lau continues, is to try to get the best balance for all three.

"You may not be the most productive, but as long as your service levels are in balance for your organization, that's where you want to target," she says.

"We had a [children's] hospital with the best ED productivity in our benchmarking group, but the kids were waiting eight hours. Is that good service? Of course not. They ended up adding staff so they could bring things into balance," Lau points out.

A moving target?

Another reason not to focus exclusively on numbers is that your target can change, Newbold notes.

"Take something like Press Ganey [patient satisfaction scores]; that's a moving target," he adds. "You may hit the 95th percentile one year, then you may do the same exact things the next year and be in the 85th percentile, because everyone else woke up and started paying attention to satisfaction; the bar has now been raised. You must realize that a percentile is a dynamic number, so the question you should be asking is, 'How do we build a core capacity, resilience, and strength, so we can continue to get better and better?'"