Is your measurement system well balanced?

Journal for Quality and Participation, The, Oct/Nov 1994 by Brown, Mark Graham

As quality management advocates we tend to think measurement is the foundation of an effective management strategy. We like to spout off statements such as: "You get what you measure" or "Improvement is not possible without measurement." That's pretty much old news for anyone who's read anything about quality. Here's the big news for those of us who happened to miss it: measuring more things doesn't get you more quality or guarantee quality. And even better news: implementing a change effort to make your organization more customer focused, does not necessarily entail more measurement of more things.

Although measurement is critical to improved performance, organizations don't always get what they measure. If measurement by itself really had that much impact on behavior, anyone who had a scale would never be fat. Measurement only provides you with data. If the data are not used to make good business decisions and to drive improvement efforts, a good measurement system is of little value.

Selecting the vital few applies to metrics as well as to problems or strategies -

Organizations have spent a large portion of time measuring and reporting data long before the TQM movement ever came along. Those who have followed Baldrige Award criteria as their roadmap for improving their organization have learned that selecting the vital few metrics on which to collect data is what is important, rather than collecting data on everything.

Creating a balanced scorecard... It is also important to have what Robert Kaplan and David Norton refer to (Sept./Oct., 1993 Harvard Business Review) as the "balanced scorecard." Kaplan and Norton suggest that most organizations who do not have a balanced set of measures, concentrate almost exclusively on short-term financial measures and ignore longer-term more strategic measures such as customer satisfaction, employee satisfaction, and growth.

A balanced set of criteria boxes -

The Baldrige Award criteria (in Section 2.0) recommends a balanced set of metrics (suitable for most organizations) that should include roughly the same amount of data in each of the following data drums:

*Customer satisfaction

*Employee satisfaction

*Financial performance

*Operational performance (e.g., cycle time, productivity, et cetera)

*Product/service quality

*Supplier performance

*Safety/environmental/public responsibility

The usual or obvious 80/20 isn't enough work anymore... In most organizations, about 80 percent of their measure are for two drums: financial and operational. Once a year they do a customer satisfaction survey and once every three years they do an employee morale survey. They have a few statistics on product quality, safety, and inspect incoming supplier shipments. This is just what we mean by not having a balanced scorecard.

I have developed the following survey to overcome the short-sighted and unbalanced way of measuring an organization's performance. The survey will assist you in evaluating whether you currently have a balanced set of measures and how you are using current data to improve organizational performance.

By completing this self-assessment instrument you will learn the characteristics of an effective measurement system, and how well yours stands up to these standards and practices.

Taking measure of your measurement system

Questionnaire directions -- The questionnaire is divided into three sections, each addressing an aspect of your measurement system.

* Part I (questions 1-5) is about your overall approach to measurement.

* Part II (questions 6-41) questions ask about specific types of measures.

* Part III (questions 41-51) is about how you analyze and use the data to improve your organization.

Read each statement and check the appropriate box, depending on the extent to which you strongly agree (5) or strongly disagree (1)with the statement. Answer every question even if you have to guess.

The scope of the questionnaire should pertain to your entire organization, or at least a large enough portion of the company or organization that could be a stand-alone business/organization. For example, you could do a business unit rather than the whole company, or one hospital in a chain of hospitals. You could and should not use the questionnaire to apply to a single department such as radiology, or human resources.

Part I: Overall approach to measurement

1. Our organization has developed a specific set of criteria for screening out extraneous measures from our data base.

-- 5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

2. Our data base was built with a plan, rather than something that just evolved over time.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

3. Our CEO or President looks at no more than 20 measures every month to evaluate the overall organization's performance.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree

4. Measures of performance are mostly consistent across our business units/locations.

--5) Strongly agree 4) Agree 3) Somewhat 2) Disagree 1) Strongly disagree