Evaluating Key Performance Indicators Used to Drive Contractor Behavior at AEDC

Engineering Management Journal, Dec 2003 by Brooks, William K, Coleman, Garry D

Once the categories of measures that should be included are known, the question becomes how are the specific measures chosen? Brown (1996) offers two approaches for choosing performance measures: Top-down, or by-unit. In either approach, how the measures are chosen is based on who does the choosing, which is often a function of the culture of the organization. Each has its advantages and disadvantages. According to Brown, the top-down approach is best suited "where corporate exerts a great deal of control" (p. 141). The advantage of this approach is it is "less likely that there will be disconnects or inconsistencies between corporate measures and those in various units or locations" (p. 141). However, if the culture is such that units have a great deal of autonomy, then the top-down approach will meet with a lot of resistance, and the by-unit approach (which can be thought of as a bottom-up approach) is more appropriate. The main reason is that anything dictated from the corporate level would meet with resistance in an organization that has autonomy. In the by-unit approach a unit or location develops a set of measures that can serve as a prototype for other units.

The chosen approach should ensure some level of consistency among measures across the organization. Thor (1998) describes the consistency among measures in terms of vertical alignment and horizontal alignment. "Vertical alignment refers to consistency and coordination between performance measures of the same category" (p. 45) as they move up to higher organizational levels. "Horizontal alignment is coordination and understanding of the measures for each organizational level across all parts of the organization" (p. 48). This ensures that each part of the organization measures the same thing the same way. So whether the approach is top-down or bottom-up, it should support vertical and horizontal alignment of measures. Data regarding how individual measures are chosen are collected via the interview process.

Indicators are measured, tracked, portrayed and reviewed in order to guide or at least influence behavior. This raises the important question: Are the indicators influencing behavior as intended? To answer this question: first, what are the behaviors the indicators were intended to facilitate? Second, what are the actual behaviors facilitated by the indicators? And are the actual behaviors desirable, whether intended or not? Intended behaviors were identified via the interview process, while actual behaviors are identified via the interview process and direct observation.

Having a measurement system that is in agreement with all of the above-mentioned characteristics is not the end of the story. The data must be presented in a meaningful way to support proper interpretation and evaluation of the results for that particular indicator, especially when comparisons to past performance are being made. Conventional wisdom may advocate that graphics are better than tables of data alone for analyzing and reporting measures; however, graphical displays are subject to numerous potential distortions (Thor, 1998, pp. 71-72). Wheeler (1993) describes statistical methods for presenting managerial data in context, to support analysis and interpretation and reduce the likelihood of distortion. He describes the problem of data containing both signal and noise elements and how XmR charts, for example, can be used to filter out noise and support understanding of the signal. Wheeler's principles and methods can be used to evaluate whether the current KPI presentation style used by the organization support thorough understanding of the performance data.