Evaluating Key Performance Indicators Used to Drive Contractor Behavior at AEDC

Engineering Management Journal, Dec 2003 by Brooks, William K, Coleman, Garry D

Abstract: This article presents a process, demonstrated in a case study, to evaluate an organization's performance measurement system. The process consists of three steps, roughly following the plan-do-study-act cycle. An ideal model of a measurement system was developed based on integration of concepts from the literature. The model was used as a basis of comparison to evaluate the measurement system used by the Support Directorate for Communications and Computer Support (SDC) at Arnold Engineering Development Center. The SDC Key Performance Indicators (KPIs) measurement system is used to drive behavior and evaluate contractor performance. Results from the initial evaluation and recommendations for improving SDC KPIs are presented.

Performance measurement is of increasing interest to managers due to the changing nature of work, increasing competition, specific improvement initiatives like six sigma and strategic planning, national and international awards, changing organizational roles, changing external demands, and the power of information technology (Neely, 1999, p. 210). Contracting out substantial parts of work processes is one way organizations deal with the changing nature of work, increasing competition, and changing external demands. Managing the performance of contractors cannot depend solely upon managing costs, as the public and "markets are driven more by value than cost" (McAdam and Bailie, 2002, p. 974). Thus a comprehensive view of performance must be taken to manage these contractor operations, requiring a comprehensive set of measures. The question of what to measure has long been raised by performance measurement experts (Brown, 1996; Kaplan and Norton, 1996; Neely, 1999; Sink and Tuttle, 1989; Thor, 1998). Most include deriving measures from strategy as part of their approach. But once the measures are chosen, they should not remain static as they too often do. Without reviewing and updating performance measures periodically, the measurement system can fail to evolve to meet the changing needs of the organization. "One of the greatest weaknesses in the field of business performance measurement today is that few people are actively exploring the issue of how the evolution of measurement systems can be managed over the long term" (Neely, 1999, p. 223). The evaluation and case described in this article are aimed at illustrating and increasing our understanding of how we might manage the evolution of performance measurement systems.

The increasing use of outsourcing, strategic partnering, and government contracting is changing the way organizations manage their performance. These "contract operations" cause the principal organization to depend upon the performance of another organization for the execution of all or part of the principal's mission. Determining how well the contractor is performing and how that performance contributes to the principal's mission is and will continue to be a challenge. This article describes a study to evaluate and improve how one government organization, Arnold Engineering Development Center (AEDC), assesses and judges the performance of its primary contractors using key performance indicators (KPIs). Engineering managers with existing performance measurement systems can use the evaluation method offered here to assess and improve their measurement systems. The approach employed here begins to address some of the performance measurement challenges faced by organizations, whether for internal or contractor operations.

Evaluating Measurement Systems

The evaluation process is based on Shewhart's Plan-Do-Study-Act cycle. Exhibit 1 illustrates the three-step evaluation process, with the third step representing a combination of Study-Act from the Shewhart cycle. The idea behind the development of this process is to establish how to evaluate the current system of KPIs. A limited literature search is performed to establish the dimensions of an ideal model against which the current system can be compared. With an ideal model established, a method for collecting data regarding how the current system compares to the ideal is established. Three methods of data collection are used to compare the current system with the ideal system: personal interviews, a standardized questionnaire, and direct observations. Results from the interviews and questionnaires are analyzed and compared to observations of the current measurement system. Conclusions are drawn based on these analyses and recommendations to management are made based on the conclusions.

The following sections describe the three steps of the measurement system evaluation process. Step 1 describes the ideal model and develops the data collection methods for assessing an existing measurement system against the ideal model. Step 2 is to collect the data and compare the existing measurement system to the ideal model. Step 3 is to evaluate the findings from the comparison and recommend improvements.

Step 1. Define the Ideal Model for a Measurement System. The first step in the evaluation process is establishing the ideal model or standard by which the system will be judged and how data are collected to make this judgment. The dimensions of this ideal model are established to address these questions: