Evaluating Key Performance Indicators Used to Drive Contractor Behavior at AEDC

Engineering Management Journal, Dec 2003 by Brooks, William K, Coleman, Garry D

In addition to evaluating a contractor, KPIs are also used as a means of providing feedback regarding how the contractor performed over a given period. In evaluation of feedback systems, Thor (1998) suggests four areas that need addressing: (1) flow and frequency, (2) accuracy and gathering effort, (3) individual and group, and (4) style and format. Observations and document review are used to evaluate the KPI presentation style and how well the KPIs address Thor's four areas for providing feedback.

Step 2. Compare the Current Measurement System to the Ideal Model. Having defined the ideal model in Step 1, one must understand the current measurement system before comparing the two. Such understanding begins with describing the organization's unique context. Describing the organization's context is an iterative process and is initiated as the data collection methods are developed in Step 1. This continues into Step 2 as key personnel are interviewed to collect data about the measurement system, and also to gain further understanding of the unique context of the organization. As new facts about the organization are uncovered, the data collection methods may be revisited to incorporate additional personnel in interviews or the survey. So understanding organizational context actually starts in Step 1, continues during data collection in Step 2, and may lead to revising data collection methods from Step 1. The output of understanding context is a description of the organization, how it develops and uses its measurement system, and an inventory of the relevant performance measures (KPIs).

The comparison begins with collecting data defining the existing measurement system. Document review, interviews, surveys, and direct observation are the primary methods used. The current measurement system is defined along each of the dimensions used to describe the ideal model. Dimension by dimension, the current system and the ideal model are compared to identify gaps and opportunities for improvement. The output of this comparison is a combination of tabular summary and narrative description. Summary tables are useful for point-by-point findings, while narrative explains the subtleties of how the current system meets the intent of the ideal model.

Step 3. Evaluate Findings and Recommend Improvements. The evaluation identifies the implications of the findings from the comparison in Step 2. Implications include judging the severity or seriousness of gaps with the ideal, assessing the effectiveness of the current system in meeting the intent of the ideal, and interpreting what the findings mean for the organization. A primary value of the measurement system evaluation process is the identification and recommendation of improvements aimed at closing gaps and increasing system effectiveness. Examples of the types of improvements that are produced in this step are shown in Step 3b of the following case study (where Step 3a was the evaluation of findings).

Case Study: Arnold Engineering Development Center