Evaluating Key Performance Indicators Used to Drive Contractor Behavior at AEDC

Engineering Management Journal, Dec 2003 by Brooks, William K, Coleman, Garry D

Step 2. Compare the Existing Measurement System to the Ideal Model. Step 2 of the evaluation process (Exhibit 1) collected data and compared the existing CCS measurement system (i.e., the KPIs and their use) to the ideal model defined in Step 1. Several top management and middle management personnel from both the Air Force and support contractor were interviewed to collect data for evaluating the measurement system against the dimensions of the ideal model. The purpose of these interviews was to document the managers' perceptions of linkages between KPIs and the strategic plan, the intended and actual impact of the KPIs used by the SDC organization, identify problems areas, and determine the overall effectiveness of the existing KPIs. Interviewees were selected based upon their role(s) in the KPI and award fee processes. Individual job managers from both the contractor and Air Force, who were responsible for developing the individual KPIs, were selected. From the contractor, in addition to six job managers, the Director of CCS, the financial analyst for CCS and the general manager for ACS were selected for interviews. These are the key contractor personnel who are directly evaluated based on the results of the KPIs. From the Air Force side, in addition to the five job managers, the chief of communications and computer support, the technical area manager (TAM) for the ACS contract, and the chief information officer (CIO) for AEDC were selected because of their role in the award fee process as well as their responsibility to the commander for daily operations at AEDC. In addition, direct observation and document review were used to identify and describe the current state and uses of CCS KPIs. Everyone interviewed to identify characteristics of the existing measurement system was also asked to complete all three parts of Brown's (1996) self-assessment questionnaire. All interviewees except two responded to the questionnaire.

Exhibit 6 provides a summary of the results from comparing the CCS measurement system to the dimensions of the ideal model. The following sections describe in detail how well the CCS KPIs met the dimensions of the ideal model. The first two sections describe the results from reviewing the existing CCS KPIs and auditing them against the identified characteristics (including how KPIs were chosen) and measurement categories of an effective measurement system using information obtained from the interviews and surveys. The next two sections describe the intended and actual impacts of the KPIs, based on observations and interviews of key Air Force and contractor personnel. The final section describes the results from evaluating the KPIs against Wheeler's (1993) principles and the prescriptive standards from Thor (1998) and Brown (1996) regarding how data should be presented to support interpretation and provide feedback.

Step 3a. Evaluate Results. Evaluating the results from the comparison in Step 2 led to the identification of a number of implications. These implications are discussed in the following subsections, organized along the dimensions of the ideal model. Discussion of the "how measures are chosen" dimension and the "characteristics of a good performance measurement system" dimension are combined.