Evaluating Key Performance Indicators Used to Drive Contractor Behavior at AEDC

Engineering Management Journal, Dec 2003 by Brooks, William K, Coleman, Garry D

Arnold Engineering Development Center is the world's most complex flight simulation and test center (AEDC, 2002). AEDC is located at Arnold Air Force Base, Tennessee. AEDC is part of Air Force Materiel Command (AFMC), headquartered at Wright Patterson Air Force Base, Ohio. The Air Force and Civil Service manage AEDC, but contractor personnel perform the day-to-day operations. From 1995 to 2003 there have been two primary contractors at AEDC. ACS performs the center support and Sverdrup Technology performs the test mission. The contract between the Air Force and the contractors is a cost-plus award fee contract. The contractor in addition to being paid the cost of performing the work is awarded a percentage above the cost of the contract twice a year based on their respective performance rating. Each contractor's overall performance rating ranges from 0 to 100 percent and is calculated based on the weighted average of their ratings on a family of measures (also known as Key Performance Indicators). Key Performance Indicators (KPIs) have been developed for every major function at AEDC and it is a composite of these KPIs that is used to determine the overall performance of the contractors.

This study evaluated the Support Directorate for Communications and Computer Support (SDC) KPIs associated with the support contractor for the area of Communications and Computer Support. Communications and Computer Support (CCS) provides center-wide communications and computer support services to both test and support customers. These services allow the accomplishment of complex and diverse mission requirements and include Local Area Network (LAN), Wide Area Network (WAN), voice communications, enterprise applications support, PC hardware and software support, mainframe computer support, as well as information assurance (the protection of information). The current KPIs have been used by SDC to evaluate CCS performance for several years, yet some Air Force managers have questioned whether these KPIs effectively and comprehensively measure performance. Equally important, some managers have long suspected these KPIs drive behaviors aimed at simply increasing performance ratings rather than actually improving underlying performance. This also raises the question of what undesired behaviors are these KPIs driving? The study described here evaluated the CCS KPIs in terms of their linkage to organizational priorities and how they were perceived as driving desired behaviors and not encouraging undesired behaviors. While the results directly apply to SDC, the method and recommendations for improvement are expected to be generalizable across AEDC and to other organizations' measurement systems as well.

Step 1. Define the Ideal Model for Measurement Systems. The ideal model and general data collection methods were described under Evaluating Measurement Systems, with detail provided in Exhibits 2 and 3. Specifics regarding data collection methods were refined as understanding of the organizational context developed. The following paragraphs describe the organizational context of the case study: