Evaluating Key Performance Indicators Used to Drive Contractor Behavior at AEDC

Engineering Management Journal, Dec 2003 by Brooks, William K, Coleman, Garry D

KPIs are an important part of how business is conducted at AEDC. KPIs at AEDC are supposed to be driven by the strategic plan. Thor (1998), Brown (1996), Kaplan and Norton (1996) and others emphasize the importance of linking measures to strategy. "Before measurement can be of much value, an organization needs a strategic focus" (Thor, 1998, p. 36). AEDC's strategic focus or major areas of emphasis are defined in the AEDC 2001 Strategic Plan (AEDC, 2002). In addition to the mission and vision, the strategic plan defines AEDC's strategy in terms of strategic thrusts. AEDC currently has five strategic thrusts, shown in Exhibit 4.

When KPIs are developed for a particular area of responsibility at AEDC, the definition must include identification of which strategic thrust that KPI supports. That is, each KPI is expected to explicitly contribute to answering the question of how well AEDC is doing for one or more strategic thrusts.

Rating levels are established for each KPIz. The mandate for the use of KPIs called for quantitative indicators. This was interpreted as having indicators that are explicitly defined up front, transparent in terms of their data sources, and relatively objective. Wherever possible the indicators yield a quantitative (i.e., numerical) output. These quantitative outputs are then evaluated relative to predetermined targets and a judgment of desirability is made. A few KPIs are judged as pass or fail only, but most KPIs are judged to fall within one of five rating levels: unsatisfactory, satisfactory, good, very good, or excellent. In addition to the rating levels, each KPI has an associated goal for improvement over past performance. The establishment of a goal associated with a KPI is intended to represent a stretch by the contractor to perform beyond the normal expectation levels of the Air Force. In some cases the goal is above the excellent rating, while others are between the good and excellent ratings. Unlike ratings, goals are not explicitly tied to the current award fee. While the award fee is based on a composite, rather than a pure aggregation, of the ratings, meeting (or not meeting) a relatively large number of the goals may be used to increase (or decrease) the contractor's overall award fee percentage.

This case study focused on computer and communication systems KPIs. The set of KPIs developed for evaluating CCS performance are shown in Exhibit 5. Descriptions of three selected CCS KPIs provide a basic understanding of what constitutes performance for CCS. (1) System Accreditation measures the percentage of communications and computer systems that have been approved by the Designated Approval Authority (DAA) to operate. To obtain approval, a system must pass specific security criteria, to ensure that a system is not vulnerable to computer hacking. Percent accreditation is measured by the number of systems accredited divided by the number of systems requiring accreditation. (2) Unclassified Network Availability measures the proportion of time the primary LAN, used to support AEDC's day-to-day business, is available to AEDC users. Availability is calculated by subtracting the time a network device is down from the total possible available time, and dividing the difference by the total possible available time. (3) PC Hardware Repair Time is the average time it takes to repair a PC once the problem has been reported to the Help Desk. The Help Desk takes the initial call and issues a trouble ticket to the hardware maintenance personnel. Upon completion of the work, a hardware maintenance person enters a resolution and closes the ticket. All times associated with the trouble ticket are logged and tracked by the automated information system.