An index to measure and monitor a system-of-systems' performance risk

Defense AR Journal, Dec, 2005 by Paul R. Garvey, Chien-Ching Cho

This article extends an earlier published methodology (Garvey & Cho, 2003) for measuring the technical performance risk of a system to that of a system-of-systems (SOS). The earlier work established an approach for combining an individual system's Technical Performance Measures (TPMs) into an overall measure of performance risk, defined as the Technical Risk Index (TRI). This article extends this approach so a similar index can be developed to assess a system that is composed of many interdependent or connected systems that come together as a whole to provide an SoS capability.

**********

Technical Performance Measures (TPMs) are traditionaily defined and evaluated to assess how well a system (or a system-of-systems [SOS]) is achieving its performance requirements. Typically, dozens of TPMs are defined. Although they generate useful information and data about performance, little is available in the system engineering and program management communities on how to integrate these measures into a meaningful measure of overall performance risk. This article presents how individual TPMs may be combined to measure and monitor the overall performance risk of a system. The approach consists of integrating individual TPMs in a way that produces an overall risk index. The computed index shows the degree of performance risk presently in the system. It identifies risk-driving TPMs, enables monitoring time-history trends, and reveals where management should target strategies to lessen or eliminate the performance risks of the system.

As a system evolves through its acquisition and deployment phases, management defines and derives measures that indicate how well the system is achieving performance requirements. These measures are known as Technical Performance Measures (TPMs) (Defense Acquisition University, 2002; Blanchard & Fabrycky, 1990). Measures such as Weight, Mean-Time-Between-Failure, and Detection Accuracy are among the types of TPMs often defined. The TPMs can be taken from a variety of sources. This includes data from testing, simulations, and experimentation. Depending on the source basis for these data, and the development phase, performance data may be derived from a mix of actual or forecasted values.

As mentioned previously, the system engineering and program management communities have little in the way of methodology for quantifying performance risk as a function of a system's individual TPMs. The approach presented herein consists of computing a risk index derived from these individual performance measurements. The index shows the degree of performance risk presently in the system (or SOS), supports identifying risk-driving TPMs, and reveals where management should focus on improving technical performance and, thereby, lessen risk. When the index is continuously updated, management can monitor the time-history trend of its value. This enables management to assess the effectiveness of risk reduction actions over time.

In general, TPMs are measures that, when evaluated over time, must either decrease to meet performance requirements or increase to meet performance requirements. Thus, each TPM can be assigned to one of two categories. For this paper, Category A is defined as the collection of TPMs whose values must decrease to achieve threshold performance requirements. Category B is defined as the collection of TPMs whose values must increase to achieve threshold performance requirements.

It is assumed that TPMs are defined judiciously; that is, only those TPMs truly needed to properly measure overall technical performance are defined, measured, and monitored. Given this, acceptable performance risk can be defined as the condition when all TPMs reach, or extend beyond, their individual threshold performance values. Conversely, unacceptable performance risk can be defined as the condition when one or more TPMs have not reached their individual threshold performance values.

A GENERALIZED PERFORMANCE RISK INDEX MEASURE

The following presents a generalized index designed to measure the performance risk of a system or SoS. The index can be applied in both contexts. It provides a numerical indicator that measures how well a developing system is progressing toward its threshold performance requirements. It serves as a yardstick that enables management to measure the "distance" the system is from its minimum performance thresholds and to monitor trends over time.

To develop the generalized risk index, it is necessary to first normalize the TPM "raw" values into a common and dimensionless scale. This scale transformation is done for each TPM in each category. This allows management to compare the progress of each performance measure in a common and dimensionless scale. From these normalized scales, an overall measure of the extent to which the performance of the system meets its threshold requirements can be determined. The following general formulas illustrate how to derive this measure. They are followed by a computation example to illustrate the application context.