Single Index Proposed to Measure Safety of Aviation System

Air Safety Week, Sept 15, 2003

A new aviation safety index - analogous to the famous Dow Jones measure of stock performance - will be used to measure the risk of death and injury in the aviation industry. Developed by the Federal Aviation Administration (FAA), the index applies to all people exposed to risk from the civil aviation system.

"This new index will serve as a vital trend indicator," according to the FAA's draft Flight Plan 2004-2008. This document lays out the agency's strategic goals. The safety index is touted as a tool to "help provide a more robust indicator of the state of aviation safety."

The index builds upon the work of Professor Arnold Barnett of the Massachusetts Institute of Technology (MIT), who for a decade has advocated quantitative measures of safety for the nation's air transportation system. Barnett supported the approach: "I applaud the idea of a statistic that summarizes recent death and injury risks over various forms of aviation. Of course, no one indicator is perfect: Some up-and-down movements over time will reflect little more than sheer luck, while some growing menaces might elude the statistic because they have not yet caused any accidents. Moreover, the initial version of the FAA indicator has to cope with data limitations we should strive to circumvent. But, while the indicator isn't everything, is there any other place to start a serious appraisal of system safety besides a clear 'bottom line' statement about how well we're doing now? Fortunately, the FAA thinks not."

Other experts were more skeptical. ASW contributing editor Alex Richman (AlgoPlus Consulting Ltd.), a veteran massager of the aviation service difficulty report (SDR) database, noted that the FAA posed a question for which the answer is questionable. In an FAA internal memorandum explaining how the safety index is computed, the question was asked in the section titled "Next Steps" whether a "single index, regardless of its nature" should be pursued.

"Clearly," the memorandum said, "the FAA believes the answer is 'yes,' as the work outlined in this document would suggest."

Richman demurred: "This conclusion is hard to understand. The document ... demonstrates that it is possible to create [a single index], but not that it is valuable."

Richman pointed out that the Dow Jones index embodies ten sub-indices of various industry sectors "to supplement the single index." A similar approach could be taken for aviation safety, he argued, since the relative safety records vary considerably between the large scheduled carriers and, say, general aviation (GA).

Other experts offered guarded plaudits for the concept, while voicing reservations about the computational details.

"Establishing a statistical level of risk from the perspective of aircraft occupants is valid and worthwhile, and we like the definition of risk as 'frequency plus exposure,' " said Nick Lacey, a consultant with Morten Beyer & Agnew in Virginia. Lacey was formerly the associate FAA administrator for regulation and certification and is familiar with the strengths and pitfalls of various approaches to quantifying air safety.

"This attempt is a good start. It also shows how difficult it is to develop a taxonomy," he added. Lacey said "there is too much in the bucket" and that the index should be limited to U.S. airspace, U.S. airports and U.S. operators. This approach would provide a more meaningful index "from the passenger's perspective," he asserted. For example, the index in its present form includes events like the fatal 1995 crash of an American Airlines [NYSE: AMR] jet on a mountain ridgeline near Cali, Columbia. "The index should [only] cover those factors over which the FAA has control," Lacey argued. By including flights that progress out of U.S. airspace, the FAA is incorporating into its safety index situations over which it has no control, such as out-of-country traffic separation and terrain separation.

Paul Hayes of UK-based Airclaims said, "The FAA is to be applauded in seeking new ways of handling and presenting accident statistics, and introducing some way to weight accidents for severity is the way to go."

"However, I do not see the point to having a single index which covers all types of aviation," he added.

Details of Lacey's and Hayes' concerns about methodology suggest that they share the same goal for any measure of safety - relevance and credibility.

Gary Eiff, an associate professor in the department of aviation technology at Indiana's Purdue University, also questioned the value of a single index. "Why are we doing this? It appears like the FAA is trying to show that things are getting better for its efforts, but there is a such a hodge-podge out there," he said, referring to aviation activity that runs the gamut from scheduled air carriers to enthusiasts flying their homebuilt ultralights.

Eiff added that the absence of incidents in the computations is a shortcoming. Incidents are often harbingers of disasters. "The only difference between an accident and an incident is circumstance," Eiff said. The difference between a runway collision and an incursion, or a mid-air near miss or collision, is frequently "a matter of a few feet or a few seconds," Eiff pointed out.