Flight test risk management: lessons learned from the edge of the envelope

Flying Safety, Dec, 2002 by William Gray, Bill Koukourikos, Karl Major

Flight testing. For the vast majority of the Air Force it conjures up images of high-risk flying by test pilots with personalities right out of the Wild West. The reality is that the risk of flight testing at the Air Force Flight Test Center (AFFTC) is carefully managed, and the test pilots are highly disciplined individuals who fly formally scripted flight test profiles. AFFTC breeds a culture of meticulous test and safety planning to manage the inherent risks of flight test.

Although the emphasis on risk management for daily operations is relatively new in the Air Force as a whole, highly effective risk management tools have been in use at the AFFTC for over 25 years. The principles of risk management are incorporated into the early stages of the flight test planning process and continue through the execution of each and every flight test mission. AFFTC's risk management system has produced extraordinary results. A derivative of the system may be right for your unit, allowing you to reduce risk and achieve excellence.

Flight test risk management starts when a concept of testing is first being developed. The AFFTC Test Safety Office provides guidance on how to manage the risks as the test team builds its program. Once a program has passed the risk assessment and control phase, the actual testing phase may begin. During this stage, the more "traditional" AFFTC Flight Safety Office oversees flight test operations. How the Test and Flight Safety Offices work to complement each other-and how you, the warrior in the field-can use their highly effective process is what this article is all about.

AFFTC Test System Safety Office

The AFFTC Test Safety Office was born out of fire. In the mid-1970s, an A-10 undergoing gatling gun testing was lost when ingested gun gas flamed out both engines. Many people reading this article have seen the slow-motion film of the pilot ejecting from the doomed jet and the subsequent crash in the Mojave Desert. (That A-10 was piloted by then Captain Francis C. Gideon, Jr. He went on to become a Major General and was the USAF Chief of Safety from Jun 97 until his retirement in Jun 00. Ed.) The purpose of the test was to identify situations where the gun would affect engine operation, and sure enough, the testers found one. In hindsight, the mishap was entirely predictable. This realization compelled the AFFTC to thoroughly adopt system safety analysis techniques into the test planning process.

The "Air Force System Safety Handbook" (published by HQ AFSC/SEPP, Kirtland AFB, NM 87117) describes system safety as "a grass roots movement that was introduced in the '40s. It gained momentum during the '50s; became established in the '60s; and was formalized in the acquisition process of the '70s." It has been over 25 years since the AFFTC first applied the current, formalized system safety process to flight test. The process is now an indispensable part of conducting safe flight test operations. The reduction in loss rates following the inception of the test safety process has translated into saving dozens of lives and hundreds of millions of dollars.

The AFFTC Test Safety Process was Operational Risk Management (ORM) decades before ORM was cool. Not surprisingly, we have learned many lessons that many benefit other, less mature, ORM process. First, here are the basics of the AFFTC test safety process.

IDENTIFY HAZARDS. Hazard identification is what test planning is all about. Test plans are the route between an unproven system and a proven system. The test team, familiar with the system and its intended use, identifies most of the hazards. Some test hazards cannot be mitigated through test plan changes. These hazards are addressed in the safety plan, which is much like a combination of an executive summary and a staff summary sheet. Signing off the safety plan signals approval of the test plan. Thus, risk management is at the core of the test plan approval process. A complete safety plan documents both the test hazards and the risk mitigation plan.

ANALYZE CONTROL MEASURES. The safety plan, which includes the test plan, is subject to an independent safety review. The Safety Review Board (SRB) is chaired by an experienced flight tester from the AFFTC Test Safety Office and is composed of experienced test pilots, engineers and other experts independent from the test team. This team reviews the test plan and safety plan, identifies additional hazards, recommends additional risk mitigation (or the elimination of risk mitigation that will be counter-productive or unnecessary) and finally, assesses the overall risk of the test.

ASSESS RISK. The final risk assessment is the responsibility of the RSB. The board assesses the overall risk of the test based upon the identified risk, risk mitigation efforts and potential for unknown risks. Risk is assessed as "Low" (no greater than normal operations), "Medium" (greater than normal risk) or "High" (significant risk). Approval authority for thetest plan is based upon this risk assessment, with the Operations Group Commander, Test Wing Commander and AFFTC Commander, respectively. To help assess the risk, the SRB uses the risk assessment chart in Figure 1. The risk category divisions of the chart are intentionally drawn through the different severity and probability boxes. This is an acknowledgement of reality, where severity and probability follow a continuum, unbound by arbitrary divisions. Board members are thus reminded of the subjective nature of their assessment. These lines should also remind the test team and SRB members that although a minor improvement in the safety plan may not cha nge the assessed "severity," "probability" or "risk," it may still reduce the actual risk.