Contractors and operational testing: A tester's perspective - Test and Evaluation - contractors and operational testing, John Stoddart, United States Army - Brief Article

Program Manager, Nov-Dec, 2001 by Steve Elliott

This article is a follow-up to an article by retired Army Cal. John Stoddart, from the March-April 2001 issue of PM, entitled "Contractors and Operational Testing -- Some Involvement is Legal and Beneficial." Stoddart makes a number of points regarding situations where it is appropriate, and even good business, to involve contractors in operational testing (OT) of the systems they build. Without question, contractors play a central and sometimes under-appreciated role in creating the technologies, capabilities, and systems around which we mold our force.

The government cannot do it alone -- every phase of our materiel development and acquisition process requires our contractors' vision, expertise, and industrial capacity. This is equally true for the Production and Deployment phase, which includes OT.

Level of Participation

Having acknowledged this, however, we must recognize that the appropriate level of contractor involvement in operational testing is variable, and depends on the nature of the event. It can range from very high to very low, or almost no involvement at all. This is so because because different kinds of operational tests and experiments serve different purposes.

Exploratory

Some tests and experiments are exploratory in nature, and may be conducted on non-production-representative systems. The Army typically uses these early in the acquisition life cycle to evaluate concepts, identify problems, and help develop requirements or address specific, anticipated issues.

Others, including Initial Operational Tests (IOT), Follow-on Operational Tests (FOT), and to some extent, Limited User Tests, are confirmatory in nature. These are typically field tests of production or production-representative systems conducted after the Milestone C decision, under realistic operational conditions, to verify a system's effectiveness, suitability and survivability when operated and maintained by typical user personnel.

Confirmatory

When dealing with this latter category, test officers must be extremely careful, and make tough, up-front decisions about the extent to which contractors should participate. There are strong arguments for excluding contractors from some facets of OT.

Stoddart points out that "operational test and evaluation is the field test, under realistic combat conditions, of any item ... for the purpose of determining its effectiveness and sustainability ... for use in combat by typical military users; and the evaluation of the results of such test." But there is a key assumption to be made about the expected operating environment (i.e., those "realistic combat conditions"). The assumption is that OT should replicate, as closely as possible, a combat environment in which soldiers will use the system. If that environment will include contractors, then their presence on the testing "battlefield" may be appropriate. If it will not, then the opposite is true.

Controlling the Environment

Stoddart contends that "strict application of the law [that prohibits persons employed by the contractor from being involved in OT] places an unnecessary 'veil of secrecy' on the whole process. Then he goes on to suggest that "[lack of] contractor involvement in the operational test phase will hinder acquisition streamlining ... [because it forces the acquisition community] to wait until the end of test before any fixes can be applied and tested."

Except in cases where operations security is a concern, government testers and evaluators should never cast a veil of secrecy over operational testing. But testers and evaluators do have a primary responsibility to control their test environment, and it may sometimes be necessary to restrict the groups and individuals who have access to test plans and events in order to preserve that environment. Contractor involvement may be appropriate for some exploratory tests. But this is less likely to be the case in confirmatory tests, where the object is to determine how well soldiers can use the system on their own, in an "as fielded" condition.

On Stoddart's second note, it's helpful to remember that IOTs and FOTs are not intended as tools for system development; their goal is to demonstrate conclusively that the system, as developed, is operationally effective, suitable, and survivable when employed by typical user soldiers in the expected operating environment. The time for partnering on system development is before IOT or FOT.

The differences between operational and developmental testing are critical. Developmental testing (DT) tends to be tightly controlled and executed through strictly defined procedures. This is not surprising, since one of DT's main functions is to gauge how well systems conform to precise contract specifications. In comparison, OT is relatively uncontrolled. Soldiers or units are issued the system(s) and logistics support, and then trained and tasked to conduct missions as they would in combat.

Operational testers allow soldiers to "do what soldiers will tend to do" with the system, because the "real world" operating environment most closely replicates the expected and relatively unconstrained combat environment. Why? Because operational testers look at the system as soldiers will use it, not as it complies with contract specifications. And operational testers aren't just testing the contractor's hardware and software; they're testing the comprehensive system-of-systems, comprising all the factors of Doctrine, Training, Logistics, Organization, Materiel, and Soldier Sustainment (DTLOMS). The task of OT is to confirm that it all works, all together, and all at the same time.