From Our Readers - Letter to the Editor

Program Manager, July-August, 2001

In the March-April 2001 issue of Program Manager Magazine, John Stoddart, an industry member of the Industrial Committee on Operational Test and Evaluation (ICOTE), discussed his vision for changes in contractor involvement (or the lack thereof) in operational testing and evaluation. We may differ in some of the details of how to implement these changes, but we in the Army Test and Evaluation Command (ATEC) agree on the general thrust and intent of the recommendations. However, based on our experience in Army testing and evaluation, we are somewhat surprised at some of the misconceptions and myths found in the article.

The purpose of this letter is threefold -- to add support to Stoddart's basic recommendations, to dispel some of the myths, and to point out some limitations that must be imposed on these recommendations. I will also offer some new challenges for the defense contractor community.

There is really only one reason why we test -- to learn. We are trying to fill the data voids in our knowledge about a new piece of equipment or system. The acquisition community (which includes the contractors) needs to know if the system meets contract specifications; and, more important, to know if it will achieve operational requirements. Army and OSD decision-makers need to know if the system is effective, suitable, and survivable before entering full-rate production. And probably the most critical reason we test is to let those responsible for the system's development and procurement know what improvements are still required to provide our soldiers the best possible equipment.

The knowledge we gain through testing is of no benefit if it is confined within the test and evaluation (T&E) community. We have no capability to correct the deficiencies noted in testing. Knowledge is only useful if it is in the hands of those who have both the capability and the authority to use it. The program manager and his or her contractors cannot improve a system without full information concerning any deficiencies in the design or manufacture of the system. The combat developer cannot develop or correct the tactics, techniques, and procedures (TTP) until the system limitations and problems are clearly articulated. Decision-makers cannot make rational decisions on programs without the knowledge derived from the test arena.

This is the heart of Stoddart's comments. Getting the contractors more closely involved with the T&E organizations will help move the knowledge to where it can be beneficial.

There is another side to this closer tie between the tester and the contractor that should not be ignored. Testers and evaluators cannot do their jobs well unless they really understand the systems they are testing, both at an engineering and an operational level. Closer links with the contractors should improve the base knowledge of the testers and evaluators, allowing them to gain even more insights into the system under test. This can be a true win-win situation.

Stoddart suggests that the contractors need access to the system requirements and T&E planning documents and processes, including the T&E integrated process team (IPT). I couldn't agree more. But, where has the ICOTE been for the past 15 years? In the scores of T&E IPT and TIWG [Testing and Interoperability Working Group] meetings I have attended, it was the exception when contractor representatives were not there, and, in most cases, active participants. There have even been cases where meetings were held in contractor facilities so the T&E IPT members could get a first-hand look at what contractors were doing.

Contractors must understand not just the contract specifications, but also the operational requirements. This means having access to the mission needs statement (MNS), the required operational capabilities (ROC) or operational capabilities document (ORD), the organizational and operational (O&O) concept, and even the critical operational issues and requirements. They should also see the operation mode summary/mission profile and the test and evaluation master plan (TEMP), and even the test scenarios. Stoddart asked why the PM doesn't just give these documents to the contractors. Not only do we have no objections to giving these documents to the contractors, but we also encourage the contractors to study them very carefully. Ask your PM for these items.

I am encouraged to hear contractors asking for these documents, especially the requirements document. Too often the contractors are totally focused on meeting a contract specification and no more. Yes, they need to meet the specifications, but that is not as important as meeting the operational requirement.

Some contractors seem unaware of what the soldier really needs, or how a system will be used on the battlefield. The only way to understand what you are trying to build is to see it through the eyes of the user. Contractors should have a few ex-soldiers on the team who are fully versed in the O&O concept and system requirements and who are constantly looking at every design decision through a soldier's eyes. If not, the contractor will probably fall short in the system design.