Test and evaluation lessons learned from the field

Defense AR Journal, Sept, 2007 by Karen M. Stadler

* Include tests at realistic operating conditions and at all corners of the envelope. These tests should occur prior to production or as early as possible. Measure all critical parameters and verify all requirements. Review the Test and Evaluation Management Plan (TEMP) and test plans versus requirements (e.g., was essential data collected and nonessential data not collected?).

TEST PLANNING (53.8 PERCENT)

Numerous students stressed the importance of thorough test planning. There are things one cannot or will not anticipate; stuff happens, people make mistakes. But proper anticipation and thorough planning will result in fewer problems in the long run, including a better chance of success and remaining within cost and on schedule. Good planning allows proper resource allocation and makes test execution far easier. Additional student comments concerning the importance of test planning and not cutting corners are as follows:

* Any test plan should have more than one person to review it. Make use of available expertise.

* Do analysis prior to testing to try to predict and anticipate results. This analysis will assist in identifying potential problems and developing contingency plans. It also helps identify needed changes to test plans.

* Plan for contingencies, especially weather. Have an alternate test plan/alternate test points available for each day of testing.

* When planning a test with others, start early. They were already busy before you came. Attempt to keep things simple. Questions should be clear and to the point. Follow their processes, if possible.

* Become an expert on systems you are testing. Tap into subject matter expert (SME) knowledge. Spend time with the user. Find out the user's priorities, concerns, and the reasons for these. Consider traveling to the factory or contractor facility. Obtain and study system documentation. Discuss design criteria with contractors, SMEs, users, maintainers, etc. The smarter you are about the system, the better decisions you will make, and others will not be able to hide issues that need to be brought out into the open.

* Early operational test agency (OTA) involvement is critical to reducing OT risk. The OTA should/could assist in requirements and concepts of operations (CONOPS) development, in early identification of T&E related systems concerns, in providing input aimed at conducting selected developmental test (DT) events in a more operational environment, etc.

* Plan for proper training, and make the case for extra training time, if needed. Training usually results in a better and safer test, better information, and better decisions. It almost always pays off in the long run.

TEAMWORK, COMMUNICATION (46.8 PERCENT)

Many students cited the importance of teamwork and communication as critical to minimizing and/or preventing T&E problems. Frequent, open, and timely communication (integrated product team [IPT] meetings or otherwise), along with consulting with SMEs when needed, undoubtedly increases the chance of program success. Students had a number of recommendations concerning the importance of teamwork and communication: