Test and evaluation lessons learned from the field

Defense AR Journal, Sept, 2007 by Karen M. Stadler

This article examines test and evaluation (T&E) lessons learned from more than 300 students with extensive T&E field experience who attended the Defense Acquisition University (DAU) test and evaluation classes during FY02-FY05. The T&E lessons learned in 18 categories were researched and correlated, and findings in the top five categories are presented. In particular, this article focuses on detailed lessons learned in the areas of test design and execution, test planning, teamwork and communication, funding, and scheduling. A compilation of student (field practitioner) comments and recommendations is presented, and overall results are compared with results from other similar studies and documents.

**********

As part of the DAU Advanced Test and Evaluation (TST 301) class, students prepare and present PowerPoint slideshows on various T&E-related topics. Many students prepare and present detailed briefings on their T&E lessons learned. The lessons-learned presentations are typically based on actual experiences in planning, conducting, analyzing, and reporting test results. Students typically have many years of T&E/acquisition field experience, and their presentations contain a wealth of valuable information, which could help others avoid common sources of error when designing and executing test events. The purpose of this article is to identify and discuss common T&E best practices and lessons learned, thereby enabling possible cost and schedule savings and improved test results from future T&E efforts. Decision makers and acquisition/program leadership can benefit from this article by better understanding the top T&E related issues, as reported by field practitioners.

The T&E lessons-learned data was obtained from students who attended this author's TST 301 classes during FY02-FY05. Of the 393 students, 301 students presented significant T&E lessons-learned information. The students came from all four services and DoD agencies, as summarized in Table 1. The lessons-learned data consists solely of student comments and opinions and is based on student knowledge and experience in the T&E area, along with any research conducted by individual students.

The student data was analyzed for common trends, and 18 different categories, covering all common trends, were selected. The student briefings were then tabulated, to determine the number of student briefings with lessons learned for each of the 18 categories. For example, 192 of the 301 total students (63.8% of the total) had significant lessons learned in the area of test design and test execution. Note that each student briefing contained lessons learned from one or more of the 18 categories. The student lessons learned data is presented in Table 2.

MAJOR FINDINGS

Of the 18 categories of T&E lessons learned in Table 2, this article will further discuss the top five categories (test design and execution, test planning, teamwork and communication, funding, and schedule). Note that far more students (41% 64%) had lessons learned in the top five categories than in the bottom 13 categories (5%-30% of students). This article contains a summary and detailed student comments and recommendations for each of the top five areas. This information can benefit the T&E community by providing detailed lessons learned, which might assist future T&E efforts and help acquisition leadership better understand the major T&E issues and concerns.

THE TOP FIVE STUDENT CATEGORIES

TEST DESIGN, TEST METHODS, TEST EXECUTION, AND ANALYSIS METHODS (63.8 PERCENT)

Test design, methods, execution, and analysis methods are unquestionably among the largest factors that determine the success or failure of test events. Students offered the following advice in this area:

* Take the time to develop a robust T&E strategy and to determine the best (anticipated) test design and methods for your situation. Consult experts where necessary. Examine all facets of T&E such as instrumentation, data collection, analysis methods, test validity, test procedures, etc. For example, one might consider a side-by-side comparison of the existing and the new system. The new system may not meet all the requirements, but may be significantly better than the existing system. Without the comparison, the true conclusion may be missed.

* The devil is in the details. Student after student indicated that the little details (as well as the overall test design and execution) greatly affected the success or failure of their test events. For example, one detail of test planning might be to consider collecting diagnostic data, as well as test data. In case of equipment failure, the data can assist in determining the problem(s).

* Understand the test objectives, including how and why the test requirements were generated. Know the what and why behind limits and guidelines. Look at the system requirements, don't just accept them. Spend the time upfront with users and requirements developers to get the requirements well defined, especially where the requirements don't make sense. Often, the user may not know what he/she wants and why. Is the test relevant? Are the requirements realistic? The user will sometimes change the requirements, if the need is explained to them. Systems engineers and testers need to work together.