Test and evaluation lessons learned from the field

Defense AR Journal, Sept, 2007 by Karen M. Stadler

FUNDING, BUDGET, AND COST (46.8 PERCENT)

This group of lessons dealt with the importance of adequate funding and the negative effects of inadequate funding on T&E programs. Student comments are as follows:

* With the current DoD budget situation, one of the hardest hit areas is T&E. But money and time saved by cutting corners up front is invariably spent in fixing problems later. Customers are unhappy, and it leads to increased cost in the longrun.

* Because of funding limitations, testers commonly execute only a small subset of the test events that should be required. Lack of funding results in delayed testing, test events that are limited in scope, data or reports that are not delivered, cancelled test events, and/or a nonrobust test program that fails to find the critical problems or issues. Unforeseen test requirements sometimes arise for which funding is unavailable.

* Seek to document and educate management and leadership that cost savings obtained by reduced testing may compromise or jeopardize final system quality or operation. And reduced testing, which allows systems to be fielded with undetected problems, could potentially endanger lives.

* To mitigate the risk of inadequate funding, seek to determine realistic cost estimates early in program development. Conduct thorough budget planning and review, consult with experts as needed, and include adequate management reserve for unanticipated problems. If there are cost and schedule constraints, fully document the impact. Include risk analysis and cost/benefit analysis to prioritize limited resources.

SCHEDULE (40.5 PERCENT)

This group of findings dealt with the importance of an adequate schedule and the negative effects of inadequate schedule on T&E programs. Testing is often hindered because of inadequate time available for testing. Overly optimistic schedule estimates commonly lead to this problem as well as unforeseen problems, which decrease the time available or increase the time needed for testing. Program managers sometimes curtail testing in order to make up for lost schedule time. Students said the following:

* Take the time to understand and develop a realistic plan and schedule, including consulting with experts as necessary. Plan for realistic test and program schedules, with time allowed for things such as maintenance, bad weather, and crew rest. Add cost and schedule contingency to each activity, not just at the end.

* Communicate the need for adequate scheduling, including probable effects if needed or planned testing is delayed or cancelled. A limited test program often results in higher overall program costs and longer overall schedule because discovery of problems is delayed. If there are schedule constraints, fully document the impact. Include risk analysis and cost/benefit analysis to prioritize limited resources.

COMPARISON WITH OTHER STUDIES

A literature search was conducted, and the results from this study were compared with the results from five other similar studies and documents. These studies and documents all examined and/or presented T&E best practices and lessons learned. A short description of each study and document follows: