"You don't need to test COTS components" and other myths: weighing the T&E benefits and risks for commercial and nondevelopmental items - Test and Evaluation - Brief Article

Program Manager, March, 2002 by Holly R. Mangum

Imagine yourself as the program manager for a new acquisition. The contractors proposal includes use of commercial and nondevelopmental items (NDI). Your test engineer is concerned about the implications of incorporating these items into the overall system. Will there be test and evaluation (T&E)? If so, what kind? How extensive should the tests be? Your T&E concerns will need to be incorporated into the applicable acquisition strategy and Test & Evaluation Master Plan (TEMP) documents. Before developing your acquisition strategy though, you must weigh the T&E benefits and risks of using these items. This article highlights some examples of the benefits and risks for T&E considerations, particularly for acquisition strategies that include commercial items and NDI.

By Law

First, understanding what commercial items and NDI are is important to understanding why we need to consider them in developing our acquisition strategies. From there, we can examine the implications on the T&E community

The Federal Acquisition Regulation (FAR) Part 2.1 defines a commercial item as:

"Any item, other than real property that is of a type customarily used for non-governmental purposes and that (1) has been sold, leased, or licensed to the general public; or, (2) has been offered for sale, lease, or license to the general public."

The item can be evolved from a commercial item but not yet available for the commercial market. It can also be modified as long as it does "not significantly alter the nongovernmental function or essential physical characteristics of an item or component, or change the purpose of a process."

A commercial item is NDI, if tne procuring agency determines the item was developed exclusively at private expense and sold in substantial quantities, on a competitive basis, to multiple State and local governments."

Title VIII of the Federal Acquisition Streamlining Act (FASA) of 1994 (Public Law 103-355) implemented the Federal Government's preference for the acquisition of commercial items. According to FAR 12.1, it also established acquisition policies "more closely resembling those of the commercial marketplace...."

Business Environment

In their article entitled, "Solutions: Opportunities and Obstacles," published in the March 2001 issue of The Edge Perspectives, J. Clapp, A. King, and A. Taub state that the "commercial sector has reorganized, restructured, and adopted revolutionary new business and management practices in order to ensure its competitive edge in the rapidly changing global marketplace."

DoD, by way of new laws enacted and senior leadership policies in support of acquisition streamlining, has responded to this shift as well. In the 1997 Report of the Quadennial Defense Review, former Secretary of Defense William Cohen stated that DoD must adapt to this new marketplace. This adaptation has represented a major operational and business paradigm shift from a customized, proprietary model to a commercial, open market model. As a result, the T&E community has actively responded to these changes, providing policy, guidance, lessons leamed, and best practices to aid in T&E of commercial items and NDI.

Policy

DoD 5000.2-R, Mandatory Procedures for Major Defense Acquisition Programs (MDAPs) and Major Automated information System (MAIS) Acquisition Programs, updated in June 2001, states that:

"T&E on commercial and nondevelopmental items shall ensure performance operational effectiveness, and operational suitability for the military application in the military environment, regardless of the manner of procurement. Test planning for these items shall recognize commercial testing and experience, but nonetheless determine the appropriate DT&E [Developmental Test & Evaluation], OT&E [Operational Test & Evaluation], and LFT&E [Live Fire Test & Evaluation] needed to assure effective performance in the intended operational environment."

Misconceptions

When FASA was enacted, the use of commercial items and NDI was expected to result in acquisitions that were significantly faster, better, and cheaper. Since the items were already developed, there really wasn't a reason to conduct further systems engineering or testing. These misconceptions are still there today

The U.S. Air Force Scientific Advisory Board recently completed a report on the successful implementation of Commercial Items in Air Force Systems. During their interviews, which encompassed 34 programs and organizations, they uncovered a common myth, namely: "You don't need to test COTS [Commercial Off-the-Shelf] components." But as noted in DoD 5000.2R cited earlier, T&E is still required to ensure that the item will perform its intended military application. As the program manager, you might want to consider, when developing your own acquisition strategy, some of the following benefits and risks associated with T&E of commercial items and NDI:

BENEFITS FOR T&E OF COMMERCIAL ITEMS AND NDI

* Government need for testing is reduced since commercial market has already accomplished functional testing.