Applying systematic testing to application development audits

Internal Auditor, Feb, 1991 by Jerry E. Durant

APPLYING SYSTEMATIC TESTING TO APPLICATION DEVELOPMENT AUDITS AS AUDITORS, MANY OF US have felt the frustration of participating in the development of a system, only to be surprised by its failure to meet basic business needs. To make things worse, this failure may be viewed by management as a reflection on our participation. Most of us have probably asked ourselves if we were deficient in some way, or if the failure somehow rests in the systems development effort.

Auditors' involvement in system development was originally based on the concept that early detection of deficiencies would allow for cost-effective correction. This idea hasn't changed. The difficulty centers on how to turn our participation into something more than simply attending meetings with hopes of seeing a control deficiency. Although this technique can be partially effective for obvious problems, it still relies heavily on post-implementation testing.

In 1986 Drs. David Gelperin and William Hetzel developed a testing methodology called STEP (Systematic Test and Evaluation Process). (*1) STEP was unique in that it broadened the definition of "test" to include more than a procedure performed after the system was fully assembled. STEP expanded that original definition to include test planning, test case design, test implementation, and test execution. Large organizations that have a test function have found that this approach provides significant benefits in quality and control. Smaller organizations that don't have a separate test function have also seen remarkable improvements. Much of the time and effort exerted on fixing problems has been sharply decreased by early detection. But more importantly, the test process helps to put all the issues on the table so that fewer open items remain in the later development stages.

How STEP works

STEP provides a process to follow while systems are being developed. The auditor's knowledge of business and technical risk will help in this assessment.

Test specifications can be considered as static behavioral models. Test models are useful in defining what the application is supposed to do and in demonstrating its capabilities. Models also provide for revalidation during the remaining life of the application. The test process makes up one piece of the evaluation segment of software engineering. The other parts, analysis (performed on an individual basis) and reviews (audits, walk-throughs, and inspections that are performed on a group basis), comprise the present involvement of the auditor.

While analysis and reviews provide an adequate degree of control, they rely heavily on the participant's preparation and dedication to these initiatives. Often, effectiveness is greatly diminished by poor management of the review process. Unlike analysis and reviews, systematic testing provides a structured framework for software evaluation. When combined with reviews and analysis, an effective preventative control is deployed.

STEP was developed based on American National Standards Institute (ANSI) 829: Test Documentation Standard and 1008: Unit Test Standard. The process parallels a development effort and can be used to validate the completeness of each phase of development.

STEP sets three major objectives: preventing defects, detecting defects, and demonstrating the capability of the software. In order to meet these goals, the test activity is broken up into levels of testing, which allow for focused attention toward specific system characteristics as they are delivered. There are typically four test levels in most organizations: acceptance test, system test, integration test, and unit test.

Acceptance Test

Assuming that all of the lower levels of test have passed successfully, the acceptance test answers the question, "Is the system acceptable for use?" Both alpha (internal, final validation) and beta (limited external distribution) test activities are parts of acceptance testing.

System Test

The majority of the test effort is spent in system test since the system characteristics are totally available at this level. System test would include validation for: * Capacity. * Concurrence. * Conversion. * Hardware configurations. * Installation. * Interface. * Performance. * Recovery. * Reliability. * Resource Usage. * Security. * Sensitivity. * Software configurations. * Use.

As much as 80 percent of the total time allotted to testing can be consumed by system test activities.

Integration Test

Integration test is targeted toward delivery of major sections of the system. It may also be referred to as "building testing." Once a portion of the system has been delivered, the integration test can start. In most cases, an integration test is approached when a set of functions has been delivered. If a risk is high, or if waiting until the full system is delivered to discover problems is impractical, then the integration test should be performed. This type of test depends heavily on the system-build schedule as established by development. Several strategies for the integration test exist: functional or thread integration test, top down, bottom up, and availability integration. These strategies are driven by factors as listed below: