An automated test evaluation tool

Hewlett-Packard Journal, June, 1997 by Jorg Schwering

The AutoCheck program extends the automated test environment described in this journal in 1991.[1] It fully automates the evaluation of the test protocol files generated by the AutoTest program.

Fig. 1 is a brief summary of the 1991 system. The main part of the figure shows the main elements of the AutoTest program and its environment. The AutoTest program reads and interprets commands line-by-line out of a test script (a test procedure). Basically there are three types of commands:

* Commands to simulate user input on the monitor under test (keypusher)

* Commands to control the signal simulators, which play the role of a critically ill patient

* Commands to request and log output data from the monitor under test (reactions to keypresses and signals applied).

[Figure 1 ILLUSTRATION OMITTED]

The execution of all commands and the data received from the monitor are logged into a protocol file. Fig. 2 is an example of a test protocol file (one test case only).

[Figure 2 ILLUSTRATION OMITTED]

The upper left block in Fig. 1 indicates how the test scripts are derived from the product specifications. More information on the testing process can be found in the article on page 89. A test script consists of a startup configuration block, which configures the monitor to a defined startup condition, and the test cases. Each test case consists of three parts:

* The actions (keypusher and simulator commands)

* The description of the expected results

* The AutoTest data request commands.

The upper right corner in Fig. 1 shows the automatic evaluation tool, which is the subject of this article.

Manual Evaluation

In the test environment of Fig. 1, the test engineer had a tool that ran sequences of test scripts in batch mode, completely unattended, overnight or on weekends. With the introduction of AutoTest, the main effort for the test engineer shifted from test execution to test evaluation, which was done with the help of an editor. The protocol files generated by AutoTest (see Fig. 2) are ASCII text files that are very often larger than one megabyte (some large tests have reached a size of more than 100 megabytes).

The evaluation task was not only tedious and time-consuming but also error-prone and dependent on the knowledge, experience, and even the mental alertness of the evaluator. As a consequence, the manual checks were restricted to a minimum for each test case, which typically meant that unexpected attributes were not detected. Furthermore, for tests that needed to be repeated, the evaluation was normally restricted to a few (one to three) selected repetitions. Statistical tests, such as adjusting an alarm limit randomly and checking the alarm string, generate particularly large protocol files that are difficult to evaluate manually, leading the test engineer to reduce the number of test cases to a minimum.

Goals for an Automatic Test Evaluation Tool

Because of these problems, an investigation was started on a tool that could replace the human evaluator. The goals for this automatic evaluation tool were:

* To relieve the test engineer of tedious, time---consuming manual evaluation and thereby increase efficiency

* To avoid overlooking discrepancies

* To get the test results faster by quicker evaluation

* To increase test coverage through side_--effect checks

* To make evaluation more objective (not tester dependent)

* To allow conditional checks (flexibility)

* To automate local language regression tests (see article, page 109).

Use Model

The basic use model (see Fig. 3) is the replacement of manual evaluation with the automatic evaluation tool. The evaluation of the protocol file runs after the test has finished. The test files already contain the expected results coded in a formal language readable by AutoCheck.

[Figure 3 ILLUSTRATION OMITTED]

Test execution and evaluation now consists of the following steps:

1. Write the AutoTest test script including the expected results in AutoCheck format. The basic test script layout as described above stays the same. The only differences are that some AutoCheck definitions such as tolerances (see "AutoCheck Features and Syntax" below) are added to the startup block and that the description of the expected results has to follow the AutoCheck format.

2. Run this test script through the AutoCheck syntax check to avoid useless AutoTest runs.

3. Execute the test script with AutoTest as usual. The expected results (AutoCheck statements) are treated by AutoTest as comments, which means that they are only copied into the protocol file together with a time stamp.

4. Run the protocol file through the AutoCheck evaluation check, which includes a syntax check. AutoCheck generates a diff file reporting the deviations from the expected results (errors) and warnings for everything that couldn't be evaluated or is suspicious in some other way (for details see "AutoCheck Output" below).

5. If and only if AutoCheck reports errors or warnings, check the protocol file to find out whether the deviation is caused by a flaw in the test script or a bug in the patient monitor under test.