An automated test evaluation tool

Hewlett-Packard Journal, June, 1997 by Jorg Schwering

* Data Error. The AutoTest data couldn't be interpreted correctly. This indicates either a corrupted protocol file or an incompatibility of AutoCheck and AutoTest versions.

* Write. This is a user-defined output. It enables the user to mark data that should be checked manually, such as user input at a pause statement.

The user can choose between four different warning levels for the syntax and evaluation warnings and can switch individual warnings on or off.

The output generated by AutoCheck has the following format:

ErrorType: statementline, dateline [is greater than]

descriptive_text

Thus, both the line containing the AutoCheck statement and the line containing the data are indicated.

If the output is written into a file, each line is preceded by the filename(statementline). This is the same format as used by many compilers, and therefore the built-in macros of many programming editors can be used in combination with AutoCheck. This means that errors can be reviewed in much the same way that a source file is debugged after compilation using an editor pointing to the source code errors.

At the end of the evaluation, AutoCheck gives the test engineer a quick overview of the result by providing a table showing how many output messages of each type have been generated. Whereas the evaluation errors indicate bugs either in the product or in the test script, the other output messages indicate potential problems in the test execution or evaluation process.

The AutoCheck output documents both that the evaluation has been carried out and the result of the evaluation, which for medical products are important for regulatory approvals and audits.

Platforms

AutoCheck and Autotest run on different platforms. AutoTest runs on a DOS[R]-based PC, which is very appropriate as a test controller because of the inexpensive hardware, an operating system that doesn't require dealing with tasking conflicts, and the availability of interface cards (the interface to the medical network is available as a PC card only). AutoCheck runs on a UNIX-based workstation because of the availability of lex/yacc and the greater resources (memory and processing power). However, both tools work on the same file system (a UNIX-based LAN server). The user doesn't have to worry about the different file formats, because AutoCheck automatically takes care of the format conversions. It accepts both DOS and UNIX formats and generates the output according to the detected protocol file format. Having different machines for execution and evaluation has also not proved to be a disadvantage for the test engineer.

Expandability

The basic architecture of AutoCheck has proven to be flexible for enhancements over time. Since the first release of AutoCheck we have implemented many enhancements because of new product features and because AutoTest provides additional data structures.

Validation

The risk of the AutoCheck approach is that, if AutoCheck overlooks an error (false negative output), the tester won't find the error. An automatic evaluation tool is only useful if the tester can rely on it, since otherwise, even if no errors were reported, the tester would still have to look at the protocol file. Therefore, the validation of an automatic evaluation tool is crucial to the success of such a tool. For this reason a thorough test of the tool was designed and every new revision is regression tested. Changes and enhancements undergo a formal process similar to that used for customer products.