An automated test evaluation tool

Hewlett-Packard Journal, June, 1997 by Jorg Schwering

Architecture

We first conducted a feasibility study, which investigated different architectural approaches and implementation tools. The first approach was in the area of artificial intelligence, namely expert systems and language recognition (this would be expected for an investigation started in 1991). It soon became apparent that protocol file evaluation is basically a compiler problem. The languages and tools investigated were Prolog/Lisp, sed/UNIX[R] shell, lex/yacc, C, and a C-style macro language for a programmable editor. We came to the conclusion that a combination of lex/yacc and C would lead to the easiest and most flexible solution.

Fig. 4 shows the AutoCheck architecture. The protocol file is first run through a preprocessor, which removes all lines irrelevant to AutoCheck, identifies the different AutoTest interfaces, and performs the local language translations. Thereafter it is analyzed by a combination of a scanner and a parser. We implemented specialized scanner/parsers for the AutoCheck metalanguage and the data provided by the different patient monitor interfaces. The AutoCheck statements and the AutoTest data are written into separate data structures. A third data structure holds some control parameters such as the accepted tolerances (see "AutoCheck Features and Syntax" below). After each data package, which is the answer to one AutoTest data request command, the compare function is started. The compare function writes all deviations into the error file.

[Figure 4 ILLUSTRATION OMITTED]

Basically, AutoTest and AutoCheck recognize two types of data requests: single tunes, which respond with exactly one data set for each requested message, and continuous tunes, which gather data over a defined time interval.

In the monitor family under test all important data has an update frequency of 1024 ms. AutoTest groups all data messages received within a 1024-ms frame into one data block and AutoCheck treats each data block of a continuous tune like a data package of a single tune.

All AutoCheck statements are then verified against each data block. The AutoCheck statements remain valid and are applied to the next data block until they are explicitly cleared or overwritten by a new AutoCheck block.

AutoCheck Features and Syntax

The AutoCheck features and syntax are best described using the example shown in Fig. 5. The numbers below correspond to the numbers in the left column of Fig. 5.

Fig. 5. An example of expected results written in the AutoCheck language. The number at the left refer to the paragraphs in the article that describe these statements.

(4)     ^   Tolerance Definition
(4)     ^      "Temp 1" : 1%;
(4)     ^   End Tolerance Definition

(1)     ^   Verify Begin
(2)(4)  ^     "Temp 1" - > value = 37.0;
(2)     ^     "Temp 1" - > unit = C;
(3)     ^     not alarm for "Press1"
(3)     ^       within (5, NaN);
(5)     ^     alarm "HR" > al_max;
(6)(7)  ^     if Neonate
(6)     ^     then
(6)     ^       "HR" - > al_min =30;
(6)     ^     endif;
(6)(7)  ^     if value of "Pat.Size" is "Adult"
(6)     ^     then
(6)     ^      "HR" - > al_min = 15;
(6)     ^     endif;
(8)     ^     write "Check user input";
(1)     ^   Verify end

1. All AutoCheck statements are preceded by a caret (^) and are treated as comments by AutoTest. As mentioned above under "Architecture," AutoCheck statements are grouped into blocks. Each block is enclosed by the two statements Verify Begin and Verify End.

2. There is a set of AutoCheck statements that enables the user to verify all data that can be read by AutoTest (numerics, alerts, sound, wave data, task window texts, etc.). An example of a numerical value is temperature, including all of its accompanying attributes such as units ([degrees] C) and alarm limits. In this example the value of the temperature numeric is expected to be 37.0 [degrees] C.