A high-level programming language for testing complex safety-critical systems

Hewlett-Packard Journal, June, 1997 by Andreas Pirrung

Dealing with an enormous amount of data is characteristic of validating complex and safety-critical software systems. ATP, a high-level programming language, supports the validation process. In a patient monitor test environment it has shown its usefulness and power by enabling a dramatic increase in productivity. Its universal character allows it to migrate validation scenarios to different products based on other architectural paradigms.

This article concentrates on the specific problem of transforming a test design into concrete automatic test procedures. For a systematic overview and context the reader is referred to the article on page 89. As described in that article, the test design identifies and documents the test set for a given product. It is derived from external and internal specifications, software quality engineer expertise, and risk and hazard analysis results. A test design is normally informal and describes test cases and test data on a high, abstract level, independent of the test environment. On the other hand, an automatic test procedure has to deal with all the details of the test environment and reflects the abstraction capabilities of the existing tools.

In our software quality engineering department the automatic test environment is based upon two major tools: Auto Test[1] and AutoCheck. The first is a test execution tool and the latter is responsible for test evaluation (see article, page 103). AutoTest is very close to the devices it controls and requires detailed commands on a low abstraction level. AutoCheck has to cope with the detailed low-level information produced by AutoTest and therefore also requires input on a detailed, low abstraction level (see Fig. 1). The strengths of the low-level interfaces are their flexibility and adaptability to various different test situations.

[Figure 1 ILLUSTRATION OMITTED]

There are some difficulties with the process shown in Fig. 1. The test engineer spends a lot of time transforming test designs into automatic test procedures. There is a large gap in abstraction level between the test design and the test procedure. The detail level is low in the test design, but very high in the test procedure. It is an error-prone, time-consuming task to bridge this gap manually. Because resources are always restricted, the software quality engineer has less time for a more intensive test design.

Because the test procedures have a high explicit redundancy, it is difficult to maintain and evolve test procedures. The explicit redundancy is high because AutoTest and AutoCheck do not support data and functional abstraction, nor do they offer control flow elements. A piece of code may exist in many copies scattered over the test procedures. If the test requires a change in the code pattern, for instance because of changes in the timing behavior of the system under test (in our case a patient monitor), the test engineer has to update numerous copies of this code pattern. The risk of forgetting one pattern or introducing an error in a test procedure increases with the number of update steps. It is very resource-consuming to adapt test procedures to a change in system behavior.

The test procedure describes a static test scenario. Therefore, the test engineer has to document the test setup completely. Every parameter that influences the test environment and consequently the test execution must be carefully controlled before starting the automatic test. Our test environment consists of so many simulators, forcing devices, and sensing devices that sometimes tests need to be repeated because the initial conditions are wrong. The problem is that the automatic test procedure describes only one specific test situation. It is not possible to use parameters for the test procedure and to feed in the actual start parameters at the beginning of the test execution to get more general and robust test procedures. Even a slight change in the start condition may require an adapted or nearly new test procedure.

The test coverage is limited because the test data is coded within the test procedure. The repetition of a test case with other test data requires a modified duplicate of the test procedure. Again, it would help if a test case were able to profit from data abstraction and parameters, enabling the test engineer to formulate more general test procedures.

AutoTest and AutoCheck do not support the statistical structural testing approach. It is therefore not possible to select test data randomly (see "Structural Testing, Random Testing, and Statistical Structural Testing" on page 97).

The following section illustrates the above problems by presenting a practical example to demonstrate the transformation of a high-level test design to an automatic test procedure.

A Practical Example

Patient monitors are electronic medical devices used to monitor physiological parameters of critically ill patients in intensive care units or operating rooms. They alert the medical staff when physiological parameters exceed preconfigured limits. In this example, we will concentrate on a well known physiological parameter, the heart rate. The nurses and doctors want to get an immediate alarm when the heart rate falls below a given lower limit or exceeds a given higher limit. A malfunction of the monitor may result in the death of a patient, so this functionality is safety-critical and must be validated very carefully by the vendor of the patient monitor. The example illustrates a test design for heart rate alarm testing and the transformation process to the appropriate automatic test procedure.