Commentary: Evaluating electronic consumer health material

British Medical Journal, Jan 15, 2000 by Jeremy Wyatt

Consumer health material consists of specific content presented in a variety of formats and is of increasing importance to health services.[1] Electronic material--including computer programs, web resources, and conventional or interactive video-allows a much wider range of formats for presenting content to consumers. Changing the format while retaining the same content may alter decisions,[2] so such electronic systems need to be evaluated.

The best research design depends on the question. If we are interested in format then a randomised trial is needed in which controls receive the same content presented in the usual format--paper or verbal. If the research question concerns the impact of improved content, controls should receive the same format but with normal content. To answer a pragmatic question (Which material is better?) controls should receive the best paper leaflet, but no inference can then be drawn about the relative contributions of improved content or electronic format.

When information systems are evaluated it is often necessary to take a broader view. For example, in this trial, control patients could have been influenced by the touch screen system if they borrowed a password, looked over the shoulder of a woman using the system, or chatted to women in the intervention group in the antenatal clinic or class. The solution to such "contamination" is a cluster randomised trial[3]--randomising clinics, health centres, or districts rather than patients. This also avoids the need for passwords and unreliable randomisation with sealed envelopes.[4]

In this trial, baseline knowledge was high (47% of participants had received higher education) so only minor improvements could ever be shown--a ceiling effect.[5] Evaluators should seek out a group who are not so well informed but are able to use the novel information system. Although it may seem necessary to balance study groups for baseline knowledge, cluster designs make this difficult and large studies make it unnecessary.[6]

In this trial an impressive 91% of women in the intervention group used the touch screen system, but in most other trials this figure will be lower. Investigators should avoid comparing outcomes in those who did use the system with those who did not. As in this trial, analysis should be by "intention to provide information" However, only 70% of participants were followed up, leaving 148 who used the touch screen but whose knowledge or attitudes may conceivably have worsened. Investigators should vigorously pursue all participants and aim for minimum follow up of 80% by keeping questionnaires short and making only essential measures.

The benefits of the touch screen system in this trial may have been underestimated by contamination and high baseline levels of knowledge. Graham and colleagues rightly state that: "Like all new technologies, these devices should be subject to rigorous evaluation." With limited evidence of benefit for these expensive tools over well designed leaflets, they seem to fit best into the National Institute for Clinical Excellence (NICE) category C: for NHS use only in the context of rigorous research studies.[7]

[1] Smith R. The future of healthcare systems: information technology and consumerism will transform health care world wide. BMJ 1997;314:1495-6.

[2] Wyatt JC. Same information, different decisions: format counts. BMJ 1999;318:1501-2.

[3] Altman DG, Bland JM. Units of analysis. BMJ 1998;314:1874.

[4] Schultz KF, Chalmers I, Hayes RJ, Altman DG. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408-12.

[5] Streiner DL, Norman GR. Health measurement scales. Oxford: Oxford University Press, 1995:80.

[6] Peto R, Collins R, Gray R. Large-scale randomised evidence: large, simple trials and overviews of trials. Ann N Y Acad Sci 1993;703:314-40.

[7] Rawlins M. In pursuit of quality: the National Institute for Clinical Excellence. Lancet 1999;353:1079-82.

Schools of Public Policy, University College London, London WC1E 7HN

Jeremy Wyatt director, Knowledge Management Center

jeremy.wyatt@ucl ac.uk