Evaluation of health interventions at area and organisation level

British Medical Journal, August 7, 1999 by Obioha C Ukoumunne, Martin C Gulliford, Susan Chinn, Jonathan A C Sterne, Peter G J Burney, Allan Donner

This is the second of our articles

Healthcare interventions are often implemented at the level of the organisation or geographical area rather than at the level of the individual patient or healthy subject. For example, screening programmes are delivered to residents of a particular area; health promotion interventions might be delivered to towns or schools; general practitioners deliver services to general practice populations; hospital specialists deliver health care to clinic populations. Interventions at area or organisation level are delivered to clusters of individuals.

The evaluation of interventions based in an area or organisation may require the allocation of clusters of individuals to different intervention groups (see box 1).[1 2] Cluster based evaluations present special problems both in design and analysis.[3] Often only a small number of organisational units of large size are available for study, and the investigator needs to consider the most effective way of designing a study with this constraint. Outcomes may be evaluated either at cluster level or at individual level (table).[4] Often cluster level interventions are aimed at modifying the outcomes of the individuals within clusters, and it will then be important to recognise that outcomes for individuals within the same organisation may tend to be more similar than for individuals in different organisational clusters (see box 2). This dependence between individuals in the same cluster has important implications for the design and analysis of organisation based studies.[2] This paper addresses these issues.

Comparison of levels of intervention and levels of evaluation (adapted from McKinlay[4])

                         Level of intervention

Level of evaluation      Individual

Individual               Clinical trial--for example, does
                         treating multiple sclerosis patients
                         with interferon beta reduce their
                         morbidity from the condition?

Area or organisation

Level of evaluation      Area or organisation

Individual               Area or organisation based evaluation--for
                         example, does providing GPs with guidelines
                         on diabetes management improve blood
                         glucose control in their patients? Does
                         providing a "baby friendly" environment
                         in hospital increase
                         mothers' success at breast feeding?
Area or organisation     Area or organisation based evaluation--for
                         example, do smoking control policies
                         increase the proportion of smoke free
                         workplaces? Do fundholding general
                         practices develop better practice
                         facilities than non-fundholders?

Box 1: Reasons for carrying out evaluations at duster level

* Public health and healthcare programmes are generally implemented at organisation rather than individual level, so cluster level studies are more appropriate for assessing the effectiveness of such programmes

* It may not be appropriate, or possible in practice, to randomise individuals to intervention groups since all individuals within a general practice or clinic may be treated in the same way

* "Contamination" may sometimes be minimised through allocation of appropriate organisational dusters to intervention and control groups. For example, individuals in an intervention group might communicate a health promotion message to control individuals in the same cluster. This might be minimised by randomising whole towns to different interventions

* Studies in which entire clusters are allocated to groups may sometimes be more cost effective than individual level allocation, if locating and randomising individuals is relatively costly

Box 2: Three reasons for correlation of individual responses within area or organisational clusters

* Healthy subjects or patients may have chosen the social unit to which they belong. For example, individuals may select their general practitioners on the basis of characteristics such as age, sex, or ethnic group. Individuals who choose the same social or organisational unit might be expected to have something in common

* Cluster level attributes may have a common influence over all individuals in that cluster, thus making them more similar. For example, outcomes of surgery may vary systematically between surgeons, so that outcomes for patients treated by one surgeon tend to be more similar to each other than to those of another surgeon

* Individuals may interact within the cluster, leading to similarities between individuals for some health related outcomes. This might occur, for example, when individuals within a community respond to health promotion messages communicated through news media

Nature of the evidence

We retrieved relevant literature using computer searches of the Medline, BIDS (Bath Information and Data Services), and ERIC (Education Resources Information Centre) databases and hand searches of relevant journals. The papers retrieved included theoretical statistical studies and studies that applied these methods. Much of the relevant work has been done on community intervention studies in coronary heart disease prevention. We retrieved the content of the papers, made qualitative judgments about the validity of different approaches, and synthesised the best evidence into methodological recommendations.