Evaluation of health interventions at area and organisation level

British Medical Journal, August 7, 1999 by Obioha C Ukoumunne, Martin C Gulliford, Susan Chinn, Jonathan A C Sterne, Peter G J Burney, Allan Donner

Findings

We identified 10 key considerations for evaluating organisation level interventions.

(1) Recognise the cluster as the unit of intervention or allocation

Healthcare evaluations often fail to recognise, or use correctly, the different levels of intervention which may be used for allocation and analysis.[5] Failure to distinguish individual level from cluster level intervention or analysis can result in studies that are inappropriately designed or give incorrect results.[3]

(2) Justify the use of the cluster as the unit of intervention or allocation

For a fixed number of participants, studies in which clusters are randomised to groups are not as powerful as traditional clinical trials in which individual patients are randomised.[2] The decision to allocate at organisation level should therefore be justified on theoretical, practical, or economic grounds (box 1).

(3) Include enough clusters

Evaluation of an intervention that is implemented in a single cluster will not usually give generalisable results. For example, a study evaluating a new way of organising care at one diabetic clinic would be an audit study which may not be generalisable. It would be better to compare control and intervention clinics, but studies with only one clinic per group would be of little value, since the effect of intervention is completely confounded with other differences between the two clinics. Studies with only a few (fewer than four) clusters per group should generally be avoided as the sample size will be too small to allow a valid statistical analysis with appreciable chance of detecting an intervention effect. Studies with as few as six clusters per group have been used to show effects from cluster based interventions,[6] but larger numbers of clusters will often be needed, particularly when relevant intervention effects are small.

(4) Randomise dusters wherever possible

Random allocation has not been used as often as it should in the evaluation of interventions at the level of area or organisation. Randomisation should be used to avoid bias in the estimate of intervention effect as a result of confounding with known or unknown factors. Sometimes the investigator will not be able to control the assignment of clusters--for instance, when evaluating an existing service,[7] but because of the risk of bias, randomised designs should always be preferred. If randomisation is not feasible, then the chosen study design should allow for potential sources of bias.[8] Non-randomised studies should include intervention and control groups with observations made before and after the intervention. If only a single group can be studied, observations should be made on several occasions both before and after the intervention.[8]

(5) Allow for clustering when estimating the required sample size

When observations made at the individual level are used to evaluate interventions at the cluster level, standard formulas for sample size will not be appropriate for obtaining the total number of participants required. This is because they assume that the responses of individuals within clusters are independent (box 2).[2, 9-11] Standard sample size formulas underestimate the number of participants required because they allow for variation within clusters but not between clusters.