Inter-rater reliability and validity of the Action Research arm test in stroke patients

Age and Ageing, March, 1998 by Ching-Lin Hsieh, I-Ping Hsueh, Fu-Mei Chiang, Po-Hsin Lin

Introduction

The most elegant design of a clinical study will not overcome the damage caused by unreliable or imprecise measurement [1]. The evidence of reliability and validity is frequently confined to specific diagnoses, service settings and administration [2]. Evidence on a specific measure may be extremely variable. Johnston et al. [3] have acknowledged that more formal studies of the reliability and validity of the measure and assessment procedures are desirable.

The Action Research arm test (ARAT) developed by Lyle [4] was based on the upper extremity function test of Carroll [5, 6]. The ARAT was constructed for assessing recovery of upper extremity function (focal disability) following cortical injury. The ARAT is designed for evaluation of both sides of patients, in order to obtain a more total description of the upper extremity function. The ARAT contains tour subscales--`grasp', `grip', `pinch' and `gross movement'--comprising 19 items in total. Each subscale fulfilled the statistical criteria for Guttman scales and so is constructed of items arranged in hierarchical order of difficulty. Items within each subscale are ordered in such a way that if a patient accomplishes the most difficult item, this predicts success with all less difficult subscale items. Thus, the patient is credited with succeeding with all items of the subtest for that limb. on the other hand, failure with the easiest item predicts failure with all items of greater difficulty on that subscale. Thus, the ARAT has been specially constructed to save testing time. It takes no more than 10 min to examine a stroke patient on the ARAT [7].

The psychometric characteristics of the ARAT have been rarely explored. Lyle [4] has identified four subscales, each fulfilling the necessary statistical criteria for reproducibility and scalability. Lyle [4] ascertained inter-rater and test-retest reliablities to be as high as r = 0.99 and 0.98 respectively, calculated by Pearson product-moment correlation. However, generalization of the results requires caution. First, the subjects of Lyle's study were chronic hemiplegic patients (mean duration 4 years) but not patients at the acute or subacute stage commonly seen at clinics or in research. Second, the hemiplegia was secondary to stroke, traffic or industrial accident or assault. The broad scope of the subject's condition may obscure some specific characteristics of a particular disease. Third, the statistical method used to examine reliability may be inappropriate. Correlation coefficient, a measure of association not of agreement [8, 9], often overestimates the degree of true agreement and may yield misleading information about reliability [10].

DeWeerdt and Harrison [7] compared the motor recovery of upper extremity, by the administration of the Fugl-Meyer assessment and the ARAT to 53 stroke patients at 2 and 8 weeks after onset. They found that both tests appear to monitor the upper limb function equally well. They preferred the ARAT because it is probably more meaningful to the patient and takes less time to administer.

The psychometric details of the ARAT have not been well established. The primary purpose of this study was to determine the extent to which different examiners agreed on the performance of stroke patients using the ARAT. The relationships between the scores on the ARAT and some other well-validated measurements evaluating upper extremity motor impairment and disability were also investigated.

Methods

Subjects

Of the 108 consecutive stroke patients admitted to the Physical Medicine and Rehabilitation department at National Taiwan University Hospital in Taipei between April and August 1996, 50 met the following criteria: (i) diagnosis according to International Classification of Diseases, Ninth Revision Clinical Modification (ICD-9-CM) codes for subarachnoid haemorrhage (430), cerebral haemorrhage (431 and 432), cerebral infarction (433 and 434) or other (436 and 437) and (ii) ability to follow verbal commands.

The clinical diagnosis of stroke was confirmed by physicians using neuro-imaging examination (computed tomography or magnetic resonance imaging). Patients who were diagnosed with transient ischaemic attack (ICD-9-CM code 435) or with late effects of cerebrovascular disease (ICD-9-CM code 438) were excluded. Most of the patients excluded were those who could not follow commands (e.g. patients with global aphasia). Before the study, all subjects gave their informed consent to be included in the study. Further information on the study sample is presented in Table 1.

Table 1. Characteristics of the study patients (n = 50)

Characteristic                    Value

Gender
 Male                             30
 Female                           20
Mean age, years (SD)              65 (13.0)
Median days after onset (range)   55 (8-535)
Frequency of diagnoses
 Subarachnoid haemorrhage          7
 Cerebral haemorrhage             13
 Cerebral infarction              21
 Other                             9
Frequency of paresis
 Right                            22
 Left                             23
 Bilateral                         5
Mean test score (SD)
 ARAT(a)                          83.7 (24.6)
 UEMAS                             7.7 (6.9)
 AMI                              46.2 (31.9)
 UEMMAC                           83.9 (21.6)