Efforts to establish the reliability of the Resident Assessment Instrument - Continuing and Rehabilitative Care for Elderly People: A Comparison of Countries and Settings

Age and Ageing, Nov, 1997 by Antonio Sgadari, John N. Morris, Brant E. Fries, Gunnar Ljunggren, Palmi V. Jonsson, Jean-Noel DuPaquier, Marianne Schroll

Introduction

There is general agreement that developing reliable data on the functional and mental status of nursing home residents is a task that challenges traditional approaches to measurement. A large proportion of nursing home residents have some level of cognitive impairment and/or communication difficulties. Furthermore, their abilities and their status vary during the day and across the week [1].

The RAI was designed to produce reliable data on even the most abstract concepts, using mutually exclusive response categories, standardized definitions, directions for inquiring about resident status over a relevant period of time and multiple sources of information [2]. In its development, three different field tests were conducted which, after dropping or revising more than half of the original items, led to the final version of the RAI in 1991 [3, 4].

The success of the RAI in the USA made it attractive to the international community of investigators in geriatrics and gerontology and it started being adopted in European countries and in Japan. National versions were translated into several languages: Czech, Danish, Dutch, Finnish, French, German, Italian, Icelandic, Japanese, Norwegian, Spanish and Swedish. Soon it became clear that RAI could represent a sort of `Esperanto'--a common language that could enable researchers from different countries and cultures to share data, compare populations and carry out cross-national studies on elderly populations.

The success of this international comparative research relies heavily on the quality of the data that are collected. As a preliminary step, in most countries an independent `back-translation' to English was accomplished to assure that the translation process had not caused major changes in the wording, phrasing and meaning of the original RAI items. Then, assessment staff were trained to perceive the need for accurate, reliable data, to understand the roles of the different participants in the data collection process and to complete the RAI form accurately and efficiently. Eventually, in several countries studies were carried out to verify that the national version of the RAI led to uniform and reliable assessment.

The aim of the present paper is to describe and accumulate evidence from the efforts that have been made to test the inter-rater reliability of RAI items in Denmark, Iceland, Italy, Japan, Sweden, Switzerland and the USA.

Methods

The results presented in this paper are based on dual assessments completed by licensed nursing personnel. All nursing assessors were trained by project investigators. In each facility, two nurses were assigned a common group of residents to assess using the RAI. The nurses independently performed their assessments and were prohibited from discussing their residents or their findings before the conclusion of the study.

An overview of the procedures used in the reliability studies in seven countries are shown in Table 1. In European countries and Japan the residents were randomly selected from a roster. In the USA, since the final RAI reliability testing also included the validity testing of the resident assessment protocols, residents were stratified according to the presence or absence of a specific problem under review before being selected.

Table 1. Characteristics of reliability testing studies

                          Denmark   Iceland   Italy    Japan

Sample selection          Random    Random    Random   Random
No. of facilities          8         3         2        15
No. of assessments        74        24        82       129
Time frame (days apart)    9- 12     7         7        14

                          Sweden   Switzerland   USA

Sample selection          Random   Random        SR
No. of facilities          3        19            13
No. of assessments        46        87           123
Time frame (days apart)    1-2      Same day       7

SR, stratified random.

The largest number of assessments was performed in Japan (n = 129), the lowest in Iceland (n = 24). In Switzerland and Sweden the two nurses assessed the same resident within 2 days, while in other countries the time frame was, on average, 7 days. In Japan the assessments were carried out over 2 weeks.

Analytic approach

In each country, the reliability of the RAI items was assessed by comparing RAIs completed by two trained nurses who independently evaluated the same residents. The reliability testing differed somewhat since the interviewers assessed the resident at the same time and concurrently accessed other sources of information. Both assessors employed the recommended information gathering procedures and the same RAI items, response categories and definitions.

Reliability estimates were based on the congruence between judgements of the two assessors using the weighted [Kappa] [5, 6] and the Spearman-Brown intraclass correlation coefficient [7]. These measures generally provide a more conservative estimate of reliability than either simple correlation or percent agreement because they adjust for chance agreement between the two assessors. It can be demonstrated that the intraclass correlation coefficient corresponds to [Kappa] when dichotomous variables are considered and to weighted [Kappa] when applied to categorical variables [8].