Analyzing multiple informant data from an evaluation of the health disparities collaboratives

Health Services Research, Feb, 2007 by A. James O'Malley, Bruce E. Landon, Edward Guadagnoli

Health services researchers often collect similar or even identical data from multiple sources (e.g., physicians, nurses, patients, teachers, parents) in order to increase the reliability of the measurements or to gain insights from several different perspectives and contexts. When multiple informants measure an independent variable, researchers are typically interested in how the variable or construct measured by each informant affects the outcome. When multiple informants assess an outcome variable, the focal point is often the overall effect of the independent variables on the outcome. In either case, more precise estimates are possible if effects can be pooled across informants.

Traditional approaches for analyzing data from multiple informants include conducting independent analyses for each informant, selecting one informant as the informant of interest and ignoring others (i.e., only using the data obtained from the chosen informant for the analysis), and reducing the data from the informants into an overall measure (e.g., the mean score across informants). However, these methods have several disadvantages. The effects of individual informants are often ignored, differences between or among informants are not testable, and the degree of correlation between informants is not estimated (Horton and Fitzmaurice 2004). In addition, summary measures (e.g., means) are not consistently defined when one or more informants' scores are missing, and will result in a loss of sample size and hence suboptimal analysis if only those units with a full set of responses are included.

The exclusion of data from some informants, or the reduction of data from multiple informants to a summary statistic, is one of the ways in which data are edited in practice. In general, investigators edit data in ad hoc, idiosyncratic ways and often do not disclose their unique strategies (Leahey, Entwisle, and Einaudi 2003; Leahey 2004). Because data-editing decisions can profoundly affect conclusions (Gould 1981; Dewald, Thursby, and Andersen 1986), it is important that these decisions are documented, or better still are avoided (Leahey 2004). For example, if the relationship between two variables varies in direction across informants the relationship between the means of the informants' ratings could be misleading. It is better to utilize multiple (even multiple discordant) views rather than reducing the data to a convenient form for analysis.

Individual responses to surveys can be influenced by several sources of variation including characteristics of individuals, which can affect an individual's interpretation of the construct being measured such as their training and experience, or position in the organization. For example, in sociology the social status of an informant has been shown to have a bearing on the response obtained (Leahey, Entwisle, and Einaudi 2003; Leahey 2004). These characteristics of the informant and context need to be accounted for in order to pool information sensibly across informants, to determine the best way of measuring a construct, and ultimately to make general inferences about the relationship of the variable being measured with other variables.

Multivariate regression models have recently been used to analyze data from multiple informants (Horton, Laird, and Zahner 1999; Goldwasser and Fitzmaurice 2001; Horton et al. 2001; Lash et al. 2003; Horton and Fitzmaurice 2004). The key characteristic of this approach is that individual data from all informants are modeled, enabling the evaluation of comparative inferences about informants (e.g., testing whether the informants have the same or different effects), and avoiding reducing or omitting the data for a unit of analysis because data from some informants were unavailable or missing.

Another model that might be considered for multiple informant data is the "method factor" model (Bollen and Paxton 1998). In the context of a method-factors model, the informant is the "method" of measurement. The analysis of a method-factors model is typically based on a structural equations model that treats the informants' ratings as subjective error-prone measurements of an unobserved trait. In contrast, the multivariate regression models referred to in the preceding paragraph allow for multiple underlying traits, and directly model informant specific effects as opposed to estimating the effect of a common underlying trait (although this could be accommodated within the multivariate regression framework). Because we want to model the effect of informant-specific effects on the outcome, rather than the effect of an underlying trait, herein we use the multivariate regression approach.

In this paper, we analyze data collected from surveys of personnel at community health centers (CHCs) that are part of an ongoing evaluation of a Bureau of Primary Health Care-sponsored quality improvement (QI) program (HRSA 2004). We compare standard approaches for analyzing multiple informant data, including picking one informant as representative of the whole and averaging over informants' ratings, with multivariate regression methods. We address the cases in which information on an independent or dependent variable of interest is available from multiple informants. Although multivariate regression models for analyses of multiple informants have already been established, the advantages of using this approach over standard approaches have not been demonstrated previously using real data. The results will be useful to health services researchers as multiple informant data become increasingly common.