Assessing construct validity in organizational research - includes appendices

Administrative Science Quarterly, Sept, 1991 by Richard P. Bagozzi, Youjae Yi, Lynn W. Phillips

Classic and contemporary methods for analyzing construct validity are compared and contrasted through reanalyses of data from the organizational research literature to establish a basis for assessing the validity of measures used in organizational research. Campbell and Fiske's (1959) criteria are found to be lacking, particularly in their assumptions, diagnostic information, and power. Confirmatory factor analysis (CFA) is shown to overcome most limitations inherent in Campbell and Fiske's procedures. Nevertheless, two potential shortcomings are identified with the CFA method: the confounding of random error with measure-specific variance and the inability to test for interactions between traits and methods. Three alternative methods are presented for addressing the former issue, and the direct product model is described as a solution to the latter. The techniques considered herein go farther than currently used procedures for enhancing our ability to ascertain the validity of variables commonly studied in organizational research.(*)

INTRODUCTION

Any measure often reflects not only a theoretical concept of interest but also measurement error. Measurement error, commonly recognized as a serious problem throughout the social sciences (e.g., Fiske, 1982), can be partitioned into random error and systematic error, such as method variance. Method variance refers to variance attributable to the measurement method rather than to the construct of interest, and examples include archival biases, key-informant prejudices or limitations, halo effects, social desirability, and acquiescence. Each of the two error components can have serious confounding influences on empirical research and yield misleading conclusions (Campbell and Fiske, 1959). Random error tends to attenuate the observed relationships among variables in statistical analyses and may induce errors in inference. Under some circumstances, random error even inflates parameter estimates (Bagozzi, 1991). Method variance may also bias results by inflating the observed relationships among variables measured with the common method.

Because measurement errors (i.e., random error and method variance) provide potential threats to the validity of research findings, it is important to validate measures and disentangle the distorting influences of these errors before testing theory. This can be achieved by using multiple measures and multiple methods in measurement (Campbell and Fiske, 1959). Using a single measure does not permit one to take measurement error into account in analyses. Similarly, with a single method one cannot distinguish substantive (i.e., trait) variance from unwanted method variance, because each attempt to measure a concept is contaminated by irrelevant aspects of the method employed.

Construct validity, which is defined broadly as the extent to which an operationalization measures the concept it is supposed to measure (e.g., Cook and Campbell, 1979), has been single-out as a central issue in organizational research (e.g., Webb and Weick, 1979; Schwab, 1980; Mitchell, 1985). Given multiple measures obtained with multiple methods, construct validation can be done with the multitrait-multimethod (MTMM) matrix, the correlation matrix for different concepts (traits) when each of the concepts is measured by different methods (Campbell and Fiske, 1959). Without assessing construct validity one cannot estimate and correct for the confounding influences of random error and method variance, and the results of theory testing may be ambiguous. That is, a hypothesis might be rejected or accepted because of excessive error in measurement, not necessarily because of the inadequacy or adequacy of theory.

Most attention in this regard has focused on method variance, where it is often claimed that such effects are "pervasive, ubiquitous" (Fiske, 1982: 82). Considerable debate can be found on both sides of the issue as to the extent of method variance in practice. Spector (1987) reanalyzed 11 data sets dealing with job satisfaction and concluded that little evidence exists for method variance. HE found that in 10 of 11 data sets "method variance was nonsignificant and of extremely small magnitude" (Spector, 1987: 440). In contrast, Williams, Cote, and Buckley (1989) reexamined the same data but concluded that significant method variance was present in 9 of 11 of the data sets, accounting for 25 percent of the variance of measures, on average.

How is it possible that researchers can reach opposite conclusions when examining the same issue on the same data? One problem is that the aforementioned researchers used different procedures to examine construct validity. Spector (1987) relied on the classic criteria advocated by Campbell and Fiske (1959), whereas Williams, Cote, and Buckley (1989) used confirmatory factor analysis. These procedures are based on different sets of assumptions and can lead to different conclusions.

Perhaps it should not be surprising that different conclusions can arise in the assessment of construct validity, since no clear standards can be found in the literature for analyzing and interpreting MTMM matrices. Although at least ten different procedures have been proposed for the analysis of MTMM matrices, each is built on a different set of assumptions and each is appropriate only under certain circumstances. (1) Unfortunately, to date, little guidance exists on when and how to select among these procedures. Individual articles in both the psychometric and applied literatures generally focus on only one procedure and give passing reference to alternatives without fully considering the standards and trade-offs of each. Moreover, little integrative critical commentary exists on the approaches. As a consequence, we lack a coherent approach to construct validity, and the body of knowledge reflected in empirical work is rather piecemeal and inconclusive.