Assessing construct validity in organizational research - includes appendices

Administrative Science Quarterly, Sept, 1991 by Richard P. Bagozzi, Youjae Yi, Lynn W. Phillips

Another problem with Campbell and Fiske's procedure is that no precise standards are provided for ascertaining how well the criteria are met, as mentioned earlier. Instead, only rules of thumb are offered that depend on a qualitative assessment of the number of confirming and disconfirming incidents of the comparisons of correlations implied by the criteria in the MTMM. By focusing on the number of times selected correlations are greater than certain others, Campbell and Fiske's procedure neglects the importance of the differences in magnitudes between pairs of correlations. The magnitudes are functions of the degree of convergent and discriminant validity.

Still another shortcoming of Campbell and Fiske's procedure is that no information is provided as to the separate amounts of variation in measures due to traits, methods, and random error. Rather, these are all confounded in the raw correlations among measures.

Each of the aforementioned drawbacks with Campbell and Fiske's procedure limits its usefulness but can be ameliorated. We turn now to the first of what might be termed second-generation methods for approaching construct validity. The following analyses of the data sets shown in Table 1 illustrate several possible outcomes with respect to making inferences about construct validity. For example, in one case in which Campbell and Fiske's procedure suggests that construct validity must be rejected, we show that construct validity is in fact achieved. For other cases in which achievement of construct validity is conducted on the basis of Campbell and Fiske's procedure, we show that, although the conclusions about construct validity are sustained, the interpretation of method and trait effects is incorrect. Thus, an application of Campbell and Fiske's procedure can result in Type I and Type II errors, a point not acknowledged in the literature.

CONFIRMATORY FACTOR ANSLYSIS

The CFA model is a powerful method for addressing construct validity and makes fewer assumptions and provides more diagnostic information about reliability and validity than Campbell and Fiske's criteria. Unlike Campbell and Fiske's (1959) procedure, the CFA model allows methods to affect measures of traits in different degrees and to correlate freely among themselves. In addition, the CFA model provides the following advantages: (1) measures of the overall degree of fit are provided in any particular application (e.g., the chi-square goodness-of-fit test), (2) useful information is supplied as to if and how well convergent and discriminant validity are achieved (i.e., through chi-square difference tests, the size of factor loadings for traits, and the estimates for trait correlations), and (3) explicit results are available for partitioning variance into trait, method, and error components (i.e., through squared factor loadings and error variance).

An intuitive description of the CFA model can be seen by reference to Figure 1, where we have taken the wholesale-distribution study by Bagozzi and Phillips (1982) as an illustration. The four hypothesized functional uses of computers--in training, forecasting, accounting, and financial analysis--and the two key-informant methods--the CEO and subordinate(s)--are drawn as ovals. These correspond to factors in factor analysis. Notice that each functional use of computers is connected to two boxes with arrows. The boxes represent the actual key-informant reports, of which a total of eight measurements 9drawn as ys) result for the four uses of computers as provided by two informants in each organization. Following the convention in the literature, the four uses of computers are termed traits and the two informant reports are termed methods.