Assessing construct validity in organizational research - includes appendices

Administrative Science Quarterly, Sept, 1991 by Richard P. Bagozzi, Youjae Yi, Lynn W. Phillips

[TABULAR DATA OMITTED]

tempered by the knowledge that the overall goodness-of-fit measures point to poor fits.

Table 2 also shows the results for the chi-square difference tests. The comparison of models 1 and 3 shows that the introduction of method factors significantly drops the chi-square value in each data set, indicating that meaningful improvements over the null models are achieved. Similarly, the comparison of models 2 and 4 shows that the introduction of method factors provides significant improvements over the trait-only models for all data sets, confirming the need to include method effects. Likewise, comparisons of models 1 and 2 and models 3 and 4 reveal that the introduction of trait factors leads to significant improvements in fit over the null and method-only models, respectively. Thus, although the magnitudes of the chi-square difference test suggest that traits explain more variance than methods in the four data sets, both traits and methods are needed in the final analyses.

Table 3 displays the individual parameter estimates and partitioning of variance for the data sets. All factor loadings for traits and methods are significant. However, one parameter in the model for Gillet and Schwab (1975)--i.e., the factor loading ([lambda] = 1.14) for the satisfaction with supervision measure on the Job Descriptive Index--showed an improper solution, in that a standardized factor loading should not exceed 1.00. Because improper solutions, which are defined as either illogical parameter estimates or those outside the range of conventional acceptability (e.g., a negative variance or correlation exceeding 1.00), often result from model misspecification (e.g., Van driel, 1978), they provide useful information about the adequacy of a model (see Bagozzi and Yi, 1990). This suggests that the CFA model with traits and methods should be rejected for the data from Gillett and Schwab (1975), despite a satisfactory overall goodness-of-fit measure for the model. As we show below when we examine the noncentralized normed fit index and the issue of practical relevance, improper solutions can result from overfitting method factors to the data.

The decomposition of variance due to trait, method, and error is done by inspecting squared factor loadings in [[Lambda].sub.[Tau]] and [[Lambda].sub.[Mu]] and unique variances in [phi], respectively (these are found in Table 3). For example, the amount of method variance can be assessed by computing the squared loadings for method factors. Performing these operations yields the following results for trait, method, and error variance. It is found that method effects vary widely within studies. Specifically, method variance ranges from 4 percent to 50 percent in Bagozzi and Phillips (1982), 13 percent to 46 percent in Phillips (1980), 4 percent to 48 percent in Gillet and Schwab (1975), and 3 percent to 100 percent in Dunham, Smith, and Blackburn (1977). Thus, the implicit assumption in Campbell and Fiske's (1959) procedure that all traits are equally influenced by method factors does not hold for the data sets considered here. In addition, the amount of method variance is found to be considerable in each of the four data sets. On average, the proportion of the variation in the measures explained by method variance is 20 percent in Bagozzi and Phillips (1982), 27 percent in Phillips (1980), 24 percent in Gillet and Schwab (1975), and 42 percent in Dunham, Smith, and Blackburn (1977). One can also use the individual estimates of method variance to identify problematic measures with excessive method bias. In Bagozzi and Phillips (1982), for example, the subordinate's judgment of the use of computers in forecasting appears to have considerable method bias (53 percent).