Analysis and design of multitrait-multirater performance appraisal studies

Journal of Management, Spring, 1996 by James M. Conway

A good deal of research has addressed the analysis of multitrait-multimethod (MTMM) data. In a recent comprehensive review, Becker and Cote (1994) compared the performance of three models: the general confirmatory factor analysis model (CFA), the correlated uniqueness model (CU), and the direct product model (DP). It was found that the CU model produced the best results. However, Becker and Cote also found that all three models sometimes produced poor solutions (improper parameter estimates and/or poor fit to the data). One purpose of the present study was to extend these results by applying the same three models to multitrait-multirater (MTMR) performance appraisal matrices (Becker and Cote's study included no such matrices). A second purpose was to go beyond previous work by linking analysis to substantive research design characteristics. In other words, this study investigated how researchers can design MTMR studies to produce high proportions of trait variance, low proportions of method variance, and to maximize the likelihood of good solutions.

CFA, CU and DP Models

Since Becker and Cote (1994) provided an extensive discussion of the three models and related evidence, brief (non-mathematical) descriptions are provided here.

The CFA model. The CFA model assumes that each measured variable is an additive function of a trait component, a method component, and a unique (error) component (Schmitt & Stults, 1986; Widaman, 1985). The CFA model therefore includes a set of correlated trait factors and a set of correlated method factors as well as a vector of unique factors. Correlations between trait and method factors are generally constrained to zero for identification purposes (Marsh, 1989). This model assumes unidimensional method factors: methods add systematic variance to each variable, so that correlations between variables measured by the same method are all inflated. An example would be a generalized halo effect, in which a worker is seen as generally good or generally poor, and would be rated similarly on all dimensions.

While the CFA model apparently provides a wealth of information on the validity of measures, recently a number of authors have documented serious problems (e.g., Brannick & Spector, 1990; Kenny & Kashy, 1992; Marsh, 1989). One example is the failure of a solution to converge (i.e., a final solution is not reached). This might occur when the model contains too many parameters, which is analogous to attempting regression with more predictors than cases. Another problem involves improper estimates (outside the permissible range, such as a negative unique variance). Further, evidence was provided that when the CFA model does converge to an appropriate solution (all estimates within permissible range), the "method" factors may really represent (to some degree) trait variance (Marsh, 1989).

The CU model. Building on work by Kenny (1976; 1979), Marsh (1989) proposed the CU model as an alternative less likely than the CFA model to suffer from ill-defined solutions. The CU model assumes additive trait and method effects but there are no method factors per se. Instead, uniquenesses of observed variables measured with the same method are allowed to be correlated. Important differences between the CU and CFA models are that the CU model assumes methods are uncorrelated while the CFA model does not, and that the CU model makes no assumptions about dimensionality of method variance while the CFA model assumes unidimensionality (i.e., the CU model can accommodate multidimensional method variance; Marsh, 1989). An example of multidimensional method variance could involve Borman and Motowidlo's (1993) distinction between task performance (core job responsibilities) and contextual performance (volunteering for extra work; helping coworkers). A rater might tend to see workers in terms of these two separate aspects of performance, rather than as uniformly good or poor (as with a halo effect). The result would be inflated correlations among task dimensions due to a task method factor, and among contextual dimensions due to a contextual method factor, but not between task and contextual dimensions.

A number of studies have evaluated the CU model and in general the model has performed quite well (e.g., Kenny & Kashy, 1992; Marsh, 1989; Marsh & Bailey, 1991). The present study, in addition to evaluating the performance of the CU model, also investigated whether method variance was multidimensional using Marsh's (1989) procedures (described later).

The DP model. A second promising alternative to the CFA model is the DP model proposed by Browne (1984; Wothke & Browne, 1990). The DP model is based on work by Campbell and O'Connell (1967; 1982), who suggested that trait and method components may combine multiplicatively. Like the CFA model, the DP model includes correlated trait effects, correlated method effects, and unique effects. However, the variance in individual measures is assumed to be a multiplicative function of traits and methods, so trait and method effects cannot be separated (i.e., there are no separate trait and method factor loadings as in the CFA model). According to the DP model, methods affect observed correlations in proportion to the size of the true correlations between traits. An example is differential attenuation, in which it is assumed that sharing a common method actually provides a better estimate of the true correlation (typically it is assumed that sharing a method inflates the correlation). According to this view true correlations are attenuated by using different methods. For example, different raters may have different conceptual definitions of dimensions being rated. These differences will attenuate high true correlations but not low correlations (since low correlations cannot be reduced very far; Campbell & O'Connell, 1982).