The impact of method effects on structural parameters in validation research

Journal of Management, Spring, 1995 by Neal Schmitt, Earl Nason, David J. Whitney, Elaine D. Pulakos

Predictor-Related Criterion Bias

Investigators have been concerned with the relevance of criteria since the beginnings of personnel research (Austin & Villanova, 1992). The usefulness of performance ratings, possibly the most common criterion in personnel research, depends on their relevance to the job (Borman, 1991). Since the performance ratings are ideally a judgment of a ratee's job-relevant strengths and weaknesses, they must reflect the systematic variation among individuals in job-relevant behavior. However, to the extent job performance ratings (or any other criterion) reflect systematic, nonjob-relevant variation or nonsystematic (error) variation, their usefulness is reduced; they are biased. Because actual criteria may include nonsystematic and irrelevant criterion variance, all criterion-related validity studies have the potential to produce attenuated validity coefficients (Guion, 1991). These attenuated validity coefficients will diminish the absolute value of the predictor weights and may alter decisions about which predictors should be used in selection programs. Systematic, but nonjob-relevant variance can change both the absolute and relative size of the weights assigned to the predictors used to select applicants.

The reliability of the criterion reflects the amount of its nonsystematic variation. In selection research, investigators have been concerned with the reliability of the criterion because low reliability produces underestimates of the validity of tests. Because criterion unreliability is a result of random error and is not correlated with selection procedure variance, it affects the validity estimates of all selection procedures equally. The effects of nonjob-related systematic criterion variance are more complex. Such systematic variance (method bias?) is not likely to be equally represented in each predictor variable, hence, regression weights for predictors will be affected differentially.

The purpose of this paper is to describe a method of analysis designed to assess the importance of predictor-related criterion bias in providing estimates of the validity and weighting of selection procedures. While we present only hypothetical data, we do suggest two bias constructs, liking and similarity, and provide analyses that assess the role they might play in distorting conclusions about test validity. In other contexts, the nature of the hypothesized bias factors may be quite different.

Bias

Researchers have proposed a variety of constructs which may produce bias. Rater liking of the ratee and similarity between rater and ratee are two constructs commonly suggested as possible sources of interaction of rater and ratee characteristics in a number of research areas (e.g., race - Kraiger & Ford, 1985; sex - Pulakos, White, Oppler & Borman, 1989; rater-ratee acquaintance - Kingstrom & Mainstone, 1985; age - Ferris, Judge, Chachere & Liden, 1991; and friendship - Love, 1981). These two bias constructs are discussed in the sections below.

Similarity among individuals has been conceptualized and operationalized in many ways. In 1953, Cronbach and Gleser proposed the distance measure (D) for assessing the similarity between profiles. Cronbach's (1955) classic discussion of the components of accuracy measures also described corresponding components of assumed similarity measures. More recent research has expanded the types of similarity studied. For example, Wexley, Alexander, Greenwalt and Couch (1980) studied actual wimilarity, and supervisor and subordinate perceptual congruity. Turban and Jones (1988) discussed three types of supervisor-subordinate similarity: perceived similarity, perceptual congruence, and actual similarity.

The content of these similarity measures has included demographic characteristics (e.g., Turban & Jones, 1988; Zalesny & Kirsch, 1989), nonwork-related attitudes (Wexley et al., 1980), and work-related attitudes (Tsui & O'Reilly, 1989) and in many instances these similarity indices have been related to measures of rated job performance. If predictors were also affected by these similarity indices as they might be for interviews, assessment centers, and some job sample measures, the potential for predictor-related criterion bias exists.

Most research on similarity and liking (e.g., Bunt & Armstrong, 1976; Ross & Ferris, 1981; Wexley & Nemeroff, 1974) has adopted Byrne's (1971) interpersonal attraction framework. Briefly, a portion of Byrne's theory proposed that similarity can lead to increased interpersonal attraction (or liking) because similarity results in consensual validation of views, which is rewarding. Overt stimulus characteristics, such as expensive clothing or attractive facial features, can also increase interpersonal attraction. Byrne and colleagues have demonstrated the far reaching effects of similarity, including its ability to influence ratings of intelligence (e.g., Byrne, London & Reeves, 1968). Byrne's theory raises a concern regarding bias in performance ratings. To the extent similarity and liking are unrelated to relevant job performance but affect both actual criteria measures and predictors, they result in predictor-related criterion bias. One example of the type of bias to which we are referring is presented by Bunt and Armstrong (1976). In their study, teachers' liking of students was not associated with the students' performance on an achievement test, but liking was associated with the grades teachers gave these students.