A policy-capturing approach to individual decision-making: a demonstration using professors' judgments of the acceptability of psychology graduate school applicants

Canadian Journal of Behavioural Science, Oct 1995 by Kline, Theresa J, Sulsky, Lorne M

Abstract

The purpose of this paper is to demonstrate how to carry out research using the policy-capturing method. The context for this tutorial was that of examining the decision-making policies of 19 university professors who rated hypothetical graduate student applicants with regard to the probability that they would accept them into graduate school in psychology. The raters were given five pieces of information (i.e., "cues" in policy-capturing terminology) for each applicant to use in determining their ratings: GRE-Verbal score, GRE-Quantitative score, GPA over the last 15 courses, grade in a full-year undergraduate statistics class, and the aggregated percentile at which three referees placed the applicant. Results are discussed in terms of interpreting the consistency of the raters' weighting of the cues, the reliability of their decisions, and how to assess complex decision-making models. The discussion centers on methodological and theoretical issues arising from the application of the policy-capturing approach to decision-making research

The purpose of this paper is to demonstrate how to carry out and interpret the results of a study that uses the "policy-capturing" approach to examine individual decision-making. The specific decision-making task that formed the context of this tutorial was how professors decide which graduate school applicants are qualified for admission to graduate school. We are aware that a voluminous literature exists examining the validity of a variety of measures (e.g., GRE scores, GPA, etc.) for predicting graduate school success (e.g., Boudreau, Killip, MacInnis, Milloy, & Rogers, 1983; Kaczmarek & Franco, 1986; Willingham, 1974; Young, 1986). The purpose of this study was not to contribute to this general literature, but instead to document the utility of the policy-capturing methodology for understanding how individuals combine information from such measures in making decisions.

The basic question asked in policy-capturing research is "What is the decision maker doing with the information available?" (Slovic & Lichtenstein, 1971). This method presents the decision-maker with a series of decision situations where the values, or scores, on the predictor variables (i.e., "cues") are varied. The decision-maker is asked to review the information given and provide an overall, global rating that best summarizes his or her judgement for each situation presented. For example, a decision-maker might be asked to predict the overall number of games a baseball team might win over the coming season (overall rating) based on the following five cues: earned run average, batting average, double plays, stolen bases, errors (Balzer, Sulsky, Hammer, & Sumner, 1992).

Policy-capturing is a widely accepted approach, having been used in a variety of applied settings (e.g., Bernardin & Buckley, 1981; Dougherty, Ebert, & Callender, 1986; Nystedt & Magnusson, 1973; Ullman & Doherty, 1984). The goal of this approach is to understand an individual's decision making "policy" by observing the relationships between the decision cues given to the individual (e.g., GPA, GRE scores), and the final decision made by the individual (e.g., probability of accepting the student into graduate school) and then modelling that relationship using an idiographic multiple regression analysis (i.e., regression analysis carried out for a single individual). The results of the analysis provide a description of how the individual decision-maker weights the various cues to arrive at his or her decision. Thus, within the constraints of the cue information presented, each individual's decision-making "policy" can be observed.

The multiple R from the regression equation describes the consistency with which the decision-maker made his or her decision, and the standardized regression weights for each cue may be used in some instances to determine the relative importance of each of the cues in making their overall judgements. One might also be interested in examining individual decision-makers' complex decision policies, such as employing the information in an interactive or non-linear manner. These are described later in detail.

Once the regression weights have been derived, they can be used to examine differences across decision-makers (e.g., Anderson, 1969). For example, in this study we examined possible individual differences, such as whether professorial rank was related to the weights given to the cues.

METHOD

Participants

Nineteen professors from the Department of Psychology at The University of Calgary volunteered to participate as subjects. Participants represented the three professorial ranks as follows: seven Assistants, three Associate and nine Full. Their mean age was 44.17 yrs (SD=10.58), ranging from 28-63. The mean number of years as an academician was 13.58 (SD=10.52), ranging from 0-32 (the "0" was a newly appointed individual). Their graduate student supervisory experience ranged from 0-40 master's level students (M = 7.4; SD = 9.1) and 0-13 Ph.D. level students (M = 2.7; SD = 3.7). Five of the 19 participants were female.