Graduate Record Examination validity in a Master's program in clinical psychology

Canadian Psychology, Feb 1999 by Symons, Douglas K

The common use of Graduate Record Examination (GRE) scores to inform graduate program selection decisions has continued despite reservations that scores might not predict much more than graduate-level course grades. For example, Sternberg and Williams (1997) have investigated the predictive validity of GRE scores at Yale University, and found that scores predicted around 4% of the variance of first-year grades. However, when they related GRE scores to internal and external ratings of the quality of dissertation produced by students, only the GRE Analytical score was predictive, and only for male students. In contrast, the GRE Technical Manual (Briel, O'Neill, & Scheuneman, 1993) reports a number of studies in which GRE scores modestly predict both first-year grades as well as faculty ratings of performance, although these relations are not moderated by gender. These weak patterns justifiably give pause to many faculty involved with graduate programs that require GRE scores (as in Ceci, 1996), given the critical nature of admission decisions for applicants.

High-quality, well-known graduate programs invariably have students with very high GRE scores. Sternberg and Williams (1997), for example, report means that range from 648 to 699 on subtests. Studies often have an adequate range in scores, but the distributions centre around the 90th percentile, few students have relatively low scores (e.g., below 50th percentiles), and students with no scores are either not included in analyses or not admitted to programs in the first instance (Darlington, 1998; Kuncel, Campbell, & Ones, 1998; Ruscio, 1998). Analyses are often based on correlational statistics, but predictive relations can also be defined by using cut-off scores that discriminate more from less successful students. This approach is only possible, however, when sufficient proportions of students score below cutoffs. In addition, outcomes are potentially biased and unreliable (Kuncel et al., 1998; Thayer & Kalat, 1998), particularly when they are embedded within the program itself (e.g., grades, faculty ratings). Reliable and objective external criteria would seem critical to use as outcome measures, although the selection of these may differ from program to program depending how "success" is defined for graduates.

Whereas predictive validity data almost exclusively come from Ph.D. programs in the United States, some Master's programs also use GRE scores in admissions procedures. In Canada, Master's programs vary between those: (a) within a Ph.D. program where most or some students continue on; (b) that do not require GRE scores; (c) that are geared almost exclusively towards producing practitioners; and, (d) that produce students that either continue on in graduate studies elsewhere or seek employment. Predictive validity studies would need to differ quite dramatically between these different kinds of programs because the presence of predictor and nature of criterion variables would vary. The purpose of the current report is to present predictive validity data collected in our Master's program in clinical psychology at Acadia University, and thus what follows can be considered a case study. Ours is one of few programs in Canada that fits the fourth kind of program, but this offers methodological advantages when choosing important and external criterion variables.

Our program typically admits 4 or 5 students per year, and the current sample consists of 37 students (29 women, 8 men) who have completed their 2-year residency requirement in the program over the past 10 years. Upon application, 12 of the admitted students did not provide GRE scores, and of the other 25, 4 did not write the Advanced test in psychology. Cumulative undergraduate 4-point GPAs and 4 GRE scores were used as predictors, and these means (see Table 1) are lower than would be found in Ph.D. graduate programs in larger schools or Ph.D.-level programs (see listing in APA, 1996).

Students generally enter the program with the goals of future admission into a Ph.D. program, getting external funding to do so, graduating on time, and doing good research. These goals are shared by faculty members involved with graduate supervision. Outcome variables were therefore selected to address these issues where criteria were as objective as possible and provided by criteria external to the department:

Acceptance to Ph.D. program (n = 14, 37%). The first outcome variable was whether students were accepted into a Ph.D. program. In Canada, there are currently no Psy.D. programs, and while two students were accepted into U.S.-based Psy.D. programs, they were not included here because selection criteria for those programs differed markedly.

External awards (n = 7, 19%). Students can apply for federal funding from national granting councils (e.g., SSHRC, NSERC) when entering or completing doctoral studies, and success in these competitions was addressed. It should be noted committees adjudicating these awards do not have access to GRE scores, and no students held these awards when admitted to our Master's program.