School Voucher Programs: What the Research Says About Parental School Choice

Brigham Young University Law Review, 2008 by Wolf, Patrick J

The fact that some students randomly offered vouchers do not use them, and some control group members attend private schools without vouchers, does not in any way bias the estimate of the impact of offering students vouchers, though it does generate a conservative estimate of the effects of actually attending private school.64 This is because the outcomes for voucher decliners, for whom the impact of the voucher is zero, are averaged in with those of voucher users in calculating the experimental impact of vouchers. Likewise, any change in outcomes experienced by control group members who attend private schools are included in an experimental analysis on the control-group side of the comparison. If analysts or policymakers want to draw from an experimental evaluation in determining the impact of actual voucher usage or private schooling, established statistical techniques exist and are regularly employed in experimental voucher evaluations to produce unbiased estimates of those impacts.65 So, if one is interested in the average effects of a program that merely offers students vouchers, random assignment studies, as traditionally implemented, generate unbiased estimates of that average "intend-to-treat" impact. If one is instead interested in the average effect of obtaining the actual experience that vouchers are supposed to enable students to receive-private schooling-then established statistical methods exist that can be and are applied to experimental voucher data to produce unbiased estimates of private schooling either through voucher usage or in general. The most commonly used such method is Instrumental Variable (IV) analysis with the original voucher lottery as the ideal instrument.66

Second, some researchers claim that experimental evaluations of voucher impacts suffer from "generalizability" bias because the populations of students who choose to apply for voucher programs are different from non-applicants. It is true that the results of any particular experimental voucher evaluation only strictly apply to the special conditions in which the program was designed and implemented. It would be risky to claim that the results of an experimental voucher evaluation of a means-tested inner-city program would automatically apply to a statewide voucher program for students of any income level but with disabilities. Those are two populations that differ in ways that could plausibly influence their response to vouchers, so analysts should not, and generally do not, make such generalizability claims. This condition is not, properly understood, a "bias" of experimental voucher studies, since it does not undermine the validity of experimental impact estimates. It is simply a limitation.

Experimental evaluations are purposely designed to be exceptionally strong in their "internal validity"-that is in their ability to reach an accurate determination as to whether or not the voucher program impacted a certain group of study participants.67 Experiments of all types are inherently limited in their "external validity," that is, in the ability to apply the results of one study of a particular student population to the context of a very different student population. Presumably, one need determine with confidence whether or not an educational intervention works with a given set of students before one considers whether it might work for a different group of, or all types of, students. That is why experimental evaluators of voucher programs qualify their findings to make it clear that different results could emerge from similar evaluations of very different student populations.68 That is not a bias, just good scholarly practice.