Cooking the questions: Phi Delta Kappa's annual poll is regarded as the definitive measure of where Americans stand on education issues. But are its surveys biased and its reporting self-interested?

Education Next, Spring, 2002 by Terry M. Moe

A few comments. First, PDK's claim that Americans are turned off by vouchers is simply untrue. When the purpose of a voucher program is well conveyed, most Americans respond positively. In this case, by a very big margin. Second, if this experiment is any indication, the well-worded item on PDK's own survey is downwardly biased, despite its reasonable wording. When it is nor placed immediately after the "at public expense" question, it produces higher support scores. Third, Gallup's well-worded question produces a support score that is 14 percentage points higher than the special-interest item's score. The gap would likely have been bigger still if Gallup had retained the pejorative phrase "at public expense." A reasonable measure of voucher support, fairly tested, gives much higher support scores than the "at public expense" item does (see Figure 3).

The Gallup experiment was available to PDK researchers well before they conducted their 2001 survey. One would think that, in light of this information, objective researchers would have modified their survey, or at least discussed the problems of measurement and interpretation that the Gallup experiment clearly raises. But nothing like this occurred. They designed their items as they always had, and when the results were in, they presented the "at public expense" findings to the media as hard evidence that Americans don't like vouchers.

Other Voucher Surveys

Over the years, many organizations have carried out surveys that ask questions about vouchers. Their surveys just haven't received as much attention as PDK's have.

It would be nice if these studies could somehow yield a single, coherent perspective on the voucher issue, but comparing them with any precision is a tricky business. Each has its own voucher item, its own ordering of questions, and its own range of topics being covered (which usually go well beyond education). All of these differences are likely to influence the results on voucher support. When these influences are combined with chance fluctuations due to sampling error--inherent in all surveys, regardless of how carefully they are designed-it is often impossible to tell exactly why surveys yield different results.

Even so, there is helpful information here. An important pattern in these studies is that voucher questions usually come in two types, which mirror the types we've been discussing. The first focuses attention on government subsidies for private school parents. The only real departure from PDK's approach is that government funding is described in some neutral way, without the pejorative" at public expense"-- a phrase that no one but PDK is inclined to use, The second type is worded to suggest that vouchers would expand choices for parents generally and that parents with children in public schools would be part of the program.

Not surprisingly, questions of the second type tend to produce much higher support scores than items of the first type do, Furthermore, they show that a majority of Americans tend to express support for the central purpose of a voucher program. The scores jump around from study to study, for all the reasons I've noted, and we are wise not to read too much meaning into particular findings or comparisons. On average, though, the existing studies tend to confirm--many times over--what we already know based on the PDK and Gallup results.