Cooking the questions: Phi Delta Kappa's annual poll is regarded as the definitive measure of where Americans stand on education issues. But are its surveys biased and its reporting self-interested?

Education Next, Spring, 2002 by Terry M. Moe

The 33rd Annual Phi Delta Kappa/Gallup Poll of the Public's Attitudes Toward the Public Schools

By Lowell C. Rose & Alec M. Gallup

Phi Delta Kappa International, 2001.

Support for vouchers is declining. Or so we are told by Phi Delta Kappa (PDK), the nation's premier association of education professionals, whose annual poll (conducted by Gallup) is widely accepted as the definitive measure of where Americans stand on education issues.

On the surface, it might seem that PDK is just documenting the obvious. After all, two voucher initiatives--one in California, the other in Michigan--were roundly defeated in the 2000 elections, and since then the buzz in education circles is that vouchers have dropped in popularity.

Much of this talk, however, is part of a public relations campaign being waged by the opponents of vouchers, whose aim is to persuade policymakers to stay away from the issue. Such a campaign is to be expected from the teacher unions and their allies, because this is the way the game of politics is played. My fear, though, is that PDK is actively participating in this spin campaign and has been for years.

I cannot read the minds of PD K'S researchers, and I do not want to accuse them of such a thing. But I'm convinced that one of two conclusions is justified. Either PDK's polls have purposely been designed to reflect negatively on the voucher issue. Or its researchers have been careless in their design decisions--which I doubt.

PDK's Key Measure

From the 1970s until 1991, PDK measured voucher support with a survey item that defined vouchers as a government-funded program allowing parents to choose among public, private, and parochial schools. After support rose to 50 percent (with 39 percent opposed) in 1991, PDK abruptly dropped this item in favor of a new one, The new question read:"Do you favor or oppose allowing students and parents to choose a private school to attend at public expense?" This question, first asked in 1993, gave results that were strikingly more negative: only 24 percent expressed support (see Figure 1). Indeed, it indicated that even private school parents were opposed to vouchers, a result no expert would be prepared to believe.

Why such different "facts"? Research has long shown that most Americans are poorly informed about public policy and don't have well-developed views on most issues, Recent polls have shown the same for vouchers. This does not mean that Americans can't connect the voucher issue to their own values and beliefs. But it does mean that, because they come to any survey with little information, they will be quite sensitive to information contained within the survey itself, especially to the specific wording of questions and the order in which they are asked, This information determines how the issue is "framed." And the framing, in turn, influences which (of many possible) values and beliefs get activated in people's minds, and thus how people respond.

All public-opinion researchers are well aware of this. And they all know that, if public opinion is to be well measured on an issue, the issue must be framed with great care, The framing should provide respondents with enough information to give them a good sense of what the issue is about. The information also needs to be balanced, so that respondents are not pushed to see the issue in a positive or a negative light.

PDK's "at public expense" item does not even come close to meeting these basic criteria, The central purpose of a voucher program is to expand the choices available to all qualifying parents, especially those who now have kids in public schools, But the PDK item does absolutely nothing to convey this information. It says nothing about choice, nothing about public school parents' being eligible to participate. Instead, it focuses entirely on private school parents and asks respondents whether the government ought to be subsidizing them. Vouchers are presented, in effect, as a special-interest program for an exclusive group.

This is bad enough. But PDK's researchers compound the bias with the way they choose to inform survey respondents that vouchers are funded by the government. They could have said just that, or they could have found some other neutral way of wording it. Instead they settled on the phrase "at public expense"--which is implicitly pejorative and begs for a negative reaction.

By scientific standards, PDK's "at public expense" question is a poor measure of voucher support. It should never have seen the light of day. Nevertheless, PDK not only adopted this item as its own, but has persisted in using it in every annual survey but one since 1993--with results that, predictably, are on the low end of what we would expect, given the results of better-worded polls on the subject.

PDK's Second Measure

Interestingly, soon after the "at public expense" item made its appearance, PDK introduced a second question to measure support for vouchers. This one is actually informative and neutral, precisely the kind of item that should have been used all along. As such, it gives the appearance that PDK is seriously trying to get a valid measure of voucher support. It reads, "A proposal has been made that would allow parents to send their school-age children to any public, private, or church-related school they choose, For those parents choosing nonpublic schools, the government would pay all or part of the tuition. Would you favor or oppose this proposal in your state?"