Can bad polls drive out good?

National Review, Oct 19, 1992 by Humphrey Taylor

TO MUCH of the media, a poll is a poll is a poll. "Newsworthiness," not quality or accuracy, determines which polls get reported. Here are some points to watch out for.

1. An unexpected poll finding is more surprising, and hence more "newsworthy," than one that confirms what other polls also report. Bad (i.e., inaccurate) polls are more likely to be surprising and therefore more likely to be reported.

2. Bad polls are cheaper than good polls. A survey of 600 people is less expensive than a survey of 1,250. A poll with very few questions is cheaper than one with more questions. High-quality sampling and interviewing cost more than poor-quality. And so on. If the media report findings regardless of their quality, why spend money on better polls?

3. One-day "instant" polls are much less accurate than polls conducted over three or four days because of all the people they miss. But the media love them because they are the "first with the news" about public reactions to events.

4. 900-number straw polls are not polls. The people who are watching the TV shows and choose to phone in their answers are often very different from the population as a whole.

5. Poll results are the answers to questions and are therefore critically dependent on the wording and, sometimes, on the order of the questions. Never interpret the results without reading the questions carefully.

6. Opinion on most issues is more complicated than a yes/no to one or two questions. Polls inevitably simplify and categorize. To really understand public opinion on any issue-- from abortion to economic policy--it is necessary to review the answers to a variety of questions addressing the issue in different ways.

7. A poll is valid only for the population surveyed and for the time of the survey. Surveys of adults, registered voters, or likely voters will all yield different answers.

8. Focus groups are not polls. Getting eight to ten people in a room to talk together has many uses, but it does not provide a measure of public opinion, as surveying a representative cross-section of people does.

9. Electronic town halls are not polls; they are not even a national version of the New England townhall meeting. For that, tens of millions of people would have to participated, and there would have to be a comprehensive debate on each issue, with all major points of view discussed before a vote was taken. Unlikely.

10. Candidates' polls are often misleading. Polls are leaked not to inform but to influence the media and the public.

11. Polls really don't predict, they just measure. Good polls report what a representative cross-section of the public (or of likely voters) say when interviewed. The only reason that pre-election polls "predict" elections is that most people tell the interviewers how they think they will vote and then actually vote that way. But some people change their minds, and some of those who say they will vote don't. The difference between intentions and behavior, not sampling error, is the main reason why polls sometimes get it wrong.

Does this mean that all polls are worthless? Obviously not. To paraphrase Winston Churchill, polls are the worst way of measuring public opinion except all the others. We are better informed with polls than we were without them. But poll readers should recognize that polls differ dramatically in quality and accuracy, that those who pay for and publish polls have different motives.

If nobody cares about the quality of polling, cheap and dirty polls will surely drive out good ones.