How to spot a bogus poll

American Demographics, Oct, 1996 by Brad Edmondson

Opinion surveys can look convincing and be completely worthless. But asking four simple questions of any poll can separate the good numbers from the trash.

Politicians use opinion polls as verbal weapons in campaign ads. Journalists use them as props to liven up infotainment shows. Executives are more likely to pay attention to polls when the numbers support their decisions. But this isn't how polls are meant to be used. Opinion polls can be a good way to learn about the views Americans hold on important subjects, but only if you know how to cut through the contradictions and confusion.

Conducting surveys is difficult. It is especially difficult to take a meaningful survey of public opinion, because opinion is a subjective thing that can change rapidly from day to day. Poll questions sometimes produce conflicting or meaningless results, even when they are carefully written and presented by professional interviewers to scientifically chosen samples. That's why the best pollsters sweat the details on the order and wording of questions, and the way data are coded, analyzed, and tabulated.

Pollsters other than the best can also set up surveys that deliberately shade the truth. They do this by acting like trial lawyers: they ask leading questions, or they restrict their questions to people likely to give the desired response. In fact, pollsters can use dozens of obscure tricks to intentionally push the results of a survey in the desired direction. So the next time a poker-faced person tries to give you the latest news about how Americans feel, ask some pointed questions of your own.

Did You Ask the Right People? In 1936, the editors of Literary Digest conducted a Presidential preference poll of more than 2 million Americans. The poll predicted that the Republican candidate, Alf Landon, would defeat Franklin Roosevelt. Landon's loss made the Digest history's most famous victim of sample bias.

The Digest mailed more than 10 million ballots to households listed in telephone books and automobile registration records. This method might create a relatively representative sample today, but in 1936, it substantially biased the sample toward those affluent enough to own cars and phones. The magazine's disgrace was made complete by young poll-takers like George Gallup and Elmo Roper, who used samples of a few thousand people to predict a Roosevelt win. Gallup and Roper carefully chose their samples to reflect a demographic cross-section of Americans, just as they do today.

The most amazing thing about this story is that some journalists and businesses in the 1990s still make the mistakes the Literary Digest made 60 years ago. Any journalist with half a pencil knows know that only a scientifically chosen survey sample will represent the country's opinions. But the temptation to take a biased poll is great if you have a tight deadline and a small budget, as many news organizations do.

The 2 million who responded to the Literary Digest poll in 1936 were even more likely than the total sample base to be wealthy and Republican, typifying a common survey problem--nonresponse bias. Even when you start out with a representative sample, you could end up with a biased one. This is a risk all pollsters take, but some particular methods lend themselves to greater error. For example, readers of women's magazines are frequently asked to fill out surveys on weighty subjects like crime and sexual behavior. Not only do such polls ignore the opinions of nonreaders, they are biased toward readers who take the trouble to fill out and return the questionnaire, usually at their own expense.

Television news and entertainment shows get into the act by posting toll-free or even toll numbers that viewers can call to "vote" on an issue. These samples are not only biased, they are prone to "ballot-stuffing" by enthusiasts. In other words, viewers who call 12 times get 12 votes.

Poll results based on "convenience" samples can be wildly misleading, even if the sample sizes are huge. A call-in poll conducted by a television network in 1983 asked: "Should the United Nations continue to be based in the United States?" About 185,000 calls were received. Two-thirds said that the U.N. should move. At the same time, the network conducted a random-sample poll of 1,000 people, and only 28 percent said the U.N. should move.

Between 1989 and 1995, the Pew Research Center for The People & The Press in Washington, D.C., monitored the public's interest in 480 major news stories. Almost half of Americans paid little or no attention to these stories, and only one in four followed the average story very closely Stories about wars and disasters were followed most closely, while those about celebrity scandals and politics finished last. When Pat Buchannan announced that he was running for President in 1991, for example, only 7 percent of Americans paid close attention to the story.

Conflicts make news. When journalists are trying to liven up a boring political story, they need angry, well-informed citizens like a fish needs water. This is one reason why older men may be quoted more often than other groups. Those aged 50 and older are more likely than younger adults to follow news stories "very closely," according to the Center, and men are more likely than women to follow stories about war, business, sports, and politics.