Measuring up: testing the pretensions of market research and polling. . - Culture and Reviews - book review

Reason, May, 2003 by Brian Doherty

While it seems to me that the mysterious health insurance company was wasting the money it paid for my focus group, it's their money to waste. Believing in free markets doesn't mean believing in an error-free, best-of-all-possible-worlds equilibrium. Free markets are rife with errors, and the market process and profit and loss system are an efficient method of sorting them all out. As the old saying goes, at least half the money you spend on marketing is always wasted.

My experience with focus groups and many of Bogart's sideways comments (though he is by no means out to bury market research) cast shadows on the bright, clean, rational world of scientific market research. Bogart notes that "many studies are planned and questionnaires are written without the theoretical underpinning that provides insight and understanding rather than mere factual detail of transient interest." He says otherwise savvy business executives often take meaningless numbers seriously, and he mocks those who posit that "the inchoate and monosyllabic utterances of respondents commenting on test advertisements [are] 'opening a window into the human soul."' He thinks the current marketing obsession with youth is more "herd behavior...than marketing wisdom." Mistakes, it seems, are being made. How many and at what cost? Ask the marketers of New Coke; they might have some clue.

It might be that polling and market research are often a pure consumption expense: entertainment for political junkies, or a way for corporate executives to feel better about risky decisions. Bogart cites a 2000 survey of executives who said that what they overwhelmingly wanted from their market research was "to be told what to do." He observes that "the validity of the advice they received seemed subordinate to the air of assurance with which it was uttered." A lot of the information Bogart presents suggests that, when it comes to marketing, we often don't know what people are going to do and we can't do anything about it. I can see why people would pay a lot to avoid that grim message.

Is this thing called "public opinion" even real and measurable? Weissberg takes care to say he isn't questioning that. But Bogart dismisses the concept as "an amorphous and unstructured combination of sentiments and loyalties." What's more, as marketing guru Oren Harari has noted, our desires and opinions about products and services are eternally changeable and only really discoverable through action. Opinions we express in any artificially created now will be constrained by extant options. The most telling critique of most polling and market research is that it leaves no room for, or doesn't pay enough attention, when people say, "I don't know."

Harari, a professor of management at the University of San Francisco's Graduate School of Business and a Tom Peters associate, noted in a 1994 speech that market research indicated public disdain for answering machines and hair mousse before their introduction. "When people are unfamiliar with a product, or cannot fathom its possibilities," he said, "market research will reflect that and nothing more....Market research can suggest, in a narrow way, what people might prefer or dislike today, but not what will excite them tomorrow."