MEASURING THE SIGNAL

Medical Marketing and Media, Mar 2008 by Hellebusch, Stephen

There is no point in running a great journal ad if no one notices it. But how do you truly determine if the signal is cutting through the noise? Stephen Hellebusch tests the theory of signal detectability on a group of primary care physicians

When an advertisement is placed in a professional journal, marketers sometimes wonder if anyone will see it. If it is great, but no one notices, what's the point? Will it break through the clutter of the other ads? Will the signal cut through the noise?

This article attempts to offer a clear, standardized way to tell. The Theory of Signal Detectability (TSD) suggests using the "signal-to-noise" ratio to determine True Recognition, which is the percent who say they have seen an ad that really have seen it, minus the percent who say they have seen it who really have not. The measure takes into account both accurate and false recognition: Hits and False Alarms.

There are also other elements of print ads worth knowing. What does the advertisement communicate? Is its message extremely important? Is it believable? New and different? Is it persuasive? What tone or image does the advertisement convey?

Most useful would be to have a Norms base to set results against, to be able to determine if an ad is above, below, or at norm with respect to True Recognition and measures of these other elements.

To address these questions, and to test the practicality of the Signal Strength technique, a study was conducted with 31 primary care physicians (PCPs-defined as family/general practitioners and internists). The objectives were to:

1. Measure True Recognition for four print journal test ads.

2. Obtain diagnostic rating and adjective checklist information for each.

3. Begin developing TSD Norms by including three "control" ads. This method is intended to help discriminate between different advertisements and to help make an informed decision about the uses to which a given ad will be put.

The 31 PCPs were split into two groups, designated the Hit group (16 doctors) and the False Alarm group (15).

Each respondent viewed online two lists of 30 different journal ads. The lists were carefully constructed to allow for signal strength measurement. Ads were always presented in random order.

In the Hits group, the second list showed 22 items not seen before, and seven of the items that were repeats from the first list.

Four of the seven ads were test ads; the other three ads were used as controls, to be included in future research so that test-retest reliability could be established.

In the False Alarm group, the same seven ads were in the second list. In fact, that list was exactly the same as the one the Hits group saw. But the False Alarm group had not previously seen any of these seven ads.

When viewing the second list, for each ad, all doctors were asked if they saw it in the first list or not (yes/no) and how certain they were of the answer (very, somewhat, not at all).

Following this task, the doctors were asked specific questions about each of the four test ads: They were asked to rate each on how important, believable, new and different, and persuasive the information presented was.

Additionally, doctors were asked to check any of a set of 10 adjectives which might apply to each test advertisement. Finally, they were asked about the main message of one test advertisement. Interviewing occurred between April 2-4, 2007.

FINDINGS: Signal Strength, True Recognition

The True Recognition percentages for the seven tested print ads, derived from the Hits and False Alarms, are shown in Fig. 1. The Norm is the simple average.

Surprisingly, given the low base size (N = 15), one of the ads differs from the Norm. The Tylenol ad was shown to have a very strong Signal, and is significantly (95% Confidence Level) above Norm.

The seven ads attained their Signal scores in different ways Specifically, the high scoring ads had many Hits and no False Alarms This suggests that they are distinctive, since no PCP said they had seen them when they had not.

In contrast, the low scoring ads had a unusally high percentage of False Alarms, suggesting they look like many other advertisements-a high percentage of physicians who had not seen them before said that they, in fact, had. For instance, the high False Alarms are the main reason the Centocor ads and Crestor ads are not truly being recognized.

Certainty

When the doctors were asked how sure they were about the Hits, the degree of certainty roughly paralleled the percent of hits overall (Fig. 2). They were most certain about Tylenol (the strongest Signal) and least certain about Crestor (one of the weakest Signals).

In looking at the certainty around False Alarms (Fig. 3), two things are evident. One is that there is a lot less certainty. This makes sense, since these are errors-claims of seeing something not seen. The other observation is that the level of certainty rises slightly with the level of error, suggesting that those ads that look most like other ads truly confused some PCPs.