Don't be fooled by direct mail test results

Folio: The Magazine for Magazine Management, May, 1989 by John Klingel

Don't be fooled by direct mail test results

Testing is extremely important. The only way you can establish the best marketing, billing and renewal strategies is to test and test and test again. Most testing is very inexpensive, and all it takes is a little work.

Just as important (if not more so) as knowing what to test and how to test is knowing how to interpret test results. It may sound contradictory, but as important as testing is to business success, a less than thorough interpretation of results can lead to mistakes. Testing can reduce risk, but not eliminate it.

One of the simplest examples of this phenomenon is testing a higher price in an inflationary or good economic environment, seeing favorable results, and then responding by rolling out a large campaign at the higher price during a recession or economic downturn (which needn't necessarily be national in scope).

A city magazine tested a postcard package against an expensive four-color, six-by-nine package. The local economy was booming; the postcard won. The magazine's newsstand sales and insert card volume went up. However, because the magazine's rate base didn't require it, no campaigns were mailed for over a year. Then the local economy went into a depression, newsstand sales plummeted, insert card volume dropped--and suddenly there was a rate base need.

Based on test results from a year and a half earlier, the city magazine used the postcard as the control and rolled out a campaign. They also included a test panel of the four-color, six-by-nine package as a retest. This time the four-color package won by a large margin. Historically, response to both packages was down from the high responses achieved when the local economy was strong. However, the postcard response had dropped much more than the six-by-nine package.

There are a number of lessons in the above example. One is that if you eliminate direct mail from your source mix, you should continue testing because you need the market information it provides. But the primary observation is that you're always testing in one market and using the results in another.

When you're analyzing test results, it's important to understand the difference between relative and absolute response differences. Let's assume that you test a new package and see the following results: Control, 5.0 percent; New package, 5.75 percent.

The new package increased response by 0.75 percent, or 15 percent. The absolute increase was 0.75 percent (5.75 percent minus 5.0 percent). The relative increase was 15 percent (5.75 percent divided by 5.0 percent and subtract 100). Most of us concentrate on the relative increase and say the new package increased response by 15 percent. But it's not always clear when we are looking at a relative response or absolute increase.

Prior to getting into the magazine business, I worked with a product that got 20 percent response from prospect lists (people who hadn't purchased from us) and 60 percent response from house lists (past buyers). The high responses were a function of multiple orders (response was based on the number of orders instead of the number of people who responded). One of my fellow employees, who has since gone on to fame and fortune in the magazine business, came up with an incredible concept that called for sending samples of the product with the direct mail. At the time, direct mail costs were about $150 per thousand and the sample cost $400 per thousand in the mail. Given some of the best product profit margins I've ever worked with, it was economically feasible to test the concept.

When the sample concept was tested, response from the prospects went up 40 percent--from 20 percent to 28 percent. Response from the house list went up 13 percent--from 60 percent to 68 percent. Note that in both cases response went up by an absolute 8 percent. Did the concept raise response by an absolute 8 percent or did the concept work relatively better on prospects than it did on actives? There's no clear answer to this question. It's only another example of the fact that you have to think a little when you're interpreting test results.

One place where looking at absolute versus relative differences is extremely helpful is in reading early pay-up results. Let's assume that you tested a relatively inexpensive postcard format against an expensive four-color package, and response to both packages was 5 percent. After three or four weeks of payments, the pay-up is as follows: Four-color package, 20 percent; Postcard, 10 percent.

If pay-up on the four-color package is normally 50 percent, do you project final pay-up on the postcard to be 25 percent? If you use the relative difference, pay-up on the postcard is 50 percent of the control. What I normally discover is that final pay-up on the postcard will be 40 percent. In other words, there will be an absolute drop of 10 percent.

As with any general guidelines, you don't want to follow this one blindly. However, in many cases, the absolute difference in pay-up after four or so weeks will be the absolute difference in final pay-up. For some reason, the difference in pay-up tends to show up early in the payment pattern.