Forecasting: Robert Fildes and Paul Goodwin investigate the effectiveness of firms' judgmental adjustments to computer-generated demand forecasts

Financial Management (UK), Dec, 2007 by Robert Fildes, Paul Goodwin

Accurate forecasts of demand are crucial for effective supply-chain planning, since faulty predictions lead to poor customer service, which is costly. In an effort to reduce error, many firms buy expensive forecasting software that uses statistical methods. Despite the sophistication of these systems, their forecasts are often adjusted by managers, who use their judgment and market knowledge to try to improve accuracy.

Is making such changes a key function of the well-informed manager or merely a waste of valuable time? By looking at data from over 60,000 forecasts made by four UK businesses, we aimed to find out what makes so many managers override the forecasts produced by their software--and whether these adjustments increase accuracy or not. We studied a nationwide retailer, an international food company, the subsidiary of a US pharmaceutical firm and a manufacturer of cleaning products. We also interviewed the firms' forecasters and observed forecast review meetings where managers deliberated over their adjustments.

[ILLUSTRATION OMITTED]

Adjusting forecasts was a popular activity: on average over our sample, three-quarters were changed in some way. Forecasters at the food manufacturer modified 91 per cent of theirs, for example. While the retailer's four forecasters adjusted only eight per cent of theirs, they had over 26,000 to produce each week, so there probably wasn't time to make many more changes. The pharmaceutical firm held 17 forecast review meetings--tying up about 80 hours of management time--every month.

Many of the changes were small, as if the forecasters sometimes simply wanted to put their calling card on forecasts by tweaking them slightly to show that they were still doing their jobs. We received anecdotal evidence indicating that managers would alter more of those forecasts that were presented earlier in a review meeting. Later forecasts were simply waved through as people got tired and felt that they'd already done enough to justify calling the meeting.

Of course, people didn't make changes purely to show how hard they were working. They usually felt that they had good reasons and, in some cases, we found that this was so. But the problem is that people have a tendency to find an explanation for every movement in their sales graphs, including swings that really are random. This makes them overconfident that their adjustments will increase accuracy, so they tend to adjust forecasts even when it isn't appropriate. We're brilliant at inventing theories for everything we observe. At 1.01pm one December day in 2003, just after Saddam Hussein had been captured, the price of US treasuries rose. Half an hour later the price fell. The Bloomberg news channel used his capture to explain both movements. The dull statistical forecast is no competition for these colourful but often groundless tales, so it gets adjusted.

All this adjustment behaviour can have some odd consequences, according to psychologists. When we perform a task that demands our skill, we normally think we can control the results. For example, if you learn to play a musical instrument, you expect to make fewer mistakes as you practise more. Many swings on a sales graph result from random, unpredictable events. Yet, because forecasters see making adjustments as skilful, they can start believing that they have some control over what they're forecasting and, therefore, that they can predict movements on the graph. This is known as the illusion of control. It's likely to cause forecasters to make even more changes. After all, the more you adjust, the more control you think you have.

Despite these concerns, judgmental adjustments to statistical forecasts can still play a useful role in improving accuracy. We found that, on average, they reduced the average absolute percentage error of the forecasts by 3.6 percentage points for all companies except the retailer. But this modest improvement masked considerable variations in the effectiveness of the changes.

Is it possible to filter out the types of adjustment that are unlikely to help? Larger changes are more likely to increase accuracy, although it takes some nerve to make them and risk being badly wrong. Bigger alterations are, therefore, likely to be made for very good reasons. For example, a forecaster might have reliable information on an important future event, such as a sales promotion, that will cause the statistical forecast to have a big error. In contrast, the smaller changes may be the type of tweak that we mentioned earlier, or they may be the result of a forecaster hedging their bets because the information they have on a future event is unreliable. The lesson is clear: although small alterations, by definition, do relatively little harm to accuracy, they're generally a waste of time.

Positive adjustments are more likely to be wrong than the negative ones. Psychologists tell us that people have an innate bias towards optimism. For example, construction projects usually take longer to finish and cost more than predicted. Some of this may be a result of deliberate misrepresentation to win contracts, but there is evidence that optimism bias still plays a role in these poor estimates. It seems, therefore, that when forecasters are asked to estimate the effects of, say, an advertising campaign they cannot resist being optimistic. And, of course, this is often exacerbated by the infectious enthusiasm of their colleagues in sales and marketing.