Evaluating Wall Street Journal survey forecasters: a multivariate approach

Business Economics, July, 2002 by Robert Eisenbeis, Daniel Waggoner, Tao Zha

The figures on individual performance can also be used to highlight those instances when forecasters take extreme positions. Yardeni made a big point about his concern for Y2K and the consequences if the United States and the rest of the world did not make the necessary preparations. His concerns were reflected in his forecasts in Figure 5 for the July 1999 and January 2000 Journal surveys: the accuracy scores are extremely low when compared with both those of other forecasters and how the economy actually performed. But not all forecasting accuracy problems are due to taking extreme positions. This is illustrated by the lower performance in terms of accuracy scores for all forecasters in January 1995 and in July 1990. This highlights the difficulty in predicting turning points. All forecasters had trouble with turning points, which is shown in Figure 6, containing the mean accuracy scores for all the forecasters, as well as the top- and bottom-ranked five forecasters. All forecasters made large errors in th eir January 1995 and July 1990 forecasts. More recently, forecasters made big errors in their forecast for January 2001, and clearly also had some difficulty with their January 2002 forecasts, as the strength of the economy was under-estimated. In addition, while there was a lot of agreement among the forecasters in January 2001 as the distribution of the forecasts was reasonably tight, all were systematically off the mark.

This figure also illustrates that at times there is more unanimity among forecasters than at others. For example the bottom five and top five forecasters were closer to each other in some periods than in others, suggesting that the variation in the forecasts, may serve as an indication of how much uncertainty there may be about where the economy is going. The dispersion, for example, widened considerably during the Asian crisis in the summer of 1997.

Conclusion

In this paper we have offered a systematic approach to evaluating a forecaster's performance relative to others and illustrated the methodology in the context of specific examples. Our approach formalizes a way of assessing forecast accuracy but could be applied to a number of different multivariate performance assessment problems. One may differ on how to estimate the variance-covariance matrix; but once it is reasonably approximated, our approach provides not only the ranking results but also the probability of how close to the actual data a particular forecast is in comparison with all other potential forecasts.

Appendix 1

When we make a forecast today of the values of a set of economic variables at some points in the future, we would not expect our forecasts to be perfect even if we had perfect knowledge of the inner workings of the economy. There are always events, such as political or natural disasters, that are impossible to predict and that affect the economy. More formally, we could not give perfect forecasts even if we knew the "correct" model of the economy. We will use the notation [[OMEGA].sup.E.sub.t] to denote the variance-covariance matrix of the forecast errors inherent in the economy and [[OMEGA].sup.F.sub.t] to denote the variance-covariance matrix of the forecast errors made by individual forecasters. If [y.sub.t] is the n-vector of variables to be forecast and [y.sub.t] is the forecast of [y.sub.t], we assume that both [y.sub.t] and [y.sub.t] have the same mean [y.sub.t], the variance-covariance matrix of [y.sub.t] is [[OMEGA].sup.E.sub.t], and the variance-covariance matrix of [y.sub.t] is [[OMEGA].sup.F.sub. t].