Evaluating Wall Street Journal survey forecasters: a multivariate approach

Business Economics, July, 2002 by Robert Eisenbeis, Daniel Waggoner, Tao Zha

This paper proposes a methodology for assessing the joint performance of multivariate forecasts of economic variables. The methodology is illustrated by comparing the rankings of forecasters by the Wall Street Journal with our alternative rankings. The results show that the methodology can provide useful insights as to the certainty of forecasts, as well as the extent to which various forecasts are similar or different.

**********

Economic forecasting usually involves making simultaneous predications of key financial and real macro economic variables at intervals of months, quarters, or even years into the future. Yet even when models are estimated simultaneously, forecasters typically focus on the out-of-sample accuracy of individual variables or dimensions of economic performance and not on the overall accuracy of their description of the economy. Bluechip collects forecasts from a panel of experts monthly, and the forecasted values of many series are presented, but no summary measures of joint accuracy are provided. In contrast, twice a year at the beginning of January and July, the Wall Street Journal surveys a group of forecasters for their forecasts of several key macroeconomic variables designed to characterize what the performance of the economy will be. The Journal publishes the individual forecasts and does provide a ranking of a few of the top forecasters, based on how close the forecasts of the variables are to their realiz ed values. The actual methodology used to provide these rankings has changed over time but at present simply ranks the forecasters on the sum of the weighted absolute percentage deviation from the actual realized value of each series, where the weight for each series is simply the inverse of the actual realized value of the series. This performance assessment method may become distorted, and even undefined, when the realized value is close to or equal to zero. Moreover, it does not consider the correlations in the data among the variables being forecast. This latter consideration is important because accuracy should reflect internal consistency in predicting the performance of the economy and not merely good luck on one particular dimension.

In this paper we propose a methodology, which not only yields a measure of joint forecast performance, but also provides a single measure of how similar a joint forecast is to those of other forecasters. The method also allows us to assess the collective forecast accuracy of all the forecasters and the accuracy of individual forecasters over time. The procedure is not even dependent upon having all forecasters represented in each forecast period. Finally, it provides some indication of how tightly the forecasts are clustered around the realized values and can he used to compare judgmental forecasts as well as those of formal econometric models. The next section describes the proposed methodology, and subsequent sections illustrate its use with data from the Wall Street Journal.

Methodology

Two considerations are important in evaluating the accuracy of a joint forecast of several economic variables. First, some variables are inherently less stable than others and thus are harder to forecast than others. For instance, the unemployment rate does not vary significantly from quarter to quarter. Hence, it is usually easier to predict than a highly volatile variable like GDP growth. Whatever measure is used to compare forecasts should take into account this difference in variability by penalizing forecast errors in easy-to-forecast variables more than similar size errors in hard-to-forecast variables.

Second, because many important economic variables are correlated; certain combinations of these variables are more or less likely to occur together than others.

For instance, because the CPI and short-term interest rates tend to he positively correlated, any model that reflects this underlying structure in the data should generate forecast errors in these two variables that would also likely he positively correlated. A forecast that over-estimated CPI inflation while under estimating interest rates should be penalized more than a forecast that over estimated both. That is, going out on a limb and missing on a key dimension that did not reflect the underlying data structure should be penalized more because such errors are less likely, on average.

The most common measure of variability is variance and the corresponding measure of correlation between two variables is covariance. In a multivariate setting the variance-covariance matrix can be formed with the variance of each variable along the diagonal and the covariances of the variables in the off-diagonal entries. This variance-covanance matrix [OMEGA] can be used to form a multivariate distance chi-squared statistic commonly used in statistical inference, if we are willing to assume that the forecast errors are multivariate normally distributed with mean zero. The statistic is of the form:

[chi square] = ([y.sub.t] - [y.sub.t])' [[OMEGA].sup.-1.sub.t] ([y.sub.t] - [y.sub.t])