Macroeconomic uncertainty and forecasting macroeconomic aggregates

2.1 The Bayesian VAR

The VAR(p) is specified as follows:

where y_t and c are n × 1 vectors of endogenous variables and intercept terms, respectively. ε _t denotes the vector of normally distributed residuals. A_j are n × n matrices of coefficients with j = 1 , … , p . I employ Bayesian estimation techniques to estimate the model. Specifically, I use the Minnesota prior developed by Litterman (1986), which assumes that every economic time series can be sufficiently described by a random walk with drift. Thus, the prior shrinks all coefficients on the main diagonal of A ₁ towards one while the remaining coefficients are shrunk towards zero. Moreover, the classical Minnesota prior assumes a diagonal covariance matrix of the residuals. In the following, I use the generalized version of the classical Minnesota prior provided by Kadiyala and Karlsson (1997), which allows for a non-diagonal residual covariance matrix while retaining the idea of the Minnesota prior described above. As demonstrated by Bańbura, Giannone, and Reichlin (2010), using a normal-inverse Wishart prior generates accurate forecasts despite the additional parameters to be estimated. In addition, I follow Doan, Litterman, and Sims (1984) as well as Sims (1993) by implementing the “sum-of-coefficients” and “co-persistence” prior. The former accounts for unit roots in the data; the latter introduces beliefs on cointegration relations among the series. Each prior is implemented using dummy observations. For details regarding the prior implementation and the estimation procedure, see Appendix A.

2.2 The Bayesian threshold VAR

The threshold VAR is defined as follows:

where y_t is the vector of endogenous variables. Contrary to the linear VAR in (1), the intercept terms c_j , the matrices of coefficients A_j , and the variance of the residuals Ω _i are state-dependent, with j ∈ {1, 2}. The regime prevailing in period t depends on whether the level of the threshold variable, r, in period t − d is below/above a latent threshold level, r ^∗. This mechanism allows for different model dynamics depending on the respective regime. As in the previous section, I use natural conjugate priors for the VAR coefficients and implement the priors using dummy observations. While it is, in general, possible to apply different prior information for the two regimes, I follow Alessandri and Mumtaz (2017) and impose the same prior beliefs for both regimes. On the one hand, this specification shrinks both regimes towards same model dynamics. On the other hand, it allows to let the data decide on how important and pronounced different model dynamics are. I follow Chen and Lee (1995) for the threshold level and the delay coefficient:

where d max = 8 denotes the maximal delay and r ¯ is the sample average of r. Given the variability of the macroeconomic uncertainty index (see Figure 1 in Section 4), I set ν = 1 to keep the prior for the threshold level loose. To simulate the posterior distribution of the model’s parameters, I apply the Gibbs sampler introduced by Chen and Lee (1995), which works as follows:

1. At iteration k = 1 set starting values for d k = d 0 , r ∗ , k = r 0 .

2. Draw Σ j k | d k , r ∗ , k , Λ j k , y j , and β j k | d , r ∗ , k , Λ j k , Σ j k , y j from their posteriors given by (21).

3. Draw a candidate value for r ^∗ by: r ∗ = r ∗ , k − 1 + Φ ϵ     with:  ϵ ∼ N ( 0 , 1 ) and Φ is a scaling factor ensuring an acceptance rate of about 20%.

4. Accept the draw with probability

   (6) p k = min { 1 , p ( Y t | r ∗ , θ ) p ( Y t | r ¯ k − 1 , θ ) }

   where p ( ⋅ ) denotes the posterior density given all other parameters of the model.

5. Draw d from

   (7) p ( d = i | Y t , θ ) = p ( Y t | d , θ ) ∑ d = 1 d 0 p ( Y t | d , θ ) for:  i = 1 , … , d max .

6. Generate e j , T + 1 , … , e j , T + h from ϵ j , t   ∼ N ( 0 , Σ j k ) and compute h-step-ahead forecasts recursively by iterating (2) and (3) h periods into the future.

7. Redo until k = D + R.

I employ 25,000 iterations of the Gibbs sampler and discard the first 20,000 as burn-ins. Convergence statistics for the algorithm are provided in Appendix B.

The key element of this model is the threshold variable r, which governs the regime dependency. Different specifications for r are proposed in the literature. Caggiano, Castelnuovo, and Figueres (2017) and Caggiano, Castelnuovo, and Groshenny (2014) argue that recessions are particularly informative regarding the identification of uncertainty shocks. These studies follow Auerbach and Gorodnichenko (2012) and use a moving average of GDP growth rates as threshold variable. Other studies emphasize the importance of the uncertainty proxy itself and condition on either the historic change (for example, Foerster 2014; Henzel and Rengel 2017) or the historic level of the uncertainty proxy (for example, Berg 2017a; Jones and Enders 2016). Since this paper aims at identifying uncertainty regimes, I follow the latter and specify r as the level of the uncertainty indicator.

However, nowadays there are various uncertainty proxies available, for example, stock market volatility (Bloom 2009), newspaper-based indices (Baker et al., 2016), firm-level data-based indices (Bachmann, Elstner and Sims 2013), indices based on macroeconomic forecast errors (Rossi and Sekhposyan 2015), and indices based on the residuals from factor augmented regressions (Jurado, Ludvigson, and Ng 2015). I choose the macroeconomic uncertainty index provided by Jurado, Ludvigson, and Ng (2015), a choice motivated by two factors. First, this proxy defines uncertainty in terms of the variation in the unforecastable component of macroeconomic variables and not in terms of the variables’ raw volatility.^[4] Second, and in contrast to other measures, it is based on a large number of economic indicators and, hence, should represent an aggregate uncertainty factor that affects many series, sectors, or markets (Jurado, Ludvigson, and Ng 2015).^[5]

I recursively construct the index to avoid that the index at a given point in time includes information that would not be available to the forecaster at this moment. As already pointed out by Jurado et al. (2015), the indices based on both in-sample forecasts and out-of-sample forecasts are highly correlated.

5.1 Forecast metrics

I measure point forecast accuracy using root mean squared errors:

where y ¯ i , T | T − h j and y_{i, T} denote the mean of the models’ predictive density and the corresponding realization. The RMSFE is only useful in assessing the accuracy of a model compared to that of other models. Therefore, I report the RMSFEs relative to a benchmark model ( RMSFE i , B h ):

I apply the test provided by Diebold and Mariano (1995) adjusted for the small-sample correction of Harvey, Leybourne, and Newbold (1997) to gauge whether the point forecasts are significantly different from each other.

To take into account the uncertainty around the point estimate, additionally, I evaluate the predictive densities. Specifically, I apply the average log predictive score, which has become a commonly accepted tool for comparing the forecast performance of different models (see Clark 2012; Geweke and Amisano 2010, among others). It is defined as the logarithm of the predictive density evaluated at the realized value:

The predictive density, p ( y t + h | j ) , is obtained by applying a kernel estimator on the forecast sample.^[11] Hence, if the competing model has a lower log score than the benchmark, its forecasts are closer to the realizations with a higher probability. As for the RMSFE, the log scores are not informative on their own, which is why I report them relative to the benchmark model ( L S B , i h ):

Furthermore, I evaluate the predictive densities in terms of the (average) continuous ranked probability score (CRPS). As argued by, for instance, Gneiting and Raftery (2007), the CRPS has two advantages compared to the log scores. First, it can be reported in the same units as the respective variable and therefore facilitates a direct comparison of deterministic and probabilistic forecasts. Second, in contrast to log scores, CRPSs are both less sensitive to extreme outcomes and better able to assess forecasts close but not equal to the realization. I follow Gneiting and Ranjan (2011) and express the CRPS in terms of a score function:

where QS α ( P t ( α ) − 1 , y i , t ) = 2 ( I { y i , t < P t ( α ) − 1 } − α ) ( P t ( α ) − 1 − y i , t ) is the quantile score for forecast quantile P t ( α ) − 1 at level 0 < α < 1. I { y i , t < P t ( α ) − 1 } is an indicator function taking the value one if y i , t < P t ( α ) − 1 and zero otherwise. ν(α) is a weighting function. Applying a uniform weighting scheme (ν(α) = 1), and dividing by the number of generated densities yields the average CRPS:

To compute this expression, P(⋅) is approximated by the empirical distribution of forecasts and the integral is calculated numerically.^[12] According to this definition, a lower CRPS implies that the predictive density is more closely distributed to the actual density. As for the log scores, I report the CRPS in terms of the average across all evaluation periods and relative to the benchmark model. To provide a rough gauge on whether these scores are significantly different from the benchmark, I follow D’Agostino, Gambetti, and Giannone (2013) by regressing the differences between the scores of each model and the benchmark on a constant. A t-test with Newey-West standard errors on the constant indicates whether these average differences are significantly different from zero.

Finally, I compute probability integral transforms (PITs) that are defined as the CDF corresponding to the predictive density evaluated at the respective realizations:

Thus, with regard to the respective predictive density, the PIT denotes the probability that a forecast is less or equal to the realization. For example, a realization that corresponds to the 10^th percentile receives a PIT of 0.1. Hence, if the predictive densities match the true densities, the PITs are uniformly distributed over the unit interval. To assess the accuracy of the predictive density according to the PIT, I divide the unit interval into k equally sized bins and count the number of PITs in each bin. If the predictive density equals the actual density, each bin contains N/k observations. I set k = 10, implying that each bin accounts for 10% of the probability mass. Moreover, I follow Rossi and Sekhposyan (2014) and compute 90% confidence bands by using a normal approximation to gauge significant deviations from uniformity.

5.2 Point forecasts

Table 2 summarizes the results of the forecast evaluation based on MFEs and RMSFEs. The dimension for the measures is percentage points. While the models provide forecasts for each variable in the dataset, for the sake of brevity, I present results only for the variables depicted in Section 4, namely, inflation (measured in terms of the GDP deflator growth), GDP growth, consumption growth, investment growth, the unemployment rate, and the federal funds rate.^[13] Let us begin by analyzing the results for MFE presented in the left panel of Table 2. The table shows that the benchmark VAR on average and in most cases overestimates the realization. Inflation for the next quarter, for instance, is overpredicted by 0.14 annualized percentage points. Adding the uncertainty index to the otherwise standard VAR (VAR^U) tends to increase this bias except for the unemployment rate and for investment growth. In the latter case, the MFE is on average over all horizons about one percentage point smaller. The MFEs of the threshold VAR (T-VAR) are distinct from the former ones. First, compared to the linear models, the MFEs from the T-VAR are in most cases larger. Only for certain variables and horizons (for example, output growth at h = 3) reductions are detectable. Thus, identifying uncertainty regimes seems to be less fruitful for generating well-calibrated predictive means. Second, while the linear models consistently underpredict unemployment and overpredict the federal funds rate, the T-VAR overpredicts unemployment and underpredicts the federal funds rate. The latter result stems from the fact that the T-VAR predicts federal funds rate values below zero even though the federal funds rate is fixed at its lower bound.^[14] Overall, the evaluation of the MFEs, thus, provides only little evidence in favor of both the VAR^U and the T-VAR. In fact, the benchmark model provides very competitive MFEs and in some cases outperforms the remaining models.

The right panel of Table 2 depicts the results for the RMSFE. With respect to the benchmark model, the RMSFEs are reported in absolute terms, while the remaining specifications are reported as ratios relative to the benchmark model, i.e. a figure below unity indicates that the model outperforms the benchmark specification. The differences between the VAR and the VAR^U are again very small and in most cases insignificant, suggesting that the uncertainty index has on average only marginal impact on the forecast performance in a linear setting. Only for the federal funds rate, the VAR^U provides significantly smaller RMSFEs. The results for the threshold VAR are mixed. In most cases, the latter is outperformed by its linear counterparts, implying that identifying uncertainty regimes is not beneficial with regard to point forecasting. The worst relative performance is obtained for inflation forecasts. Moreover, neither for GDP growth nor for investment or consumption growth, the T-VAR delivers a reduction in RMSFEs.

While for the former indicators regime-dependency apparently does not pay off, unemployment and interest rate forecasts benefit significantly. Regarding the federal funds rate at the one and two-quarter ahead horizons, the T-VAR’s forecasts are on average 14% and 8% more precise, respectively, while with regard to the unemployment rate forecast, accuracy increased by 6% and 7% for these horizons. These results are particularly appealing since labor market variables possess an especially strong regime dependency with regard to uncertainty (see Figure 2). In addition, these findings underpin the results of Alessandri and Mumtaz (2017) and Barnett, Mumtaz, and Theodoridis (2014). While the former demonstrates that regime-dependent VARs are inferior to linear VARs and VARs with time-varying parameters with regard to GDP growth and inflation, the latter provides evidence that financial variables particularly benefit from regime dependency. Thus, it is suggested that for activity variables there is, if any, only gradual structural change, which cannot be covered by a threshold VAR, while for labor market and financial variables the structural shift is more abrupt and thus can be captured by the T-VAR.

Figure 3 explores the models’ forecast performance over time. To this end, I calculate four-quarter moving averages of the MFE (upper panel) and relative RMSFE (lower panel) for one-quarter-ahead forecasts of the unemployment rate (left column) and federal funds rate (right column). Evidently, the degree of predictability varies substantially over time. Regarding unemployment rate forecasts, the VAR^U and the T-VAR work particularly well during the Great Recession and the subsequent recovery when uncertainty was high. In the remaining periods, when uncertainty was rather low, the forecast performance is very similar (VAR^U) or even worse (T-VAR) compared to the linear VAR, suggesting that uncertainty is especially relevant when it is high. A similar pattern arises for the federal funds rate. The largest gains in forecast accuracy are obtained from 2008–2012 when uncertainty was high. However, in contrast to the unemployment rate, federal funds rate forecasts are also more precise from 2013 to 2016, while the short hike of the federal funds rate in 2012 is captured best by the linear VAR; both the VAR^U and the T-VAR strongly overestimate the actual increase. Overall, the results suggest that including information on economic uncertainty can improve point forecast accuracy for some variables and for short horizons, with the largest gains during episodes of high uncertainty.

Figure 3:

Forecast performance over time – point forecasts.

Figure displays mean forecast errors (upper panel) and relative root mean squared forecast errors (bottom panel) computed as a four-quarter moving average over the forecast sample for unemployment and federal funds rate forecasts.

5.3 Density forecasts

Subsequently, we evaluate the models’ forecasts with respect to the predictive densities. Thus, apart from the predictive mean evaluated above, the variances have to be precisely estimated as well to ensure an accurate predictive density. Table 3 sets out the results for the CRPS and the LS. First, we consider the results for the LS, which are reported in levels for the benchmark and in differences for the remaining models. Positive differences indicate that the respective model outperforms the benchmark. With regard to the linear models, the LS provide a pattern similar to that in the previous section. Again, the differences between both models are rather small, indicating that the marginal impact of the uncertainty index in a linear setting is on average almost negligible. However, in some cases, already the linear VAR using additional information on economic uncertainty provides significantly better (lower) LS. Turning to the T-VAR reveals that for medium- to long-term forecasts, controlling for regime-dependency with respect to uncertainty leads to considerably less accurate predictive densities. Regarding short-term forecasts, though, the T-VAR provides, for most variables, remarkably better log scores, with the largest improvements obtained for the activity variables. For instance, the LS for output growth at h = 1 is 19% lower than the benchmark’s score. Hence, while the T-VAR is inferior in generating precise point forecasts for the activity variables, it is superior in computing the complete predictive distribution of these indicators and thus is better suited for describing the uncertainty around the point estimate.

In total, the CRPS underpin the findings of the LS. However, there are noteworthy differences in regard to the unemployment rate. While according to the LS the predictive distributions of the T-VAR are virtually identical to the ones of the benchmark, according to the CRPS, the T-VAR provides significantly more accurate densities. For instance, the one-quarter-ahead CRPS for the unemployment rate is 16% lower than the benchmark’s CRPS while the average log score is virtually identical. The latter suggests that the log scores regarding the unemployment forecasts are partly distorted by outliers. Overall, the evaluation of both the LS and CRPS underpins the findings of previous studies demonstrating that nonlinearity is particularly useful in calibrating accurate predictive densities (see Chiu, Mumtaz, and Pintér 2017; Groen, Paap, and Ravazzolo 2013; Huber 2016, among others). However, while the former studies mainly focus on forecasting output, inflation, and interest rates, this paper shows that unemployment rate forecasts also benefit significantly. Figure 4 presents evidence on time-varying predictability. Similar to Figure 3, the T-VAR provides more accurate densities during the Great Recession and the subsequent recovery.

Between 2011 and the end of 2013, the T-VAR’s entire forecast distribution is stretched by a few forecasts far away from the realizations, which leads to low log scores. Since the CRPS is better able to reward the observations close to the realization and is more robust to outliers, according to the CRPS, the T-VAR provides more precise densities even for this period and thus for almost the entire evaluation period. For the federal funds rate, the picture is more clear-cut. The LS indicate that the T-VAR is superior at the beginning of the Great Recession, but the CRPS display more accurate predictive densities for almost the entire sample. As for the unemployment rate, the T-VAR provides the best relative forecast performance during the Great Recession and the subsequent recovery when economic uncertainty was very high. In total, Figure 4 provides evidence for important changes in the predictive ability of the models.

Figure 4:

Forecast performance over time – density forecasts.

The figure displays log scores (upper panel) and continuous ranked probability scores (bottom panel) computed as a four-quarter moving average over the forecast sample for unemployment and federal funds rate forecasts.

Finally, I compute PITs to gauge the calibration of the predictive densities. Figure 5 facilitates a graphical inspection of the PITs and shows that the predictive densities look similar for the different models.^[15] As I computed 50 forecasts for each horizon, each bin in Figure 5 should contain five observations (depicted by the solid black line) to ensure uniformity. Thus, the closer the histograms are to the solid black line, the more accurate are the predictive densities. In the case of inflation, output, investment, and consumption, the PITs appear hump-shaped, with significant departures from uniformity. In fact, the models assign too much probability to the center of the distribution with too many PIT-values around 0.5. The latter indicates that the kurtosis of predictive densities at each horizon and recursion is higher than the kurtosis of true density, which implies that the models overestimate the actual uncertainty around the point estimate. This pattern is frequently found in the VAR forecasting literature – see, for example, Bekiros and Paccagnini (2015) and Rossi and Sekhposyan (2014) or Gerard and Nimark (2008) – and can be caused by a too dense parametrization of the model.^[16] With regard to one-quarter-ahead forecasts (blue bars), the T-VAR mitigates this issue by generating more forecasts that correspond to the lower percentiles of the actual distribution and thus provides a better description of the data. At higher horizons, however, the densities are again too wide. Regarding unemployment rate forecasts, the PITs of each model are closer to uniformity for h = 1 and h = 2; both the lower and the upper percentiles of the actual distribution are captured by the models. At the remaining horizons, the models again overestimate the actual uncertainty. The PITs for the interest rate forecasts appear to be right-skewed, and thus missing the left tail of the actual distribution. The latter stems from the phase of extraordinary low interest rates at the end of the sample, which are barley captured by the models. Only the VAR^U is able to generate forecasts corresponding to the lower percentiles. Jointly with the results from Table 3, the evaluation of the PITs suggests that estimating regime-dependent covariance matrices with respect to the prevailing level of uncertainty helps calibrating accurate predictive densities.

Figure 5:

Probability integral transform (PITs).

The figure displays the CDF of the probability integral transforms (PITs). Solid and dashed black lines denote uniformity and 90% confidence bands, respectively.

5.4 The role of financial uncertainty

While the previous sections have demonstrated that using information on macroeconomic uncertainty can be helpful regarding economic forecasting, they excluded the question whether fluctuations in uncertainty can be rather specified as an exogenous force, driving the business cycle, or an endogenous response to fluctuation in the business cycle. Indeed, as shown by Ludvigson, Ma, and Ng (2019), macroeconomic uncertainty is best characterized as an endogenous response to the business cycle, while a measure of financial uncertainty is more likely to be an exogenous impulse to the business cycle. In the following, therefore I repeat the forecast experiment and substitute the macro uncertainty index for the financial uncertainty index provided by Ludvigson, Ma, and Ng (2019).^[17] The results from this analysis are depicted in Table 4 and suggest that – from a forecasting perspective – it is less important to differentiate between macro and financial uncertainty. In a linear framework, the impact of financial uncertainty on point forecast accuracy is only minor; the RMSES of VAR with financial uncertainty (VAR^FU) are rather close to the ones of the VAR^U and the VAR without information on uncertainty. Using the threshold VAR with financial uncertainty (T-VAR^FU) improves on the linear VAR. Compared with the T-VAR from Section 5.2, however, it provides a very similar forecast performance. Only for the interest rate and for investment growth, the T-VAR^FU provides slightly more precise forecasts in the short-run. This result might be driven by two factors. First, as suggested by the previous sections, gains in forecast accuracy are mainly driven by identifying uncertainty regimes, but not by the additional information on uncertainty. However, the T-VAR^FU identifies uncertainty regimes that are broadly in line with those of the T-VAR.^[18] Second, both series are, partly, based on the same information and display significant comovement (the correlation between both series is roughly 0.60 in the final recursion). Thus, both the models’ regimes and the information on uncertainty are similar, leading to similar forecast performance.

Figure 6:

Estimated uncertainty regimes.

Shaded areas correspond to NBER recessions. The Dashed-dotted line refers to the high uncertainty regime, i.e. the median estimate of ( 1 − S t ) from (2) and (3).