The question
Can anything beat the historical average when forecasting the stock market? Welch and Goyal (2008) showed that most of the variables proposed in the literature, from valuation ratios to interest-rate spreads, fail to do so out of sample. One of the few exceptions is short interest: Rapach, Ringgenberg and Zhou (2016) found that the amount of stock held short forecasts the S&P 500 both in and out of sample, and argued that it may be the strongest predictor known.
Meanwhile, a large literature shows that uncertainty matters for the economy and for asset prices. Uncertainty shocks depress real activity (Bloom, 2009; Jurado, Ludvigson and Ng, 2015), and stocks’ exposure to economic uncertainty is priced in the cross-section of returns (Bali, Brown and Tang, 2017). Surprisingly little work asks the aggregate question: does financial uncertainty forecast the market as a whole? This paper does.
Measuring uncertainty
We use the financial uncertainty index of Ludvigson, Ma and Ng (2021). Rather than relying on stock market volatility, it measures how unpredictable a broad panel of 147 financial series is: valuation ratios, dividend and price growth, default and term spreads, corporate and Treasury yields, and returns on industry, size, value and momentum portfolios. Ludvigson, Ma and Ng show that shocks to financial uncertainty help drive downturns in real activity rather than simply responding to them, which sets the index apart from measures of macroeconomic or policy uncertainty.
We use two predictors: the level of the index (UF) and its monthly log change (ΔUF). The change turns out to matter most. After the index peaked in November 2008, its level stayed high for months, but its change turned negative almost immediately, an early signal that conditions were improving. The change is also only weakly correlated with every other predictor we consider (never above 0.26 in absolute value), so it carries genuinely new information.
What we do
Data. The target is the monthly log return on the value-weighted S&P 500 in excess of the one-month Treasury bill rate, from 1978:01 to 2021:12; the start date is set by the availability of short-interest data. We run a horse race between 18 predictors: the 14 Goyal–Welch variables (valuation ratios such as the dividend–price and earnings–price ratios, payout, volatility, book-to-market, net equity issuance, interest rates and spreads, and inflation); two short-interest measures, including the Rapach, Ringgenberg and Zhou series updated beyond 2014 using Compustat and CRSP; and the two uncertainty measures.
In-sample tests. Predictive regressions are notoriously tricky: when a predictor is persistent and its shocks are correlated with returns, standard t-tests are biased (Stambaugh, 1999). Alongside least squares, we therefore use the IVX-Wald approach of Kostakis, Magdalinos and Stamatogiannis (2015, 2023), which remains valid whatever the persistence of the predictor. We test one-month returns and cumulative returns over horizons of up to four years, and repeat the tests separately for 1978–2000 and 2000–2021.
Out-of-sample tests. Forecasts are produced recursively with an expanding window, using only data available at the time, and compared with the historical mean using the Campbell–Thompson out-of-sample R² and Clark–West tests. So that nothing hinges on one choice, we use initial estimation windows of 100, 200, 264 and 300 months.
Encompassing tests. Finally, we ask whether uncertainty and short interest carry the same information, by testing whether each can explain the forecast errors left by the other.
What we find
Finding 1. Uncertainty beats the standard predictors in sample.
Both uncertainty measures predict lower future returns, and both are significant under least squares and under IVX-Wald. The change in uncertainty is the standout: its in-sample R² is 6.8%, against 1.5% for the level, 1.0% for short interest and at most 0.8% for any of the 14 Goyal–Welch variables, only one of which (the default return spread) is significant at all. The coefficients stay negative and significant in both halves of the sample, with ΔUF explaining 5.6% of monthly return variation in 1978–2000 and 8.2% in 2000–2021. At longer horizons the change in uncertainty remains significant out to a year, while the level fades after about a quarter.
| Predictor | Coefficient | IVX-Wald | R² (%) |
|---|---|---|---|
| Change in uncertainty, ΔUF | −0.335 | 36.66*** | 6.83 |
| Level of uncertainty, UF | −0.032 | 8.33*** | 1.51 |
| Short interest | −0.005 | 5.81** | 1.05 |
| Default return spread | 0.003 | 4.27** | 0.82 |
| Dividend–price ratio | 0.004 | 0.83 | 0.17 |
Least-squares coefficients and R²; IVX-Wald statistics robust to predictor persistence. ** and *** denote significance at 5% and 1%. Source: Table 3 of the paper.
Finding 2. The change in uncertainty is the strongest out-of-sample predictor.
Out of sample, the change in uncertainty delivers R² values of 6.9% to 9.7% depending on the initial window, all significant at the 1% level. Short interest also works (0.9% to 2.8%), but its gains are much smaller. None of the 14 Goyal–Welch predictors beats the historical mean reliably. In a literature where an out-of-sample R² of 0.5% is considered economically relevant, these are large numbers. The advantage of ΔUF declines as the forecast horizon lengthens, but remains significant out to six months.
Notes: Campbell–Thompson out-of-sample R² relative to the historical mean, expanding window with a 264-month initial estimation period. ΔUF and short interest are significant at the 1% level (Clark–West); UF and the Goyal–Welch predictors are not. Source: Table 6.
Data table
| Out-of-sample R² | |
|---|---|
| Change in uncertainty (ΔUF) | 7.49 |
| Short interest | 2.25 |
| Level of uncertainty (UF) | 1.50 |
| Best Goyal–Welch predictor (volatility) | 0.21 |
| Median of the 14 Goyal–Welch predictors | −0.55 |
Finding 3. Uncertainty contains information that short interest does not.
The encompassing tests point one way. Both uncertainty measures significantly explain the forecast errors of the short-interest regressions (p ≤ 0.001), but short interest cannot explain the forecast errors of the uncertainty regressions. Uncertainty therefore captures something that short interest misses. Preliminary comparisons with macroeconomic, economic-policy, news-based and monetary-policy uncertainty indices suggest that none of them matches the change in financial uncertainty as a predictor, and its correlation with the changes in those measures is at most 0.41.
Why uncertainty predicts returns
Why should rising uncertainty predict lower returns? Our evidence points to a flight to safety. The same uncertainty measures that predict lower stock returns predict higher future Treasury bill yields, consistent with investors shifting wealth out of equities and into the safest assets when uncertainty rises. This fits the argument of Bali, Brown and Tang (2017) that when uncertainty is high and investors’ views diverge, the most uncertainty-averse investors cut back or stop participating in the stock market. It is also consistent with the view, set out by Rapach and Zhou (2013), that variables tracking the state of the economy should help forecast asset returns, and financial uncertainty is one of the clearest such state variables.
Because the out-of-sample exercise uses only information available at the time, the results matter in practice for anyone forecasting the equity premium, and more broadly for asset pricing and macro-financial models that depend on return forecasts.
Takeaways
- Financial uncertainty, and especially its change, forecasts the U.S. stock market. Higher uncertainty predicts lower returns over the following months.
- It beats the usual suspects. It outperforms all 14 Goyal–Welch predictors and short interest, both in and out of sample.
- The result is not an econometric artefact. It holds under persistence-robust IVX inference, across subsamples and across estimation windows.
- It is new information. Uncertainty explains what short interest misses, not the other way round.
Key references
- Bali, T. G., Brown, S. J., and Tang, Y. (2017). Is economic uncertainty priced in the cross-section of stock returns? Journal of Financial Economics, 126(3), 471–489.
- Bloom, N. (2009). The impact of uncertainty shocks. Econometrica, 77(3), 623–685.
- Campbell, J. Y., and Thompson, S. B. (2008). Predicting excess stock returns out of sample: Can anything beat the historical average? Review of Financial Studies, 21(4), 1509–1531.
- Jurado, K., Ludvigson, S. C., and Ng, S. (2015). Measuring uncertainty. American Economic Review, 105(3), 1177–1216.
- Kostakis, A., Magdalinos, T., and Stamatogiannis, M. P. (2015). Robust econometric inference for stock return predictability. Review of Financial Studies, 28(5), 1506–1553.
- Ludvigson, S. C., Ma, S., and Ng, S. (2021). Uncertainty and business cycles: Exogenous impulse or endogenous response? American Economic Journal: Macroeconomics, 13(4), 369–410.
- Rapach, D. E., Ringgenberg, M. C., and Zhou, G. (2016). Short interest and aggregate stock returns. Journal of Financial Economics, 121(1), 46–65.
- Stambaugh, R. F. (1999). Predictive regressions. Journal of Financial Economics, 54(3), 375–421.
- Welch, I., and Goyal, A. (2008). A comprehensive look at the empirical performance of equity premium prediction. Review of Financial Studies, 21(4), 1455–1508.