Applied philosophy: The Shiller P/E and S&P 500 returns

Valuations are the ultimate refuge of contrarians, and they tell us that the S&P 500 is expensive compared to historical norms even for those with a decade-long investment horizon. We consider the Shiller P/E to be one of the most reliable guides to long-term forward returns. We use it as a case study to approximate expected returns on the S&P 500 for the next 10 years. Should we ignore our contrarian instincts? 

The lights are down, the tree has gone. The Christmas cheer faded a long time ago. We are staring at the grey skies and the darkness envelops us in its cold embrace. In the bleak midwinter only our questions remain: what does this year have in store for us? 

We outlined our expectations and forecasts in November 2023 feeling hopeful about risk assets. We were moderately positive about even the S&P 500, despite its rich valuations. With the benefit of hindsight our forecast turned out to be too conservative in the short term. Perhaps that is a disadvantage of being early publishing our 2024 Outlook, perhaps we were not optimistic enough. 

How could we have been more optimistic? Valuations told us the same story then that they are telling now: US equities are expensive when compared to historical norms, no matter the valuation metric used. Our favoured measure, the cyclically-adjusted price/earnings (CAPE) ratio stood at 35.7x at the end of December 2023 (using the Datastream US Total Market Index), while the Shiller P/E (its inflation-adjusted equivalent) was at 31.3x, according to our estimates (Figure 1). 

To put that into perspective, these ratios were higher only during extreme market events: around the “tech bubble” between 1998 and 2002 and the recent post-pandemic market peak between 2020 and 2022. Since January 1983 (the first available data point) the CAPE was higher only 12% and the Shiller P/E 15% of the time (using monthly data).  

Of course, a logical argument would be that we are using the wrong tools to try to determine equity returns (or at least their direction) over a relatively short investment horizon. Indeed, the relationship between our CAPE and 1-year forward returns on the S&P 500 index between 1983 and 2022 is practically random. However, this predictive power improves using 10-year forward returns to an R-squared of 0.78 (between 1983 and 2013). 

Interestingly, using the longer history of the Shiller P/E – available from 1881 – does not strengthen the relationship: the R-squared falls to 0.11. It seems that this predictive power may be a recent phenomenon. However, if we use the 30-year period before 1983 (starting in 1953), the R-squared jumps to 0.78. If we combine the two periods into 1953-2013 the R-squared drops to 0.42 suggesting that a structural split happened in the mid-1980s, which changed the relationship.

Even a quick glance reveals that valuations seem to have shifted higher in the 1980s: from an average of 15.6x between 1953 and 1983 to 24.2x between 1983 and 2023. What were considered extreme values in the first 100 years of the history of the Shiller P/E reached only in 1929, became more “normal”. 

When valuations change, our first suspect is usually the discount rate. Indeed, while 10-year Treasury yields rose from 2.3% in 1954 to around 15% in 1981 following the path of inflation, they declined from that level until our present day during the “Great Moderation” of the 1990s and deleveraging of the post-GFC period.  

This implies that, assuming no structural change in the direction of inflation and interest rates, we ought to rely on the post-1983 model to predict long-term returns on the S&P 500 (Figure 4). Not to mention that on the pre-1983 model the current Shiller P/E would be literally “off the charts” (the peak between 1953 and 1983 was 23.9x). Also, at this point, we think the probability of inflation reaccelerating is low over both the short and medium term. Without that, we do not think valuations would revert to pre-1980s levels. 

Figure 2 illustrates the relationship between the Shiller P/E and 10-year annualised forward returns on the S&P 500 since 1983. If that persists, our model suggests that annualised returns over the next 10 years could be around 3.5%, below the long-term average of 4.7% (since 1871 based on Shiller data), though significantly higher than the -10% returns implied by Figure 1. 

Assuming the S&P 500 dividend yield stays close to the 10-year average of 1.8%, a 3.5% capital return would imply a total return of around 5.3%. This appears disappointing compared to the 9.2% annualised total return since 1871 and the 12% seen in the last ten years. If the dividend yield stayed close to that 10-year average, the increase in future capital returns needed to generate a 9.2% total return would require a 24% fall in the S&P 500 from the December 2023 average (taking the Shiller P/E to 23.9x).

We think that would either necessitate a recession, which is outside our base case for now, or could be the result of higher real treasury yields on equity discount factors. On the other hand, the rise in earnings required to bring the Shiller P/E down to that level would also be immense (about 430% with a one-year timeframe, and a doubling followed by a 50% rise if we model it over two years). Perhaps we must simply accept lower-than-average returns. 

Or we could look outside the US. Within equities no other region seems obviously overvalued based on CAPE, their valuations all being below their respective historical averages (Figure 3). At the same time, a comparison of yields across asset classes reveals that only Japanese government and corporate bonds have less attractive absolute valuations as of 31 December 2023. In light of these, we see no reason to change our allocation to US equities, especially after strong returns in 2023 mainly driven by a small group of technology stocks. We remain Underweight (see Figure 8).