More Mentions, Less Information? Diagnosing Identity and Alignment in Reddit Embeddings for Volatility Forecasting
Zhiyu Zheng
Abstract
Since the 2021 GameStop short squeeze, r/WallStreetBets (WSB) has been associated with episodes of extreme volatility, and its text is increasingly used to forecast volatility. A model using only large language model (LLM) embeddings of WSB posts attains an out-of-sample $R^2$ of 0.13 for log realized variance. That number shows predictive power, but it does not reveal where the information comes from or how much the text adds beyond a firm's own volatility history. We answer these questions with a diagnostic protocol that nests embeddings on a heterogeneous autoregressive (HAR) benchmark, compares them with text-free firm-identity controls, shuffles the day-to-text pairing within firm-years to test day-specific alignment, and checks whether gains persist across periods and coverage groups. We apply the protocol to frozen LLM embeddings in a panel of 202,145 mentioned stock-days, using retrospective rolling out-of-sample tests over 2015-2022. The results indicate that most of the standalone fit reflects firm or firm-year information: on firms seen in training, a text-free firm baseline reaches $R^2 = 0.32$, shuffling preserves 89-92% of the standalone fit, and masking ticker strings reduces standalone $R^2$ to -0.005. Adding embeddings to HAR reduces equal-date MSE by 1.22% (95% CI [0.94, 1.49]). This small but statistically significant gain survives firm controls and ticker masking, but is largely eliminated by alignment shuffling. The gain is concentrated in 2015-2019 and is not distinguishable from zero in 2020-2022. Expanding coverage by 48% through off-hours and weekend posts adds stock-days with no reliable positive increment. More mentions therefore do not imply more incremental forecasting information.
Read the AI summary, key takeaways and discussion on WOBR Quant Research.