Your AI Backtest Is Lying to You: What 1,600 Failed Hypotheses Taught Us
We scored 165,182 earnings-call transcripts (1990–2026) with a large language model and tested roughly 1,600 yes/no hypotheses against 12-month excess returns, under split-half validation, placebo questions, shuffled-label nulls, temporal gates, and pre-registration. One combination — a clean call with growing customer prepayments (“C1”) — passed everything: positive in six separate backtest years with zero shuffled-null survivors. We then froze a 144-name forward watchlist and waited twelve months. C1 returned a median of −12.4% versus SPY (35.4% beat rate ≈ base rate). The autopsy revealed the cause: the model can “predict” historical outcomes 66–72% of the time from the ticker and date alone, with no transcript — training-data memory that inflates any backtest run on public companies. We publish the full anatomy of the failure, and the one signal that survived.
- Roughly 1,600 hypotheses were tested against 12-month returns; none produced a directional edge that survived out-of-sample.
- The best candidate passed six backtest years (+6% to +11% median excess) and every statistical control, then delivered −12.4% median in a frozen forward test of 130 graded names.
- Stripped of transcripts entirely, the model still “predicted” whether historical stocks beat the market 66–72% of the time from ticker + date — direct evidence of training-data memorization.
- Forward performance decayed sharply by call month (+1.9% for Aug-2024 calls to −41.9% for May-2025 calls), the signature of an edge that exists only where the model’s memory reaches.
- The single surviving signal predicts the size of future moves (|reaction| + pre-call volatility, t ≈ 24), not their direction.
1The experiment
The setup was simple to state and expensive to run: have a language model read every earnings-call transcript we could get — 165,182 calls across 9,117 companies — and answer the same battery of questions about each one. Is management evasive? Is deferred revenue growing? Would a skeptical short-seller be reassured? Then test whether any answer, or combination of answers, predicted the stock’s next twelve months against SPY.
Because we knew most “findings” in this genre are artifacts, every test ran inside an apparatus built to kill false positives: random split-half validation (a signal must work on both halves), placebo questions in every bank (“Does anyone say thank you?”), shuffled-return nulls (the identical search on scrambled outcomes must find nothing), temporal gates (it must work in 2016–18 and 2022–24), and COVID-era exclusion. About 1,600 hypotheses went through: standard analyst questions, forensic-linguistics angles, and hundreds of deliberately strange ones generated through rotating personas — a bankruptcy trustee, a poker professional, a customs inspector.
2The survivor
Combination mining surfaced one robust pattern we call C1: the model answered NO to “is there something a critic could fairly use against the company?” and YES to “are deferred revenue or customer prepayments growing?” — a clean call from a company its customers pay in advance. C1’s median 12-month excess return was positive in six separate backtest years spanning two market regimes. Three full shuffled-null repetitions of the entire search produced zero comparable survivors. It passed a pre-registered 2022/2023 holdout. When we controlled for company fame, the edge held in every fame tercile (+10 to +20 points over matched baselines). By every standard the quant literature offers, C1 was real.
| Control | Requirement | Result |
|---|---|---|
| Split-half validation | Positive median on both random halves | Passed |
| Temporal gate | Positive in 2016–18 and 2022–24 separately | Passed |
| Shuffled-return null ×3 | Zero survivors on scrambled outcomes | Passed (0 hits) |
| Pre-registered holdout | Positive on untouched 2022/2023 data | Passed |
| Fame control | Edge present in every fame tercile | Passed |
| Forward test, real time | Beat SPY on a frozen watchlist | Failed |
3The forward test
In the only test that cannot be gamed, we froze a watchlist of 144 C1 names from calls dated August 2024 onward and did nothing for twelve months. Then we graded it with fresh market prices: 130 names had complete windows. Median excess return: −12.43%. Beat rate: 35.4% — indistinguishable from the base rate of all stocks in that period. Every refinement that had looked even better in backtests inverted harder: the “C1 plus no growing backlog” variant promised +19.3% and delivered −35.6%; the founder-led tier delivered −30.7% (n = 38).
4The smoking gun: your model remembers
Why did six years of controlled backtests point one way and reality the other? The decisive experiment removed the transcripts entirely. We gave the model nothing but a ticker and a date and asked it to guess whether that stock beat the market over the following year. It was right 66–72% of the time — far above chance, on information it could only possess by having absorbed the actual financial history of these companies during training.
That means any LLM judgment about a historical public company — “does this call read like a winner?” — is quietly leaking the answer key. The model isn’t reading the transcript; it is, in part, remembering the outcome. Controls built for classical overfitting (split-half, shuffles, holdouts drawn from the same history) cannot catch this, because the contamination lives inside the model, not in the researcher’s pipeline. The only clean test is one the model’s training could not have seen: a frozen list, graded by the calendar. Ours failed, and Figure 1 shows the memory gradient directly — performance decays month by month as calls move past the training data.
If you are evaluating any “GPT picks stocks” result — your own, a vendor’s, or a paper’s — and its evidence is historical, this failure mode applies to it.
5What actually survived
One signal passed out-of-sample testing, and it is not directional: the size of a stock’s next move is predictable from the size of its immediate reaction to the call plus its pre-call volatility (linear fit on 2016–2023 data, t ≈ 24). Direction — up or down — was never predictable in anything we tested once contamination was removed. This is why every page on this site publishes call-quality grades and an expected move (±%), and no buy/sell calls: it is the strongest claim our data honestly supports.
6How to run a test your model can’t cheat
What we now require of any result before believing it: (1) a forward test on a frozen, timestamped list — the only evidence that counts; (2) a ticker-and-date-only ablation — if the model scores above chance without the document, the historical benchmark is contaminated; (3) placebo questions and shuffled-outcome nulls run through the identical pipeline; (4) separate claims about magnitude from claims about direction; and (5) publication of the failures alongside the survivors. Our full apparatus, and the C1 pattern’s live page, are public: the methodology documents everything, and the question bank lists all 367 archived hypotheses with every company that matched.