Does the thesis engine actually work?
We replayed it across 1232 company-years of SEC filings, scoring each one using only what was public on the day it was filed, then checked what the company reported next. Here is the result, including the calls it got wrong.
Every figure below has to beat this. A signal that flags decline half the time is worth nothing if decline happens half the time anyway.
| SCORE AT THE TIME | N | GOT WORSE | VS BASE | NEXT-YR REV |
|---|---|---|---|---|
| On Track (75-100) | 730 | 31.9% | -4.8pp | 5.9% |
| Watch (50-74) | 241 | 43.2% | +6.5pp | 3.1% |
| Broken (0-49) | 182 | 47.3% | +10.6pp | 1.8% |
The three bands separate in the right order, on both measures. A Broken score was followed by further deterioration 47.3% of the time against a 36.7% base rate, and the median revenue growth reported the year after falls steadily across the bands — 5.9% for On Track, 3.1% for Watch, 1.8% for Broken.
An earlier version of this page reported almost no signal, and that is worth explaining rather than quietly replacing. It was measured over 32 mega-caps — companies that nearly all went on to do well, where a bad year is usually a dip that mean-reverts. Widening the universe to include airlines, cruise lines, department stores, steel, oil services and legacy media moved the base rate from 30.4% to 36.7% and the Broken band's edge from +5.8pp to 10.6pp. The engine did not improve. The first measurement had been taken on a sample where almost nothing broke.
It still describes rather than forecasts: it reports what the last filed year showed, with the figures attached. A Broken score is not a prediction that a company will keep declining — 96 of these calls were followed by improvement.
How it was tested
The hard part of a backtest is not the maths, it is refusing to use information that did not exist yet. Apple's 2024 fiscal year ended on 28 September and was filed on 1 November; scoring a thesis in October using those figures is not slightly optimistic, it is meaningless, because the prediction is drawn from the answer.
SEC's filings carry a filing date on every reported figure, so each score here was built only from values already public on that date. Restatements are handled the same way: read from an early date you see the figure as first reported, from a later one the correction — exactly what a person reading at those dates would have seen.
The live engine is called, not reimplemented. A backtest of a copy of the rules only proves things about the copy.
What this test cannot support
- The universe is still large US companies. 83 names, now including businesses that genuinely declined — but every one of them was big enough to be worth listing, so outright failures and small caps are absent. Widening it once already changed the answer materially, which is reason to treat this number as provisional.
- Four of seven checks. Some inputs the live engine uses cannot be reconstructed historically. The engine omits a check rather than guessing, so replayed scores rest on fewer signals than live ones.
- No significance claimed. 1153 observations, and companies in one market move together, so these are descriptive statements about this sample — not statistics you should extrapolate.
- Business outcomes, not returns. We measured whether the business got worse, which is what the checks read. Nothing here is a claim about share prices.
The calls it got wrong (96)
Scored Broken, and the business improved the following year. All 96 are counted in the figures above; the 40 largest are listed here, ordered by how wrong they were. A scorecard showing only its hits is an advertisement.
Why publish this
Because the alternative is asking you to trust a number because it looks rigorous. The engine reads filings and reports what changed, with the figures attached, faster than waiting for someone to write it up — and the Intel case shows that working: it scored 0 on the day the filing landed, and revenue then fell 14% with margins down 22 points. What it does not do is tell you what happens next, and we would rather show you the evidence for that than let the interface imply otherwise.
Educational information only, not investment advice. Past figures do not indicate future results. Snapshot generated 2026-08-12 from SEC EDGAR filings.