The scoreboard
Calibration
Every forecast publishes an 80% interval before resolution and is graded against the official first print when it lands. This page is computed from the same records that back log.json and reward.json; the daily pre-registration chain lives in the public records repository.
Scoring methodology v5 (2026-07-10): headline numbers count only witness-verified scores. A run enters the headline when its sealed custody root was externally witnessed — an RFC 3161 timestamp in the witnessed chronology extracted from the public record chain — before the observation, its custody inventory is complete and headline-eligible, and its recorded run time precedes the observation (sub-day ordering trusted only for explicit UTC-offset timestamps). A claimed timestamp alone never enters the headline: scores whose chronology rests on claimed times stay published in log.json, flagged claimed-time-verified, outside the official numbers, alongside unverified and violated legacy runs. CRPS is normalized only by same-series ledger dispersion frozen at target registration; scores without three pre-cutoff ledger observations publish raw CRPS and stay out of normalized means and rewards. The agent-versus-persistence headline is a per-target RAW CRPS ratio against the paired ledger baseline, which needs no scale at all — nothing a forecast authors can move its denominator.
Scores whose runs predate the witnessed custody chain: their recorded timestamps order them before their outcomes, but no independent timestamp proves it, so they sit one tier below the headline — published and flagged in log.json, never deleted. The headline starts at zero by construction and fills as custody-v2 runs resolve against official prints.
The calibration curve
Each score records the forecast distribution evaluated at the official print (its probability integral transform), so coverage is checkable at every stated interval, not just the elicited 80%. A calibrated forecaster tracks the diagonal: above it means intervals are too wide, below it too narrow. The PIT histograms show the same thing distributionally — a calibrated forecaster fills each bin equally; a U shape is overconfidence, a central hump underconfidence. The curve reads coverage off each forecast's materialized distribution, while the headline 80% stat counts stated interval endpoints directly, so the two can differ by a few scores.
Forecasters against the baseline
Per-target raw CRPS ratio against the paired ledger persistence baseline (geometric mean; below 1 beats persistence), lowest first. Unpaired means remain visible for context. The persistence baseline forecasts every target as its last official print with a realized-volatility interval — an agent earns its place by beating it. Forecaster rows score here only when witness-verified; the deterministic baseline is a replayable function of pre-cutoff ledger data and needs no witness of its own. Rows with few scored runs are noisy; read them accordingly.
| Forecaster | Scored / total runs | Claimed-time scored | CRPS ratio vs persistence | Paired win rate | Unpaired mean nCRPS | 80% coverage |
|---|---|---|---|---|---|---|
| thesis.analystgpt-5.5 | 69 / 209 | 2 | 0.98 | 59% | 1.081 | 74% |
| thesis.analystgpt-5.6-sol | 5 / 50 | 0 | 1.44 | 33% | 2.350 | 20% |
| brier.time_series_priorpersistence.last_printbaseline | 21 / 21 | 0 | — | — | 1.369 | 52% |
| thesis.analyst.laddergpt-5.5 | 0 / 12 | 0 | — | — | — | — |
| thesis.analyst.ladder_v2gpt-5.5 | 0 / 9 | 0 | — | — | — | — |
| thesis.analyst.laddergpt-5.6 | 0 / 1 | 0 | — | — | — | — |
| thesis.analystgpt-5.6-luna | 0 / 5 | 0 | — | — | — | — |
| thesis.analystgpt-5.6-terra | 0 / 18 | 0 | — | — | — | — |
| thesis.analyst.laddergpt-5.6-sol | 0 / 5 | 0 | — | — | — | — |
| thesis.analyst.ladder_v2gpt-5.6-sol | 0 / 6 | 0 | — | — | — | — |
| thesis.analyst.ladder_v2gpt-5.6-terra | 0 / 6 | 0 | — | — | — | — |
| thesis.analyst.ladder_v2gpt-5.6-luna | 0 / 5 | 0 | — | — | — | — |
Evaluation splits
Rows are split by resolutionDate, not run order. Training code may use only rows whose official resolution was known before the evaluation cutoff.
Latest resolutions
Both published chronology tiers appear here. Only rows marked witnessed — custody root externally witnessed before the observation — count toward the headline numbers above; claimed rows rest on recorded timestamps alone.
| Forecast | Predicted | Observed | In interval | Elicitation | Chronology | Score |
|---|---|---|---|---|---|---|
| FY2026 CDFI Assistance Obligations | $281.1M | $-7.917M | yes | interval seeded | witnessed | raw CRPS 218.698 |
| USFS Minnesota obligations, FY2026 | $29.5M | $54.572M | no | interval seeded | witnessed | raw CRPS 17.145 |
| NTIA 11.038 FY2026 obligations | $45M | $-30.077M | no | interval seeded | witnessed | raw CRPS 76.978 |
| US DoD prime award obligations FY2026 | $455B | $404.8B | yes | interval seeded | witnessed | raw CRPS 29.690 |
| FY2026 DoD prime contract obligations | $505B | $395.1B | no | interval seeded | witnessed | raw CRPS 84.210 |
| DoD New Prime Awards FY2026 | 1.4M | 2.9M | no | interval seeded | witnessed | raw CRPS 1.358 |
| DoD Prime Award Transactions FY2026 | 3.9M | 3.4M | no | interval seeded | witnessed | raw CRPS 0.316 |
| DoD Unique Prime Contract Recipients, FY2026 | 56k | 38k | no | interval seeded | witnessed | raw CRPS 17.420 |
| DoD Small-Business Contract Obligation Share FY2026 | 20.2% | 18.51% | no | interval seeded | witnessed | raw CRPS 1.059 |
| DHS Title VI FY2026 award obligations | $24,800,000,000 | $34,908,401,625 | no | interval seeded | witnessed | raw CRPS 6990169189.860 |
| Canada CPI YoY Sep 2026 | 3.0% | 3.1% | yes | interval seeded | witnessed | raw CRPS 0.116 |
| US continued claims, week ending 2026-09-26 | 1.7M | 1.7M | yes | interval seeded | witnessed | nCRPS 0.859 |
Full history: the Thesis Log · method: why forecasting is the harness