Strategy Lab · baseline discipline
Forecast strategies have to beat simple replayable benchmarks
This page compares deterministic baselines with the actual Brier agent run on resolved target panels. The first slice is SNAP FY2025 state payment error rates: every baseline uses only FY2023 and FY2024 values, while the agent row is the immutable published forecast. Everything here is a retrospective reconstruction — strategy rows are replayed over outcomes that already resolved, carry no chronology verification, and never enter headline calibration, rewards, or leaderboards.
LLM Judge Layer
Judge records score forecast process quality from the public trace: base rates, source grounding, resolution clarity, uncertainty, mechanisms, counterarguments, and coherence. They are auxiliary diagnostics; reward still comes only from resolved CRPS.
| band | judged | scored | judge | nCRPS | 80% cover |
|---|---|---|---|---|---|
weak 0.0-2.0 | 0 | 0 | pending | pending | pending |
developing 2.0-3.0 | 0 | 0 | pending | pending | pending |
strong 3.0-4.0 | 344 | 80 | 3.712 | 1.139 | 74% |
| disagreement | target | run | judge | nCRPS | 80% cover |
|---|---|---|---|---|---|
high judge bad score | continued-claims-week-2026-08-08 | Ledger persistence baseline | 3.160 | 3.997 | outside |
high judge bad score | initial-claims-week-2026-07-18 | Ledger persistence baseline | 3.160 | 2.996 | outside |
high judge bad score | continued-claims-week-2026-08-08 | Headline | 3.780 | 2.984 | inside |
high judge bad score | initial-claims-week-2026-07-18 | Headline | 3.780 | 2.845 | outside |
high judge bad score | continued-claims-week-2026-09-12 | Ledger persistence baseline | 3.160 | 2.821 | outside |
Agent Adjustments From Time-Series Prior
Each eligible agent run now gets a last-print persistence prior. The adjustment is the primary Brier point estimate minus that prior, scaled where useful by the prior's 80% interval.
| target | latest | prior | agent | adjustment | interval share |
|---|---|---|---|---|---|
| dol.eta.continued_claims.sa.week_2026-08-01.first_print US continued claims, week ending Aug 1 2026 | 1.8M 2026-07-11 | 1.8M | 1.8M | -0M | -0.89x |
| us.dol.initial_claims.sa.week_2026-08-01 US initial jobless claims, week ending Aug 1 2026 | 187k 2026-07-18 | 187k | 206k | +19k | +0.63x |
| dol.eta.continued_claims.sa.week_2026-08-08.first_print US continued claims, week ending Aug. 8, 2026 | 1.8M 2026-07-18 | 1.8M | 1.8M | +0M | +0.54x |
| dol.eta.continued_claims.sa.week_2026-07-25.first_print US continued claims, July 25 | 1.8M 2026-07-04 | 1.8M | 1.8M | +0M | +0.28x |
| us.dol.initial_claims.sa.week_2026-07-25 US initial claims, week ending July 25 | 208k 2026-07-11 | 208k | 212k | +4k | +0.21x |
| dol.eta.continued_claims.sa.week_2026-08-22.first_print US Continued Claims, Week Ending 2026-08-22 | 1.8M 2026-08-01 | 1.8M | 1.8M | +0M | +0.20x |
| us.dol.initial_claims.sa.week_2026-09-12 Initial claims, week ending Sep. 12, 2026 | 203k 2026-08-22 | 203k | 207k | +4k | +0.19x |
| dol.eta.continued_claims.sa.week_2026-09-05.first_print Continued claims, week ending Sep. 5, 2026 | 1.8M 2026-08-15 | 1.8M | 1.8M | -0M | -0.18x |
| us.dol.initial_claims.sa.week_2026-08-08 US initial claims, week ending Aug. 8, 2026 | 197k 2026-07-25 | 197k | 201k | +4k | +0.15x |
| dol.eta.continued_claims.sa.week_2026-09-12.first_print US continued claims, week ending 2026-09-12 | 1.8M 2026-08-22 | 1.8M | 1.8M | +0M | +0.14x |
| us.dol.initial_claims.sa.week_2026-08-22 Initial claims, week ending Aug. 22, 2026 | 209k 2026-08-08 | 209k | 207k | -2k | -0.09x |
| dol.eta.continued_claims.sa.week_2026-08-29.first_print US continued claims, week ending 2026-08-29 | 1.8M 2026-08-08 | 1.8M | 1.8M | -0M | -0.07x |
SNAP FY2025 state payment error panel
Resolved state and territory payment error rate forecasts scored against simple baselines and the recorded Brier agent run.
Strategy Scoreboard
lower nCRPS and MAE are better| strategy | mode | rows | nCRPS | MAE | vs persistence | bias | 80% cover |
|---|---|---|---|---|---|---|---|
Last-print persistence Hold the most recent official value flat and score it as a replayable benchmark. | historical replay deterministic baseline mixed | 0 | pending | pending | pending | pending | pending |
Panel shrinkage trend Blend 25% of the entity's last annual change with 75% of the panel median change. | historical replay deterministic baseline mixed | 0 | pending | pending | pending | pending | pending |
Brier primary agent The public Thesis forecast run actually recorded before resolution. | forward only agent forward mixed | 0 | pending | pending | pending | pending | pending |
Largest Agent nCRPS Misses Against Persistence
top 0| target | actual | persistence | agent | nCRPS delta | MAE delta |
|---|