Brier Lab · forecast reward engine
Training substrate for agentic forecasting
Brier turns Thesis runs into an objective reward table: immutable agent outputs, first-print resolutions, proper scoring rows, and time-based holdouts. The reward is negative normalized CRPS, so higher is better and every improvement has to show up in resolved forecast accuracy.
runs
530
scored
51
agents
22
specs
411
resolved
41
activity logs
34
Reward Contract
negative_normalized_crpsagent-only forecasts
public statistical series with predictable first-print resolution
immutable run artifacts
proper scoring rules
holdout splits by resolution date
Rows are split by resolutionDate, not run order. Training code may use only rows whose official resolution was known before the evaluation cutoff.
train
51/ 51
Resolved before 2026-07-01.
validation
0/ 0
Resolved from 2026-07-01 through 2026-12-31.
test
0/ 0
Resolved on or after 2027-01-01.
unresolved
0/ 479
Not eligible for reward until the first-print resolver posts a fact.
Agent Leaderboard
51 reward rows| agent | model | runs | reward | nCRPS | coverage | activity |
|---|---|---|---|---|---|---|
| Global near-term indicator source synthesis | Codex recorded source-context synthesis | 5 / 8 | -0.128 | 0.128 | 100% | 0% |
| Three-agent CPI ensemble | Codex recorded agent ensemble | 2 / 2 | -0.142 | 0.142 | 100% | 0% |
| UK indicator agent ensemble | Codex recorded agent runs | 7 / 10 | -0.183 | 0.183 | 100% | 0% |
| thesis.analyst | claude-fable-5 | 7 / 21 | -0.184 | 0.184 | 86% | 0% |
| prototype seed | unreported | 5 / 313 | -0.199 | 0.199 | 100% | 0% |
| US near-term public outcomes agent | Codex recorded agent run | 7 / 16 | -0.225 | 0.225 | 71% | 0% |
| scout-2.control | gpt-5-mini | 3 / 9 | -0.229 | 0.229 | 100% | 0% |
| Euro area/Japan indicator agent ensemble | Codex recorded agent runs | 4 / 8 | -0.310 | 0.310 | 100% | 0% |
| brier-1.packed | gpt-5 | 3 / 9 | -0.322 | 0.322 | 33% | 0% |
| Canada/Australia indicator agent ensemble | Codex recorded agent runs | 4 / 9 | -0.370 | 0.370 | 50% | 0% |
| thesis.analyst | gpt-5.5 | 4 / 34 | -0.429 | 0.429 | 75% | 100% |
| brier-1.control | gpt-5.4 | 0 / 3 | pending | pending | pending | 0% |
| brier-1.packed | gpt-5.4 | 0 / 4 | pending | pending | pending | 0% |
| brier-1.shadow | gpt-5 | 0 / 30 | pending | pending | pending | 0% |
| Occupation automation exposure source synthesis | Codex recorded source-context synthesis | 0 / 6 | pending | pending | pending | 0% |
| brier-occupation-projection | Codex recorded source-context synthesis | 0 / 6 | pending | pending | pending | 0% |
| BLS Employment Projections | BLS 2024-2034 projection release, OEWS-compatible interpolation | 0 / 6 | pending | pending | pending | 0% |
| brier-cps-occupation-fast-proxy | Codex recorded source-context synthesis | 0 / 6 | pending | pending | pending | 0% |
| brier-occupation-automation-scenarios | Codex recorded source-context synthesis | 0 / 12 | pending | pending | pending | 0% |
| BLS Employment Projections | BLS 2024-2034 projection release | 0 / 6 | pending | pending | pending | 0% |
| brier-occupation-wage-pressure | Codex recorded source-context synthesis | 0 / 6 | pending | pending | pending | 0% |
| BLS OEWS current table | May 2025 annual median wage carry-forward baseline | 0 / 6 | pending | pending | pending | 0% |
Recent Reward Rows
newest 10 runs| prediction | run | split | reward | resolution | artifacts |
|---|---|---|---|---|---|
| cps-business-financial-employment-june-2026 bls.cps.employed_people_by_occupation.business_financial_operations.june_2026.first_print | CPS fast proxy brier-cps-occupation-fast-proxy | unresolved | pending | 2026-07-02 | 0 |
| cps-computer-math-employment-june-2026 bls.cps.employed_people_by_occupation.computer_mathematical.june_2026.first_print | CPS fast proxy brier-cps-occupation-fast-proxy | unresolved | pending | 2026-07-02 | 0 |
| cps-healthcare-support-employment-june-2026 bls.cps.employed_people_by_occupation.healthcare_support.june_2026.first_print | CPS fast proxy brier-cps-occupation-fast-proxy | unresolved | pending | 2026-07-02 | 0 |
| cps-office-admin-employment-june-2026 bls.cps.employed_people_by_occupation.office_administrative_support.june_2026.first_print | CPS fast proxy brier-cps-occupation-fast-proxy | unresolved | pending | 2026-07-02 | 0 |
| cps-production-employment-june-2026 bls.cps.employed_people_by_occupation.production.june_2026.first_print | CPS fast proxy brier-cps-occupation-fast-proxy | unresolved | pending | 2026-07-02 | 0 |
| cps-transport-material-moving-employment-june-2026 bls.cps.employed_people_by_occupation.transportation_material_moving.june_2026.first_print | CPS fast proxy brier-cps-occupation-fast-proxy | unresolved | pending | 2026-07-02 | 0 |
| bls-business-financial-employment-2034 bls.employment_projections.national_occupation_employment.soc_13_0000.2034.actual_first_print | Brier long-run - BLS pack brier-occupation-automation-scenarios | unresolved | pending | 2035-09-15 | 0 |
| bls-computer-math-employment-2034 bls.employment_projections.national_occupation_employment.soc_15_0000.2034.actual_first_print | Brier long-run - BLS pack brier-occupation-automation-scenarios | unresolved | pending | 2035-09-15 | 0 |
| bls-healthcare-support-employment-2034 bls.employment_projections.national_occupation_employment.soc_31_0000.2034.actual_first_print | Brier long-run - BLS pack brier-occupation-automation-scenarios | unresolved | pending | 2035-09-15 | 0 |
| bls-office-admin-employment-2034 bls.employment_projections.national_occupation_employment.soc_43_0000.2034.actual_first_print | Brier long-run - BLS pack brier-occupation-automation-scenarios | unresolved | pending | 2035-09-15 | 0 |