Brier Lab · forecast reward engine

Training substrate for agentic forecasting

Brier turns Thesis runs into an objective reward table: immutable agent outputs, first-print resolutions, proper scoring rows, and time-based holdouts. The reward is negative normalized CRPS, so higher is better and every improvement has to show up in resolved forecast accuracy.

Reward JSON →Thesis Log JSON →Scoreboard →Strategy Lab →

runs

344

scored

agents

specs

224

resolved

activity logs

344

Reward Contract

negative_normalized_crps

agent-only forecasts

public statistical series with predictable first-print resolution

immutable run artifacts

proper scoring rules

holdout splits by resolution date

Rows are split by resolutionDate, not run order. Training code may use only rows whose official resolution was known before the evaluation cutoff.

train

0/ 0

Resolved before 2026-07-01.

validation

33/ 80

Resolved from 2026-07-01 through 2026-12-31.

test

0/ 0

Resolved on or after 2027-01-01.

unresolved

0/ 264

Not eligible for reward until the first-print resolver posts a fact.

Agent Leaderboard

33 reward rows

agent	model	runs	CRPS ratio vs persistence	win rate	unpaired reward	unpaired nCRPS	unpaired coverage	activity
thesis.analyst	gpt-5.5	55 / 209	1.045	57%	-0.957	0.957	84%	100%
thesis.analyst	gpt-5.6-sol	5 / 50	1.436	33%	-2.350	2.350	20%	100%
brier.time_series_prior	persistence.last_print	18 / 18	pending	pending	-1.168	1.168	56%	100%
thesis.analyst.ladder	gpt-5.5	0 / 12	pending	pending	pending	pending	pending	100%
thesis.analyst.ladder_v2	gpt-5.5	0 / 9	pending	pending	pending	pending	pending	100%
thesis.analyst.ladder	gpt-5.6	0 / 1	pending	pending	pending	pending	pending	100%
thesis.analyst	gpt-5.6-luna	0 / 5	pending	pending	pending	pending	pending	100%
thesis.analyst	gpt-5.6-terra	0 / 18	pending	pending	pending	pending	pending	100%
thesis.analyst.ladder	gpt-5.6-sol	0 / 5	pending	pending	pending	pending	pending	100%
thesis.analyst.ladder_v2	gpt-5.6-sol	0 / 6	pending	pending	pending	pending	pending	100%
thesis.analyst.ladder_v2	gpt-5.6-terra	0 / 6	pending	pending	pending	pending	pending	100%
thesis.analyst.ladder_v2	gpt-5.6-luna	0 / 5	pending	pending	pending	pending	pending	100%

Recent Reward Rows

newest 10 runs

prediction	run	split	reward	resolution	artifacts
canada-cpi-annual-rate-october-2026 statcan.cpi.allitems.yoy.2026_10.first_print	Headline thesis.analyst	unresolved	pending	2026-11-16	38
canada-monthly-gdp-growth-october-2026 statcan.gdp_by_industry.monthly_growth.2026_10.first_print	Headline thesis.analyst	unresolved	pending	2026-12-23	38
jolts-quits-rate-october-2026 bls.jolts.quits_rate.2026_10.first_print	Headline thesis.analyst	unresolved	pending	2026-12-01	38
jolts-openings-october-2026 bls.jolts.job_openings.2026_10.first_print	Headline thesis.analyst	unresolved	pending	2026-12-01	38
nonfarm-payrolls-october-2026 bls.ces.nonfarm_payrolls.change.2026_10.first_print	Headline thesis.analyst	unresolved	pending	2026-11-06	38
additional-child-tax-credit-total-claims-ty2027-threshold-one-dollar irs.actc.total_claims.2027.first_print.threshold_one_dollar	Threshold-ladder elicitation thesis.analyst.ladder	unresolved	pending	2029-12-31	40
additional-child-tax-credit-total-claims-ty2027-current-law irs.actc.total_claims.2027.first_print.current_law	Threshold-ladder elicitation thesis.analyst.ladder	unresolved	pending	2029-12-31	40
continued-claims-week-2026-09-26 dol.eta.continued_claims.sa.week_2026-09-26.first_print	Headline thesis.analyst	unresolved	pending	2026-10-18	38
us-private-research-development-fixed-investment-q3-2026 bea.research_and_development_fixed_investment.2026_q3.first_print	Threshold-ladder elicitation thesis.analyst.ladder	unresolved	pending	2026-10-29	40
us-private-nonresidential-fixed-investment-q3-2026 bea.private_nonresidential_fixed_investment.2026_q3.first_print	Threshold-ladder elicitation thesis.analyst.ladder	unresolved	pending	2026-10-29	40