Brier score
Mean squared error between the market’s YES probability and the actual 0/1 outcome. 0 is perfect, 0.25 is a coin flip, 1 is consistently wrong. Lower is better.
Calibration
Market calibration
When a prediction market prices something at 70%, does it resolve YES about 70% of the time? This page tracks the answer — venue by venue, category by category — using the probability each market showed 24 hours before it resolved, not the near-certain price at settlement. That lead-time horizon measures genuine foresight, not convergence.
Calibration curve
Markets priced at 70% should resolve YES about 70% of the time. Each dot is a probability bucket — on the dotted diagonal means the prices were honest; above it, the market was underconfident; below, overconfident. Dot size is sample count (1,214 graded markets).
| Priced at | Resolved YES (│ = priced) |
|---|
What we’re measuring — and what we’re not
This page measures
- · Whether venue prices are well-calibrated (70% markets resolve YES ~70% of the time)
- · Brier score and directional accuracy at the 24h lead-time horizon, per venue
- · How calibration varies by category and by how far in advance the price was recorded
This page does NOT measure
- · ProbCast’s own independent outcome forecasts (we don’t make them)
- · Whether a high trust score guarantees accuracy (that’s the claim we’re still earning)
- · Markets with ambiguous resolution rules or disputed outcomes
Accuracy by lead time
How much is a price worth a day, a week, a month before resolution? Each row grades the probability the market showed at that lead time. Expect Brier to worsen as the horizon grows — the gap quantifies how much late information is priced in. The at-settlement row is a baseline, not a forecast: it grades the last observed price, which for most markets was observed once the outcome was already known, so its near-perfect score measures convergence rather than foresight. Read the lead-time rows for skill.
| Segment | Resolved | Brier (lower = better) | Accuracy |
|---|---|---|---|
| At settlement (not a forecast) | 1112 | 0.081 | 87% |
| 24 hours out | 1135 | 0.100 | 84% |
| 7 days out | 540 | 0.120 | 81% |
| 30 days out | 115 | 0.117 | 83% |
Venue leaderboard
Graded on the YES probability 24 hours before resolution. Read Brier first: directional accuracy flatters near-certain contracts (a market at 99% earns an easy “correct call” that says little about calibration quality), and per-venue samples differ in size and composition.
How we measure venue calibration
Step 1
Capture prices at fixed lead times
We select the latest valid snapshot strictly before each evaluation horizon — 24 hours, seven days, and 30 days before the event deadline. The settlement-adjacent price is kept only as a convergence baseline, not a graded forecast.
Step 2
Wait for the real outcome
When the market settles YES or NO, that binary result becomes ground truth. No estimates, no backfill.
Trust-tier scorecard
The falsifiable test of the Trust Score: if it means anything, High-trust markets should realize a lower Brier loss than Low-trust ones. Forward-only — scored from markets captured and resolved after tracking began (Jul 26, 2026), never backfilled.
| Trust tier | Resolved | Brier | Abs error | Large-error rate |
|---|---|---|---|---|
| High | 9 | 0.000 | 0.002 | 0% |
| Medium | 296 | 0.000 | 0.002 | 0% |
| Watch | 308 | 0.005 | 0.008 | 1% |
| Low | 1,556 | 0.053 | 0.117 | 16% |
| Overall | 2,169 | 0.039 | 0.085 | 11% |
Some tiers hold only a handful of resolved markets, too few to order High against Low — no verdict is claimed until every tier has ~10. This is a real result either way — the harness reports it honestly rather than assuming the score works.
Directional accuracy
The share of resolved markets where the side the market favored (above 50%) was the side that won. A blunter measure than Brier, but easy to read. Coin-flip (≈50%) predictions are excluded — they carry no directional signal.
Voided or non-binary settlements are excluded. Samples are small until enough markets resolve, so treat early numbers as directional. See the methodology for how scores are derived. ProbCast provides prediction-market data and analytics for informational purposes only. Not financial, trading, betting, investment, legal, or tax advice.