Brier score
Mean squared error between the market’s YES probability and the actual 0/1 outcome. 0 is perfect, 0.25 is a coin flip, 1 is consistently wrong. Lower is better.
Calibration
Market calibration
When a prediction market prices something at 70%, does it resolve YES about 70% of the time? This page tracks the answer — venue by venue, category by category — using the probability each market showed 24 hours before it resolved, not the near-certain price at settlement. That lead-time horizon measures genuine foresight, not convergence.
What we’re measuring — and what we’re not
This page measures
- · Whether venue prices are well-calibrated (70% markets resolve YES ~70% of the time)
- · Brier score and directional accuracy at the 24h lead-time horizon, per venue
- · How calibration varies by category and by how far in advance the price was recorded
This page does NOT measure
- · ProbCast’s own independent outcome forecasts (we don’t make them)
- · Whether a high trust score guarantees accuracy (that’s the claim we’re still earning)
- · Markets with ambiguous resolution rules or disputed outcomes
Markets priced at 70% should resolve YES about 70% of the time. Each dot is a probability bucket — on the dotted diagonal means the prices were honest; above it, the market was underconfident; below, overconfident. Dot size is sample count (1,761 graded markets).
Calibration by price bucket
Each row is one bucket from the curve: what markets were priced at, and how often they resolved YES.
Accuracy by lead time
How much is a price worth a day, a week, a month before resolution? Each row grades the probability the market showed at that lead time. Expect Brier to worsen as the horizon grows — the gap quantifies how much late information is priced in. The at-settlement row is a baseline, not a forecast: it grades the last observed price, which for most markets was observed once the outcome was already known, so its near-perfect score measures convergence rather than foresight. Read the lead-time rows for skill.
| Segment | Resolved | Brier (lower = better) | Accuracy |
|---|---|---|---|
| At settlement (not a forecast) | 1885 | 0.081 | 89% |
| 24 hours out | 1761 | 0.098 | 86% |
| 7 days out | 865 | 0.126 | 81% |
| 30 days out | 272 | 0.140 | 79% |
The full analysis of this decay curve — what it says about how much foresight market prices actually contain — lives here.
Venue leaderboard
Graded on the YES probability 24 hours before resolution (captures that postdate the deadline are excluded). Read Brier first: directional accuracy flatters near-certain contracts (a market at 99% earns an easy “correct call” that says little about calibration quality). The graded pools differ sharply in composition — most of one venue's contracts may sit at near-certain prices while another's sit mid-range — so venue rows are descriptive of each pool, not a like-for-like comparison of venues or of real-money vs play-money incentives.
How we measure venue calibration
Step 1
Capture prices at fixed lead times
We select the latest valid snapshot strictly before each evaluation horizon — 24 hours, seven days, and 30 days before the event deadline. The settlement-adjacent price is kept only as a convergence baseline, not a graded forecast.
Step 2
Wait for the real outcome
When the market settles YES or NO, that binary result becomes ground truth. No estimates, no backfill.
Trust-tier scorecard
The falsifiable test of the Trust Score: if it means anything, High-trust markets should realize a lower Brier loss than Low-trust ones. Forward-only — scored from markets captured and resolved after tracking began (Jul 26, 2026), never backfilled.
| Trust tier | Resolved | Brier | Abs error | Large-error rate |
|---|---|---|---|---|
| High | 17 | 0.000 | 0.004 | 0% |
| Medium | 137 | 0.007 | 0.024 | 1% |
| Mixed | 317 | 0.016 | 0.041 | 5% |
| Low | 456 | 0.114 | 0.228 | 36% |
| Overall | 927 | 0.063 | 0.130 | 20% |
Error does fall as trust rises on this sample, but the tiers are not answering questions of equal difficulty: high-trust contracts are far more often priced near 0% or 100% a day out, and those are nearly free to get right. Until the tiers are comparable, that ordering is not evidence the Trust Score works. This is a real result either way — the harness reports it honestly rather than assuming the score works.
Directional accuracy
The share of resolved markets where the side the market favored (above 50%) was the side that won. A blunter measure than Brier, but easy to read. Coin-flip (≈50%) predictions are excluded — they carry no directional signal.
Voided or non-binary settlements are excluded. Samples are small until enough markets resolve, so treat early numbers as directional. See the methodology for how scores are derived. ProbCast provides prediction-market data and analytics for informational purposes only. Not financial, trading, betting, investment, legal, or tax advice.