Framing (simple)
A probability is not a promise. It’s a frequency statement. If we say “60%”, it should behave like “about 6 times out of 10” on a large enough sample. Calibration is what makes probabilities honest and comparable over time.
One-sentence summary
Calibration checks whether predicted probabilities match observed outcomes: 60% should look like ~6/10 over enough matches, and reliability curves help visualize and correct bias.
Why calibration matters
Without calibration, probabilities can be misleading: a model may look “confident” while being wrong too often, or too cautious while hiding useful signal. Calibration is the bridge between “model output” and “reliable reading”.
What calibration means (concretely)
Suppose you look at all matches where the model outputs ~60% for an outcome. If calibration is good, you should observe that outcome about 60% of the time, over a sufficiently large set.
- Well calibrated: 60% → ~6 outcomes out of 10.
- Overconfident: 60% → much less than 6/10 in reality.
- Underconfident: 60% → more than 6/10 in reality.
How to measure calibration
- Reliability curve: predicted probability vs observed frequency.
- Brier score: squared error on probabilities (lower is better).
- LogLoss: heavily penalizes “very confident, very wrong” predictions.
Common pitfalls
- Too few matches: you need volume; small samples are noisy.
- Mixing contexts: calibration can differ by league, season phase, or data quality.
- Over-correction: “fixing” too aggressively can make things worse.
Go further
Calibration is closely linked to how you choose thresholds and how you handle drift over time:
Conclusion
A probability is only useful if it is reliable. Calibration ensures that “60%” behaves like “about 6 out of 10” over time, which makes interpretation clearer, comparisons fairer, and overconfidence easier to spot.