Glossary · Reliability, calibration, and statistics
Reliability diagram
A plot of the observed event rate against a monitor's or judge's predicted score, by score bin, used with recalibration on held-out data. For a calibrated score, about 70% of events scored 0.7 are real.
Defined in
Handbook · RegimeMonitoring and control in productionMonitoring is not continuous benchmarking: base rates, calibration, conformal caveats, and the moment a monitor becomes a training signal.Related terms
- Calibration
- Agreement between predicted confidence or score and empirical frequencies of correctness.