Glossary · Reliability, calibration, and statistics
Alert precision
3RegimeThe share of a monitor's alerts that are real, Pr[danger | alert] = pt / (pt + (1 − p)f), for sensitivity t, false-positive rate f, and base rate p. When p is tiny, even a small f can dominate.
Also called: base-rate problem, positive predictive value
Defined in
Handbook · RegimeMonitoring and control in productionMonitoring is not continuous benchmarking: base rates, calibration, conformal caveats, and the moment a monitor becomes a training signal.Related terms
- Sensitivity and specificity (TPR, TNR, FPR)
- A binary evaluator's error rates against a reference: sensitivity (TPR) is the probability it accepts a true positive, specificity (TNR) the probability it rejects a true negative, and the false-positive rate FPR = 1 − TNR.
- Runtime control
- An evaluator that can block, reroute, escalate, or roll back deployed behavior, rather than only observe it.
- Calibration
- Agreement between predicted confidence or score and empirical frequencies of correctness.
Where it comes up
- Use regimes: same judge, four jobsHandbook · Regime
- F: Production safety monitorHandbook · Case studies
- Notation and formula cardHandbook · Reference
- Alert precision at production base ratesCalculator