Glossary · Reliability, calibration, and statistics
Cohen's κ
Chance-corrected agreement, (p_o − p_e) / (1 − p_e). It depends on prevalence: identical sensitivity and specificity give very different values at different base rates, so report the confusion matrix and prevalence with it.
Defined in
Handbook · ValidationThe judge as a noisy sensorModel judge error like a diagnostic test: correct pass rates, carry calibration noise into the result, and audit error per system.Related terms
- Sensitivity and specificity (TPR, TNR, FPR)
- A binary evaluator's error rates against a reference: sensitivity (TPR) is the probability it accepts a true positive, specificity (TNR) the probability it rejects a true negative, and the false-positive rate FPR = 1 − TNR.
Where it comes up
- LLM judges in practiceHandbook · Instrument
- Reliability and generalizabilityHandbook · Validation
- Notation and formula cardHandbook · Reference