Glossary · Reliability, calibration, and statistics
Differential misclassification
Evaluator error rates that differ between compared systems, for example because one was optimized against the evaluator. One shared calibration then biases corrections and comparisons; audit error per system or at least per generator family.
Defined in
Handbook · ValidationThe judge as a noisy sensorModel judge error like a diagnostic test: correct pass rates, carry calibration noise into the result, and audit error per system.Related terms
- Rogan–Gladen correction
- Inverts a binary sensor model to estimate the true pass rate from an imperfect judge: p̂ = (p_obs − FPR) / (TPR − FPR), assuming error rates transport.
Where it comes up
- LLM judges in practiceHandbook · Instrument
- Comparing systemsHandbook · Validation
- Adaptive validityHandbook · Validity under pressure
- Goodhart pressure and self-confirming loopsHandbook · Validity under pressure
- E: Reasoning-model trainingHandbook · Case studies
- G: Documentation agentHandbook · Case studies
- Notation and formula cardHandbook · Reference
- Correcting a noisy judgeCalculator