Glossary · Optimization and failure dynamics
Exploitability curve
Disagreement between an evaluator and an independent reference anchor as a function of optimization exposure b. b* is the largest tested exposure before divergence exceeds a predeclared tolerance.
Defined in
Handbook · Validity under pressureAdaptive validityAccuracy before any optimization is not the question. How evaluator error grows under best-of-N and training, and how to measure an exploitability curve.Related terms
- Tested exposure
- The largest optimization exposure tested before the evaluator and an independent reference anchor diverge beyond a predeclared tolerance. It makes adaptive validity expire: report it with the tested range, never as robustness beyond it.
- Adaptive validity
- Validity of an evaluator after the system it measures has been optimized against it by selection, search, training, or developer iteration. Measured with an exploitability curve; accuracy on ordinary outputs does not establish it.
Where it comes up
- Use regimes: same judge, four jobsHandbook · Regime
- Evaluators inside training loopsHandbook · Regime
- The validation profile and validity statementHandbook · Validity under pressure
- Channels, independence, and anchor burnHandbook · Lifecycle
- The One SpineHandbook · Practice
- Anti-patterns and failure vocabularyHandbook · Practice
- E: Reasoning-model trainingHandbook · Case studies
- Notation and formula cardHandbook · Reference