Skip to content
Benchmaker

Glossary · Validity and evidence

Adaptive validity

4Validation

Validity of an evaluator after the system it measures has been optimized against it by selection, search, training, or developer iteration. Measured with an exploitability curve; accuracy on ordinary outputs does not establish it.

Defined in

Handbook · Validity under pressureAdaptive validityAccuracy before any optimization is not the question. How evaluator error grows under best-of-N and training, and how to measure an exploitability curve.
Natural validity
Validity of an evaluator's interpretation for ordinary measurement on the target distribution, before any system has been optimized against it. Contrast adaptive validity.
Exploitability curve
Disagreement between an evaluator and an independent reference anchor as a function of optimization exposure b. b* is the largest tested exposure before divergence exceeds a predeclared tolerance.
Tested exposure
The largest optimization exposure tested before the evaluator and an independent reference anchor diverge beyond a predeclared tolerance. It makes adaptive validity expire: report it with the tested range, never as robustness beyond it.

Where it comes up

See it in the full glossary →