Glossary · Validity and evidence
Natural validity
4ValidationValidity of an evaluator's interpretation for ordinary measurement on the target distribution, before any system has been optimized against it. Contrast adaptive validity.
Defined in
Handbook · Validity under pressureAdaptive validityAccuracy before any optimization is not the question. How evaluator error grows under best-of-N and training, and how to measure an exploitability curve.Related terms
- Adaptive validity
- Validity of an evaluator after the system it measures has been optimized against it by selection, search, training, or developer iteration. Measured with an exploitability curve; accuracy on ordinary outputs does not establish it.
Where it comes up
- The frameworkPage