Part 4b · The five layers
Validity under pressure
What breaks when someone, or something, searches against the instrument.
- 4b.1Falsification and failure regionsEstimation counts failures; falsification finds them. Search for the most likely breaks and map the region, not the anecdote.10 min read
- 4b.2Adaptive validityInteractiveAccuracy before any optimization is not the question. How evaluator error grows under best-of-N and training, and how to measure an exploitability curve.10 min read
- 4b.3Goodhart pressure and self-confirming loopsReward hacking needs an exploit; self-confirmation needs only a shared blind spot. Tell them apart and measure the coupling behind both.11 min read
- 4b.4The validation profile and validity statementInteractiveReplace hierarchy labels with measurements, Π = (N, E, T, G, C, D), and ship every evaluator with a falsifiable statement of scope.11 min read