Adaptive validity

Adaptive validity concerns an evaluator after the system it measures has been optimized against it; natural validity concerns ordinary measurement on the target distribution. The two come apart: an evaluator that is accurate on typical outputs can be wrong on exactly the outputs a selector or a trainer picks.

Natural and adaptive validity#

Classical measurement asks how an instrument behaves on a target distribution. Adaptive evaluation adds a harder question: what happens when the measured system searches for cases that maximize the instrument? So never ask only whether an evaluator is accurate, but on which distribution, before or after optimization: a judge calibrated on ordinary outputs may become badly miscalibrated on outputs selected to maximize its score.

Take the grounding judge of DocsAgent, the documentation agent of Case G, with TPR 0.90 and FPR 0.10 measured on adjudicated answers from ordinary traffic (illustrative): natural-validity evidence. Use it to keep the best of 16 answers, and what matters is its error on the answers it keeps; train DocsAgent against it, and the answers move toward whatever it rewards. Pressure rises from reporting to selection, search, and training (Use regimes: same judge, four jobs), and differential misclassification is the same problem met when comparing systems (The judge as a noisy sensor).

Best-of-N under a benign selection model#

Best-of-N generates N candidates and keeps one that the evaluator accepts or scores highest, which shifts the selected distribution toward the evaluator's tail. The simplest model treats the evaluator as binary, with a true-positive rate $t$ and a false-positive rate $f$ that are the same for every candidate, and each candidate as correct with probability $p$. Among the candidates it flags, the share that are correct is:

$$\Pr[C \mid +] = \frac{p\,t}{p\,t + (1-p)\,f}$$

where:

• $C$ is the event that a candidate is correct, and $+$ the event that the evaluator flags it;
• $p$ is the share of candidates that are correct, the policy's base rate on the task;
• $t$ and $f$ are the evaluator's TPR and FPR on these candidates.

In words: a flag mixes the true passes the evaluator catches with the failures it lets through, and only the first kind is worth selecting.

N does not appear in the formula. The arithmetic is the same as for alert precision at a production base rate (Monitoring and control in production). Take a mediocre evaluator, with t = 0.9 and f = 0.3, on a hard task with p = 0.2. Then $0.18/(0.18 + 0.24) \approx 42.9\%$ of flagged candidates are correct, and the selected output is right only about 43% of the time no matter how large N grows.

More samples change something else: the chance that at least one candidate is flagged, $1 - (1-q)^N$ with $q = pt + (1-p)f = 0.42$, passes 99% by N = 9. If the selector picks uniformly among flagged candidates (among all N when none is flagged), the chance that its pick is correct climbs from 20% at N = 1 to the 42.9% ceiling and stays there. A perfect evaluator would deliver pass@16, the chance that at least one of 16 candidates is correct (pass^16 would need all 16): 97.2%. More samples raise the chance that something passes; only a better evaluator raises what passing means.

At this base rate the false-positive rate dominates: lowering $f$ to 0.10 lifts the ceiling to 69.2%, and to 0.03 lifts it to 88.2%, while raising $t$ to 0.99 moves it only to 45.2% (illustrative).

When the selector keeps the top score#

A selector that keeps the highest score depends on the extreme tail of the evaluator's score distribution, and the binary model says nothing about tails. One stylized expression for the chance that the winner is correct is:

$$\Pr[C_{\text{winner}}] = N \int p\, f_1(s)\, G(s)^{N-1}\, ds$$

where:

• $F_1$ and $F_0$ are the cumulative distributions of scores for correct and incorrect candidates, and $f_1$ is the density of correct scores;
• $p$ is the share of candidates that are correct, and $N$ the number of candidates;
• $G(s) = p\,F_1(s) + (1-p)\,F_0(s)$ is the chance that one other candidate, correct or not, scores below $s$.

In words: the winner is correct when some correct candidate scores $s$ and all $N - 1$ others score lower, summed over every score and every candidate.

As N grows, the winner comes from deeper in the tail, and precision can deteriorate, improve, or plateau depending on how the two score distributions separate there; ordinary calibration does not answer the question. An illustrative case: scores are Normal with means two standard deviations apart, p = 0.2, and 2% of incorrect candidates land in an exploit cluster three standard deviations above the correct mean. The top-scored candidate is correct 66.3% of the time at N = 16 (84.3% without the cluster), peaks at N = 14, and is correct 0.2% of the time at N = 1,024. Without the cluster, the same evaluator keeps improving, to 99.7% at N = 1,024. A sliver of incorrect candidates that no average-case calibration would notice decides what a large N buys.

So evaluate a selector's evaluator on the outputs it actually selects, at the N you deploy; under strong selection, false-positive structure becomes especially important.

The calculator starts at the mediocre evaluator above. Try three things:

• Drag N from 1 to 256: precision among flagged candidates stays at 42.9%, while the chance that something is flagged passes 99% by N = 9.
• Lower the FPR to 10%: the ceiling rises to 69.2%, more than any N can add.
• Switch to continuous scores and drag N to 1,024: the pick peaks near N = 14 and then collapses. Set the exploit share ε to zero and it keeps climbing.

The exploitability curve#

Models frame the problem; measurement settles it. The framework defines an evaluator's error after optimization exposure $b$ as its disagreement with an independent reference anchor on the optimized outputs:

$$\varepsilon(b) = \Pr\big[\,\text{evaluator} \neq \text{anchor} \mid Q_b\,\big]$$

where:

• $b$ is the exposure budget, in the optimizer's own unit;
• $Q_b$ is the distribution of outputs after optimization at exposure $b$;
• "evaluator ≠ anchor" means the evaluator disagrees about an output with an independent reference anchor that the optimizer can neither see nor modify, such as adjudicated labels or hidden tests.

In words: how often the evaluator and an untouched reference disagree about what the optimizer produces at budget $b$.

There is no universal unit for $b$: it is N for best-of-N, KL divergence for KL-regularized RL, and iterations for developers. Record it in the optimizer's unit and, where possible, the shift it induces (Exposure, modification, and access).

The exploitability curve traces $\varepsilon(b)$, or proxy score against reference-anchor score, as exposure grows. Its summary is the tested exposure $b^{*}$: the largest exposure tested before divergence exceeds a predeclared tolerance. That definition turns adaptive validity into a falsifiable envelope rather than a vague warning about Goodhart.

Measuring the curve needs the actual optimizer, not a stand-in: run it against the evaluator at increasing budgets, score its outputs with the independent reference anchor, and plot proxy against reference anchor. Report the tested range and the failure boundary rather than merely saying "robust to Goodhart". The study is falsification of the evaluator by the optimizer that will use it (Falsification and failure regions).

Each point is an estimate with its own n and interval, so the tolerance rule must say whether the point estimate or the whole interval has to stay inside it, as for any release gate. An illustrative best-of-N report, with 400 adjudicated selected outputs per budget and a 10% tolerance:

Here $b^{*} = 16$ within a tested range of 1 to 64, and every interval up to it stays below the tolerance. The study supports best-of-16 and says nothing about the untested best-of-32.

The best-known exploitability curves come from reward modeling. Gao et al. (2023) showed that optimizing against a learned reward model can eventually reduce performance as judged by a stronger reference signal, the rise, peak, and decline that the RSI survey treats as the archetypal Goodhart curve (Chen, Wang & Qu, 2026, §5.1). Evaluators inside training loops treats that case in detail.

The widget draws illustrative curves with invented coefficients; their gap is a score difference standing in for $\varepsilon(b)$. It opens on a well-grounded evaluator: at a tolerance of 0.10 the gap stays inside it up to b = 1,024, the largest budget tested, so the honest report is "within 0.10 of the reference anchor up to b = 1,024", not "robust to Goodhart". Try two things:

• Lower the tolerance to 0.05: now $b^{*} = 174$. A tested range holds only at the tolerance stated with it.
• Return the tolerance to 0.10 and switch to Overoptimization, shaped after Gao et al.: $b^{*} = 8$, while the reference anchor keeps improving until about b = 17. Where the evaluator stops measuring is not where optimization starts to do harm.

Whatever the curve shows, $b^{*}$ expires. It belongs to one optimizer, budget grid, policy, and reference anchor, and a stronger optimizer, a larger N, a new checkpoint, or a leaked reference anchor calls for a new curve. In training, re-estimate judge error on the policy's outputs at checkpoints; the validation profile shows how $b^{*}$ dates a validity statement.

Keep the reference anchor out of the loop#

An exploitability curve is only as trustworthy as its reference anchor. If the reference anchor shares the evaluator's model family, data, prompts, or evidence, or if its verdicts have flowed back to the optimizer, the curve measures agreement between two coupled channels; see common-mode assurance.

The answer is evaluation separation of powers: one channel for dense development feedback, a second, independent channel that gates releases, and a third, production or audit, channel that checks both against reality, channels that should not share every failure mode. Channels, independence, and anchor burn covers the architecture and the anchor burn ledger that keeps the anchor channel independent.

In practice#

Write an adaptive-validity record for every evaluator whose signal selects, searches, or trains.

1. Name the optimizer and its unit. Best-of-N, search, RL, or developer iteration; record exposure in that unit.
2. Model before you measure. For binary selection, compute $\Pr[C \mid +]$ from measured $t$, $f$, and base rate. If the selector keeps the top score, the binary model does not apply; plan to measure the tail.
3. Choose an independent reference anchor. Adjudicated labels, hidden tests, or a channel on a different mechanism, outside the optimizer's view and modification surface.
4. Predeclare the grid and the tolerance. Fix the budgets, the tolerance on $\varepsilon(b)$, and whether the point estimate or the interval must stay inside it.
5. Run the actual optimizer. At each budget, have the reference anchor judge a sample of the outputs it selects or produces, and estimate $\varepsilon(b)$ with n and an interval.
6. Report $b^{*}$ with its range. State $b^{*}$ and the largest budget tested, and plot proxy against reference anchor.
7. Set re-test triggers. Re-run the curve for a new optimizer, a larger budget, a new checkpoint, or any sign that the reference anchor has leaked.

The artifact is an adaptive-validity record (optimizer, exposure grid, reference anchor, tolerance, $\varepsilon(b)$ with intervals, $b^{*}$ with its tested range, re-test triggers): the exploitability-curve entry of the validation profile.

Common failure modes#

• Natural accuracy quoted for an optimized use — agreement with raters on ordinary outputs justifies a judge as a best-of-64 selector or an RL reward → re-estimate its error on the outputs it selects or trains toward. See reward hacking.
• More samples for a weak evaluator — N is raised because selected outputs are often wrong → lower the false-positive rate; N buys coverage, not precision.
• Timeless robustness — "robust to Goodhart", with no tested range → report $b^{*}$ and the largest budget tested. See Goodharting.
• A coupled reference anchor — it shares the evaluator's model family, or its failures reach developers → build it on a different mechanism and keep it out of the loop. See anchor leakage.

Check yourself#

Analyze. A team reports that its reward model agrees with human raters 85% of the time on a held-out set, and concludes that it is safe to use as an RL reward. Separate what this evidence covers from what an adaptive-validity study would add.

Hint

Held-out agreement is natural-validity evidence about ordinary outputs. The open question is $\varepsilon(b)$ on the policy's outputs as training proceeds, against a reference anchor the policy never trains on, with $b^{*}$ in KL or training steps.

Evaluate. With t = 0.9, f = 0.3, and p = 0.2, one colleague proposes moving from best-of-16 to best-of-256 to raise accuracy; another proposes spending the same budget on halving the evaluator's false-positive rate. Which proposal does the benign model favor, and what would make you distrust its answer?

Hint

Best-of-256 leaves the 42.9% ceiling where it is; f = 0.15 raises it to 60%. Distrust the model if the selector keeps the top score, if candidates are correlated, or if $t$ and $f$ were measured on other outputs.

Create. Design an exploitability study for a learned judge that keeps the best of N answers from DocsAgent. Specify the optimizer, the budget grid, the reference anchor, the tolerance and decision rule, and the sample size per budget.

Hint

Use N as the exposure on a doubling grid, have experts who never see judge scores adjudicate a sample of selected answers at each N, and decide in advance whether the interval or the point estimate must stay inside the tolerance.

Further reading#

• Goodhart pressure and self-confirming loops — why optimized evaluators fail, with and without an exploit.
• Evaluators inside training loops — reward-model overoptimization and on-policy re-estimation in detail.
• Channels, independence, and anchor burn — keeping the reference anchor that measures $\varepsilon(b)$ independent.
• Gao, Schulman, and Hilton (2023), "Scaling laws for reward model overoptimization," ICML — a learned proxy diverging from a stronger reference under optimization.