# Goodhart pressure and self-confirming loops

> Reward hacking needs an exploit; self-confirmation needs only a shared blind spot. Tell them apart and measure the coupling behind both.

**Goodhart pressure** is the degradation of the relation between a proxy and its construct caused by optimizing against the proxy. This chapter separates two routes to it. Reward hacking needs an exploit that optimization finds; a self-confirming loop needs only a blind spot that generator and evaluator share.

> [!KEY] In one sentence
> Reward hacking needs an exploit and self-confirmation needs only a shared blind spot, so measure the coupling behind both: between the generator and its evaluator, and among the channels meant to catch what each misses.

## What optimization does to a proxy

Once a score gates releases, tops a leaderboard, or rewards training, developers and models search for ways to raise it, in several forms:

| Form | What gets optimized | Example |
|---|---|---|
| Verbosity exploitation | An LLM judge's preference for length | The agent learns the judge's verbosity bias |
| Test-suite overfitting | The visible tests instead of the behavior they sample | A patch passes the suite and fails on the next input *(illustrative)* |
| Benchmark leakage | The items themselves, or close variants | Implausibly high scores that do not transfer to production |
| Superficial rubric cues | Surface features a rubric rewards | A "conciseness" check breeds truncation |
| High-scoring wrong artifacts | The evaluator's false-positive tail | Best-of-N keeps a fluent answer that is wrong *(illustrative)* |
| Reward hacking and wireheading | A flaw in the evaluator, or the reward process itself | A coding agent edits the test fixtures that grade it |

Pressure acts on people too: the team learns which slices are cheap to win, and [regime migration](https://benchmaker.ai/glossary/regime-migration) turns a reporting benchmark into a target after repeated tuning ([Use regimes](https://benchmaker.ai/handbook/regime/use-regimes)).

Pushed far enough, the pressure produces **overoptimization**: continued optimization improves the proxy score after true or reference-anchor quality has plateaued or declined. [Gao et al. (2023)](https://benchmaker.ai/handbook/reference/sources#gao-2023) demonstrated it with learned [reward models](https://benchmaker.ai/glossary/reward-model); [Evaluators inside training loops](https://benchmaker.ai/handbook/regime/training-loops) covers that case, and [Adaptive validity](https://benchmaker.ai/handbook/pressure/adaptive-validity) shows how to measure where it begins.

## Reward hacking needs an exploit

**Reward hacking** is behavior that obtains high measured reward while violating the intended objective, a special case of evaluator exploitation: the optimizer finds something the evaluator gets wrong and climbs it. The exploit can come from a misspecified reward, a judge's blind spots, incomplete tests, or access to the reward channel itself.

In agent evaluation its everyday form is grader gaming: hard-coded answers that match string checks, fabricated citations that pass format checks, and edited test fixtures in coding tasks. The [coding-agent case](https://benchmaker.ai/handbook/cases/coding-agent) shows the remedy, which is to constrain the routes to a pass until the only one left runs through the actual work.

**Wireheading** is the extreme case, in which the system manipulates the reward-generating process instead of doing the task. It is a fact about the modification surface, so the defense is architectural: keep tests, rubrics, judge prompts, and reference anchors outside what the optimizing loop can change ([Exposure, modification, and access](https://benchmaker.ai/handbook/regime/exposure-and-surfaces)).

Reward hacking leaves a signature. The optimized score keeps rising while an independent [reference anchor](https://benchmaker.ai/glossary/anchor) stalls or falls, a divergence the framework reads as likely proxy corruption or overoptimization. Red-teaming the grader and running exploitability studies look for the exploit before the optimizer finds it ([Falsification and failure regions](https://benchmaker.ai/handbook/pressure/falsification)).

## Self-confirmation needs only a shared blind spot

A **self-confirming loop** forms when generator and evaluator share correlated blind spots, so that selection or training amplifies their common errors. Nobody has to find an exploit. The framework draws the line between the two routes: reward hacking is a special case in which exploitation of the evaluator is explicit, while self-confirmation requires no adversary, only correlated bias. The evaluator only has to be wrong in the same direction as the generator, which shared weights, data, or inductive biases make likely.

[Intrinsic signals](https://benchmaker.ai/glossary/intrinsic-signal) such as confidence, self-consistency, and ungrounded self-critique are the most exposed, because generator and evaluator can share the same model assumptions. The RSI survey describes the mechanism in self-rewarding RL: when the reward tracks the model's own confidence, the mistakes the model is surest about earn the most credit, and training entrenches them ([Tan et al., 2025](https://benchmaker.ai/handbook/reference/sources#tan-2025), via [Chen, Wang & Qu, 2026](https://benchmaker.ai/handbook/reference/sources#chen-wang-qu-2026), §5.3). Selection has the same structure: a same-family judge picking among a policy's answers no longer has the calibration set's error rates on them, the cascade drifts toward the shared blind spot, and cross-family judging is the structural defense.

The framework's diagnostic is **generator coupling**:

$$
\Pr[E \text{ errs} \mid G \text{ errs}] \;\;\text{vs.}\;\; \Pr[E \text{ errs}]
$$

where:

- $E$ is the evaluator and $G$ the generator whose outputs it judges;
- "$G$ errs" means that an output fails the specification according to adjudicated labels;
- "$E$ errs" means that the evaluator's verdict disagrees with those labels, which on a failed output is a false accept.

In words: if the evaluator is more likely to be wrong exactly when the generator is wrong, it is not an independent check. Large positive dependence warns that the evaluator is not an independent corrective channel.

Read the diagnostic by error type. On a failed output the only possible error is a false accept, so the conditional side is a false-accept rate, while $\Pr[E \text{ errs}]$ mixes false accepts with false rejects; an evaluator that accepts failures more often than it rejects passes shows a gap even with no coupling at all. Holding the error type fixed avoids the confound: compare the evaluator's false-accept rate on this generator's failures with its rate on failures from other sources, such as other model families. That comparison asks the framework's question, whether the evaluator's errors concentrate on the generator's errors or on particular model families, and tests the warning that a same-family judge's error rates do not carry over to its own family's outputs. An illustrative audit of two judges on 400 adjudicated outputs from one policy, 100 of them failures, plus 100 adjudicated failures from other models (rates with 95% Wilson intervals):

| | Same-family judge | Cross-family judge |
|---|---|---|
| $\Pr[E \text{ errs}]$ on all 400 policy outputs | 11% (8–14%) | 10% (7–13%) |
| $\Pr[E \text{ errs} \mid G \text{ errs}]$: false accepts on the policy's 100 failures | 30% (22–40%) | 12% (7–20%) |
| False accepts on 100 failures from other models | 10% (6–17%) | 11% (6–19%) |

The two judges err at similar overall rates, but the same-family judge's false-accept rate triples on its own family's failures, with intervals that do not overlap; the cross-family judge's barely moves. Used as a selector or a reward, the first judge would amplify the policy's blind spot. Generator coupling is [differential misclassification](https://benchmaker.ai/glossary/differential-misclassification) with a cause: the error rates differ on this generator's outputs because the evaluator shares its blind spot ([The judge as a noisy sensor](https://benchmaker.ai/handbook/validation/judge-as-noisy-sensor)).

## Do the holes line up?

The defense against both routes is a portfolio: partially independent instruments, holdouts, audits, and production feedback rather than one immortal score. Validation theory's name for it is the **Swiss cheese model**, in which every validation method is a slice with holes, the limitations of that method, and assurance comes from stacking slices so that no hole runs clean through ([Kochenderfer et al., 2026](https://benchmaker.ai/handbook/reference/sources#kochenderfer-2026)).

The stack is only as good as the independence of its slices: multiple evaluators provide little assurance if their blind spots are perfectly correlated. The framework's measurement is the joint miss rate, the share of true failures that every channel misses. If two channels missed independently, their joint miss rate would be the product of their miss rates; at worst, when one channel misses only failures the other also misses, it equals the smaller of the two. An illustrative check for the grounded-answer criterion of DocsAgent, the documentation agent of [Case G](https://benchmaker.ai/handbook/cases/docs-agent), on 200 adjudicated ungrounded answers (measured joint misses with 95% Wilson intervals):

| Channel pair | Miss rates | If independent | Joint miss |
|---|---|---|---|
| Two judges from one model family, different prompts | 12% and 10% | 1.2% | 9.0% (5.8–13.8%) |
| The first judge plus a cross-family judge shown the retrieved passages | 12% and 14% | 1.7% | 2.5% (1.1–5.7%) |

The second judge in the first pair adds almost nothing: 9% of ungrounded answers pass both, against 12% for the first alone. The cross-family judge is worse on its own and far better as a second slice. Near-duplicate channels fail together, which the framework calls common-mode assurance; [channel coupling](https://benchmaker.ai/glossary/channel-coupling) among development, certification, and production channels is measured the same way ([Channels, independence, and anchor burn](https://benchmaker.ai/handbook/lifecycle/channels-and-independence)).

> [!WARNING]
> Majority vote across judges reduces independent noise; it does not cure correlated bias, and judges derived from similar training data, model families, or prompts can form a more confident shared blind spot. Measure a panel's joint miss rate before counting its votes as independent evidence. See [common-mode assurance](https://benchmaker.ai/handbook/practice/anti-patterns#common-mode-assurance).

## When a loop trains on its own outputs

When a loop trains on what it generates, correlated errors accumulate in the data as well as the reward. **Model collapse** is the degradation of distributional quality or capability under recursive training on generated data; **diversity collapse** is the homogenization of behavior or tasks in self-play and self-generated-data loops.

The RSI survey describes whether such loops keep improving or degrade as the most disputed empirical question in the area (Chen, Wang & Qu, 2026, §5.3). On the pessimistic side, models trained recursively on their own outputs lose the tails of their distribution and degenerate ([Shumailov et al., 2024](https://benchmaker.ai/handbook/reference/sources#shumailov-2024), via Chen, Wang & Qu, 2026, §5.3). On the optimistic side, collapse is an engineering problem: one systematic study argues that self-play survives when data gating (what enters the training set) and reward grounding (what ties the signal to reality) are managed as separate levers, and collapses by default when either fails (Chen, Wang & Qu, 2026, §4.4). The survey's synthesis is that pure closed loops degrade, but no practical system has to run one (Chen, Wang & Qu, 2026, §5.3).

Diversity collapse has its own signs: problem proposers drift toward the few kinds of problem the reward favors, leaving the solver a thinning curriculum, and in one reported campaign single-attempt accuracy rose and then collapsed with no shift in the task distribution (Chen, Wang & Qu, 2026, §5.3). Against model collapse, gate data and ground rewards externally; against diversity collapse, track diversity explicitly and refresh task sources ([Anti-patterns](https://benchmaker.ai/handbook/practice/anti-patterns#model-collapse)). [Self-improvement and evaluator co-evolution](https://benchmaker.ai/handbook/agents/self-improvement) follows these dynamics into loops that can change their own evaluator.

> [!OPEN]
> How much external grounding is enough? Theory says pure closed loops degrade, and practice mixes in external signal ad hoc; no one has characterized the minimum rate of exogenous signal that sustains improvement, the [exchange rate of grounding](https://benchmaker.ai/glossary/grounding-exchange-rate) (Chen, Wang & Qu, 2026, §5.3, §8). This handbook does not propose one.

## In practice

The framework answers Goodharting with four responses: independent reference anchors, multiple channels, adversarial search, and explicit exposure limits. Turn them into a coupling audit for every evaluator under optimization.

1. **Map the loops.** For every evaluator, record who optimizes against it, how, and what the loop can modify. Move tests, rubrics, judge prompts, and reference anchors out of that surface.
2. **Check for shared origins.** Note whether evaluator and generator share a model family, training data, prompts, or evidence.
3. **Measure generator coupling.** Compare the evaluator's false-accept rates on adjudicated failures from the generator and from other models, with intervals.
4. **Measure joint miss rates.** For every pair of channels meant to back each other up, compare the share of true failures both miss with the product of their miss rates.
5. **Search for exploits.** Red-team the grader, and run the optimizer against an independent reference anchor at increasing exposure ([Adaptive validity](https://benchmaker.ai/handbook/pressure/adaptive-validity)).
6. **Limit and log exposure.** Cap how often reference anchors are queried and what flows back; keep rotating held-out checks that never gate a decision, and read transcripts on a schedule.
7. **Watch the loop's own data.** Where a loop trains on its outputs, track the diversity of its tasks and outputs and the share of externally grounded signal.

The artifact is a coupling audit: generator-coupling and joint-miss estimates with intervals, the exposure limits in force, and the exploits found. It fills the generator-coupling and channel-coupling entries of the [validation profile](https://benchmaker.ai/glossary/validation-profile).

## Common failure modes

- **Self-judging** — a judge from the policy's own model family grades its outputs, and the score climbs steadily → measure generator coupling and add a cross-family or execution-grounded channel. See [self-confirming loop](https://benchmaker.ai/handbook/practice/anti-patterns#self-confirming-loop).
- **Redundant panels** — three judges from one family vote, and their agreement is reported as confidence → measure joint miss rates, and vary the mechanism or the evidence view, not only the prompt. See [common-mode assurance](https://benchmaker.ai/handbook/practice/anti-patterns#common-mode-assurance).
- **A writable evaluator** — the optimizing loop can edit tests, rubrics, or judge prompts → move evaluator artifacts outside the modification surface. See [wireheading](https://benchmaker.ai/handbook/practice/anti-patterns#wireheading).
- **Closed-loop narrowing** — a self-training loop's accuracy rises while its tasks and outputs narrow → gate data, ground rewards externally, and track diversity. See [diversity collapse](https://benchmaker.ai/handbook/practice/anti-patterns#diversity-collapse).

## Check yourself

**Analyze.** A coding agent writes its own unit tests and keeps the patches that pass them. Sort the risks into reward hacking and self-confirmation, and name a measurement that tells them apart.

> [!DETAILS] Hint
> Model-written tests can share the misconceptions of the code they were written alongside (Chen, Wang & Qu, 2026, §3.2), a self-confirming route that needs no exploit; the agent can also learn to write tests that are easy to pass. Score the kept patches against reference tests the agent never sees.

**Evaluate.** A team adds a second LLM judge "for independence". Both judges come from one provider's model family and see the same evidence. On 200 adjudicated failures they miss 10% and 9%, and both miss 7%. Is the second judge worth its cost, and what would you change?

> [!DETAILS] Hint
> Independent misses would give about 0.9%. At 7%, the pair behaves almost like one judge: change the mechanism or the evidence view, not the prompt.

**Create.** Design a generator-coupling study for a judge that will serve as the RL reward for a policy from its own model family. Specify the samples, the adjudication, the comparison, and the threshold of concern at which you would replace the judge.

> [!DETAILS] Hint
> Sample failures from the policy and from other model families, adjudicate them blind to the judge's verdicts, compare false-accept rates with intervals, and fix in advance the gap that triggers a cross-family or execution-grounded replacement.

## Further reading

- [Adaptive validity](https://benchmaker.ai/handbook/pressure/adaptive-validity) — measuring where Goodhart pressure starts to bite.
- [Channels, independence, and anchor burn](https://benchmaker.ai/handbook/lifecycle/channels-and-independence) — channel coupling across development, certification, and production.
- [Self-improvement and evaluator co-evolution](https://benchmaker.ai/handbook/agents/self-improvement) — self-confirming loops when the evaluator itself evolves.
- [Anti-patterns and failure vocabulary](https://benchmaker.ai/handbook/practice/anti-patterns) — all seventeen failure names, with their tells and fixes.
- Chen, Wang, and Qu (2026), "Recursive self-improvement in AI" — §5.3 for self-confirming loops, model collapse, and diversity collapse.
- Gao, Schulman, and Hilton (2023), "Scaling laws for reward model overoptimization," *ICML* — the reference demonstration of overoptimization.

---
Source: https://benchmaker.ai/handbook/pressure/goodhart-and-self-confirming-loops