Exposure, modification, and access

The regime tuple records how an evaluator's signal is used: who consumes it, who optimizes against it, how much optimization it absorbs, what the optimizing loop can change and see, and what each call may cost. Writing it down turns a vague warning about Goodhart's law into an inventory you can check.

Six variables#

The framework writes the regime as a tuple:

$$\mathcal{R} = (\kappa, O, b, S, A_{\text{access}}, K)$$

where:

• $\kappa$, the consumer, is who or what acts on the signal: a report reader, selector, optimizer, release gate, or runtime controller;
• $O$, the optimizer, is who adapts against it: developers, best-of-N selection, search, reinforcement learning (RL), or the system itself;
• $b$, the exposure budget, is how much optimization pressure the evaluator absorbs;
• $S$, the modification surface, is everything the optimizing loop can change;
• $A_{\text{access}}$, the information access, is what flows from the evaluator or its reference anchor back to the optimizer;
• $K$, the cost and latency envelope, is what each evaluator call may cost and how fast it must answer.

In plain words: who reads the signal, who adapts to it, how hard, and what they can touch, see, and afford. Use regimes maps uses onto regimes.

Take SQLAgent, the text-to-SQL agent of Case C, graded by executing its read-only queries against a pinned snapshot of a finance warehouse. An illustrative team runs the same checker in two regimes:

Exposure has no universal unit#

One currency for all optimization pressure, such as KL divergence, is tempting. Best-of-N's idealized distribution shift can be related to $N$, and for KL-regularized RL the policy's KL is a natural exposure axis, but adaptive benchmark queries, developer iterations, and self-modification events are not literally the same unit.

So record $b$ in the optimizer's own unit — candidates, KL, iterations, queries, self-modification events — plus, where possible, the induced shift: a divergence $D(Q_b \,\|\, P_0)$ between the distribution $Q_b$ the evaluator sees after exposure $b$ and the starting distribution $P_0$. A shared axis does not make optimizers interchangeable: measured against the KL each one induced, best-of-n sampling and RL against a learned proxy reward model followed overoptimization curves of different functional forms (Gao et al., 2023; see Evaluators inside training loops).

An exploitability study runs the actual optimizer at increasing budgets and finds the tested exposure $b^{*}$, the largest exposure tested before the evaluator and an independent reference anchor diverge beyond a predeclared tolerance (see Adaptive validity). Validity statements hold only up to that exposure: $b^{*}$ makes adaptive validity expire rather than pretending robustness is timeless.

The modification surface#

The modification surface lists everything the optimizing loop can change: policy, prompts, tools, harness, tests, rubric, judge prompt, judge weights, reference anchors, and environment. A system that can modify only the policy faces a different assurance problem from one that can rewrite the evaluator or theorem statement.

Two members are easy to miss. One is the harness. It is editable, including by the agent itself, so an agent that rewrites its own prompts, tools, memory, or skills modifies itself even with frozen weights (Chen, Wang & Qu, 2026, §2.1–2.2). Count those edits as exposure. The other is the trusted base. A formal guarantee holds relative to a configured trusted base, which includes whether the optimizer can modify its axioms, theorem statements, checker, or environment assumptions: the Goodhart boundary moves to these surfaces rather than disappearing.

When the evaluator is inside $S$, the loop can raise its score by changing the scorer. The extreme case is wireheading, manipulating the reward-producing process itself. The coding agent of Case B shows the everyday version: it writes to the repository that holds the tests, so the grading tests are copied in from outside its workspace afterward.

Information access spends independence#

A reference anchor can sit outside the loop and still lose its independence. Freezing an anchor is not enough if the optimizer can repeatedly query it, and each failure explanation, score, or revealed hidden test reduces independence. The anchor condition therefore has two parts:

$$A \notin S \quad \text{and} \quad I(A \to O) \ \text{is controlled}$$

where:

• $A$ is the independent reference anchor or certification channel;
• $S$ is the modification surface;
• $I(A \to O)$ is the information that flows from that anchor to the optimizer $O$.

In plain words: the optimizer may not change the reference anchor, and what it learns from it is limited and counted. Leaks come through ordinary engineering: certification failures reused as training examples, prompt tuning against hidden sets, repeated benchmark probing. Repeated leaks are anchor leakage, and the independence they cost is anchor burn, tracked in a ledger (see Channels, independence, and anchor burn). That is why SQLAgent's certification split reports only pass or fail.

In practice#

Run this questionnaire before any optimizer is pointed at an evaluator.

1. Consumer. Who or what acts on the signal, and what does it change?
2. Optimizer. Who will adapt against it? Was the evaluator built from the same model family or data as the system being optimized?
3. Exposure. In what unit will you count exposure, what budget do you plan, and how will you measure the induced shift?
4. Surface. What can the loop change? Move tests, rubric, judge, reference anchors, theorem statements, and checker configuration out of its reach.
5. Access. What flows back: scores, failure explanations, examples, rubric details, tests? Is it enough to overfit?
6. Independence. Is there an anchor channel or human audit that is never optimized directly, with every query logged?
7. Cost. What may each call cost, and how fast must it answer? Choose the strongest affordable mechanism for this regime.

The artifact is a regime sheet for each use of the evaluator. It fills the regime field of the evaluator dossier and the regime and tested-exposure clauses of the validity statement.

Common failure modes#

• Robustness without a budget — "robust to Goodhart", with no optimizer, unit, or reference anchor → report the optimizer, the exposure in its own unit, and the tested range up to $b^{*}$. See Goodharting.
• Evaluator inside the surface — the loop can edit the tests, rubric, or judge prompt → move them out of $S$ and diff-check them after every run. See wireheading.
• Uncounted harness edits — an agent rewrites its tools or memory between runs and nobody records it → treat harness changes as self-modification and count them in $b$.

Check yourself#

Analyze. A coding agent can edit its own prompts, tools, and memory, runs tests it can read, and sees a hidden certification split's pass rate after every attempt. Write down its $S$ and $A_{\text{access}}$, and mark where the anchor condition fails.

Hint

Readable tests can be overfit; tool and memory edits are harness self-modification; a per-attempt certification score is information flowing from the reference anchor to the optimizer.

Evaluate. A team reports that its judge "held up after 50 rounds of prompt tuning." What would you need before accepting that as adaptive-validity evidence?

Hint

An independent reference anchor scored at several budgets, a predeclared tolerance, the resulting $b^{*}$, and what the developers saw in each round.

Create. Fill in a regime sheet for a rubric-guided judge that will label preference pairs to train a model from its own family.

Hint

Count exposure in the training run's own unit and keep the rubric and judge outside $S$. Same-family judging couples generator and evaluator errors, so keep a cross-family or human-audited anchor channel out of the training objective.

Further reading#

• Use regimes: same judge, four jobs — how uses map to regimes and how signals migrate.
• Channels, independence, and anchor burn — separate development, certification, and production evidence, with an exposure ledger.
• Adaptive validity — measuring the exploitability curve and $b^{*}$.
• Self-improvement — when the surface includes the evaluator itself.
• Chen, Wang & Qu (2026), "Recursive self-improvement in AI" — §2 on harnesses as self-modifiable components.