Use regimes: same judge, four jobs

A use regime is the causal role an evaluator's signal plays: it measures (supports an inference or a decision), optimizes (shapes candidates or a policy), or monitors and controls (watches a deployed system and intervenes). The same evaluator, unchanged, needs different evidence in each regime, and its signal can drift from one regime into another.

Three regimes#

The framework separates three primary regimes because the same instrument changes meaning when its signal enters a different causal loop. Each asks its own question and fails in its own way:

Take DocsAgent, the documentation agent of Case G, whose answers must be grounded in the applicable documentation. Part of its grounding criterion is checked by an entailment judge, a learned judge that decides whether each material claim is entailed by the documentation span it cites. In an illustrative deployment, the same judge measures in a monthly quality review, optimizes when it keeps the best of several drafted answers, and controls when it holds back an ungrounded answer before a user sees it. Its weights and prompt never change; what its verdict causes does.

The regime is more than a label. The framework records it as six variables, $\mathcal{R} = (\kappa, O, b, S, A_{\text{access}}, K)$: the consumer $\kappa$ who acts on the signal, the optimizer $O$ who adapts against it, the exposure budget $b$, the modification surface $S$ the optimizing loop can change, the information access $A_{\text{access}}$ that flows back to the optimizer, and the cost and latency envelope $K$ of each call. A measuring signal has, by design, no optimizer; an optimizing signal exists to have one. Exposure, modification, and access treats each variable in detail.

Six uses, three regimes#

A regime covers several uses. Four uses of one learned judge's score — reporting, selection, search, and training — put increasing optimization pressure on it from the first to the fourth; the release gate and runtime control bring the uses to six. This table is the handbook's single mapping between uses and regimes:

The pressure ranks order only those four uses. For DocsAgent's judge, the remaining uses would be a threshold that a new retrieval configuration must clear before release, revising a draft until the judge passes it, and fine-tuning the model on the judge's verdicts.

Regime migration#

Regime is not permanently attached to an evaluator. Regime migration is a change in the causal role of a signal while the evaluator itself stays the same. The framework names three routes:

• A public benchmark used for reporting becomes a de facto optimization target after repeated tuning.
• A production monitor becomes part of optimization when training updates react to its alerts.
• A hidden certification test loses independence when failure details are repeatedly fed back to developers.

Migration is to be expected of any evaluator whose score matters: once a score becomes a release gate, leaderboard target, reinforcement-learning (RL) reward, or training objective, developers and models search for ways to raise it, and the measurement procedure is now part of the causal environment.

From this follows a general causal rule: classify a signal by what downstream behavior it changes, not by the dashboard on which it is displayed. A release gate that developers run after every prompt change, keeping the changes that pass, is selecting prompts. It has become an optimization signal, although it still appears on the release dashboard.

The rule also says what to do next: the signal's validation obligations migrate with it. A gate that has become a development target needs adaptive-validity evidence, not only the natural error rates that justified it as a measurement, and the release decision needs a fresh, independent channel. For SQLAgent, the text-to-SQL agent of Case C, which queries a pinned snapshot of a finance warehouse, migration can start small: developers fix one failed certification query, then a few more. Certification failures used as training examples, prompt tuning against hidden sets, and repeated probing each spend some of the set's independence, a loss called anchor burn.

Development and certification differ#

Because regimes ask different questions, one evaluator rarely serves all of them well. The Berkeley lecture on agent evaluation makes the point across a system's life (Song, 2025, slide 5). Signals for training and development should be very fast, very cheap, and free of shortcuts. Benchmarks for publication should be standardized and reproducible, and able to tell close systems apart. Evaluation for deployment should be trustworthy, task-specific, and absolute.

The practical answer is a separation of powers: a dense development channel that may co-evolve with the system, an independent certification channel that is queried sparsely and never trained on, and a production channel that checks both against real outcomes, each with the strongest mechanism affordable in its regime. Channels, independence, and anchor burn shows how to build and audit them.

One judge, four jobs#

Take one learned judge with high human agreement and ask what more evidence it needs before serving as a benchmark reporter, a best-of-256 selector, an RL reward, and a production gate — and why the answers differ although the judge's weights are identical. The short answer: used once for reporting, a judge is a sensor; used for best-of-N, a search objective; used as a policy reward, a training target. Same model, different causal regime.

• Reporter (measure). The judge is a noisy sensor. Its reading needs true- and false-positive rates measured on adjudicated samples of each compared system's outputs, a correction, and an interval. A system tuned against the judge can carry different error rates, which turns an exploit into an apparent gain; see The judge as a noisy sensor.
• Selector (optimize). Even under a benign model with fixed error rates, more candidates raise the chance that something passes, not what a pass means. With scalar scores, best-of-256 picks from the extreme tail, where precision can deteriorate, improve, or plateau and ordinary calibration does not say which. The evidence needed is tail behavior at the actual $N$; see Adaptive validity.
• Reward (optimize). Sustained training searches the judge's blind spots. The evidence needed is an exploitability curve: run the actual optimizer at increasing budgets, score its outputs with an independent reference anchor, and re-estimate the judge's error on the current policy; see Evaluators inside training loops.
• Production gate (monitor-control). The judge now faces a rare base rate, live drift, and a latency budget. The evidence needed is alert precision and calibration at the production base rate, the cost of false alarms, and drift monitoring; see Monitoring and control in production.

The widget runs one illustrative judge, with a true-positive rate of 0.90 and a false-positive rate of 0.10, through all four jobs. Try three things:

• On Report, the judge reads 58% for a system that truly passes 60%, and the correction recovers 60%. Turn on System was tuned against this judge, and the corrected estimate rises to 67.5% although nothing real changed.
• On Select best-of-N, with one candidate in five correct, a passed pick is correct about 69% of the time at $N = 16$. Drag $N$ to 256: the pool now almost surely contains a correct answer, and the pick is still correct only about 69% of the time.
• On Production gate, the same judge watches 100,000 requests a day for an event with a base rate of 1 in 1,000. It raises about 10,080 alerts a day, of which 90 are real: 0.89%.

In practice#

Keep a regime register for every evaluator signal your team produces.

1. List the signals. Include dashboard tiles, CI checks, leaderboard scores, reward functions, guardrail thresholds, and alerts.
2. Name the behavior each one changes. Write who consumes it and what they do differently because of it. A signal that changes nothing is eval theater: attach it to a decision or retire it.
3. Classify by that behavior. Assign the use and the regime from the mapping table, whatever dashboard the signal appears on.
4. Record the regime variables. For each optimized signal, name the optimizer, the exposure so far, what the loop can modify, and what it can see.
5. Match the evidence to the regime. For measure, natural error rates and intervals; for optimize, adaptive validity measured with the actual optimizer; for monitor-control, precision and calibration at the production base rate.
6. Write the migration triggers. Note what would move each signal to another regime — developers iterating against a gate, failure details leaving a hidden set, training reacting to alerts — and who re-classifies it when that happens.

The artifact is a regime register: one row per signal, with its use, regime, regime variables, required evidence, and migration triggers. It fills the regime field of the evaluator dossier and the consumer and regime clauses of each validity statement.

Common failure modes#

• Unnoticed migration — the evaluator is unchanged, but developers now hill-climb it or training now reacts to it → classify the signal by the behavior it changes and move its validation obligations with it. See Goodharting.
• Reporter promoted to reward — a judge validated by human agreement on ordinary outputs becomes an RL reward with no new evidence → measure an exploitability curve with the actual optimizer before trusting the gains. See reward hacking.
• Leaky certification — failure details from a hidden set reach the people or loops being certified → report pass or fail only, log every query, and refresh the set when its independence is spent. See anchor leakage.
• A signal with no consumer — scores appear in reviews, but no decision cites them → write the decision first; if there is none, retire the signal. See eval theater.

Check yourself#

Analyze. List every signal your team's evaluation stack emits, from CI checks to alerts. Classify each by the downstream behavior it changes, then mark those whose regime differs from what the dashboard they appear on suggests.

Hint

Ask of each signal: if its value changed tomorrow, what would someone or something do differently? A release-dashboard tile that developers consult before merging prompt changes is a selection signal.

Evaluate. A learned judge agrees well with adjudicated labels on ordinary outputs. A colleague proposes using it first as a reporter, then as a best-of-256 selector, then as an RL reward. For each step, say what new evidence you would require and why the judge's agreement does not supply it.

Hint

The agreement was measured before any optimization. Pressure increases at each step, and the evidence moves from error rates per system, to tail behavior at the actual $N$, to an exploitability curve against an independent anchor.

Create. Your hidden certification set gates weekly releases, and the developers have started asking which tasks failed. Redesign the regime variables, channels, and reporting so that release decisions stay in the measure regime.

Hint

Decide what flows back, $A_{\text{access}}$: a pass or fail per release, not per-task failures. Give developers their own dense development split, log every certification query, and set a rule for refreshing the set once its independence is spent.

Further reading#

• Exposure, modification, and access — the six regime variables in detail, with a questionnaire to run before optimization begins.
• Adaptive validity — how evaluator error grows under best-of-N and training, and how to measure it.
• Evaluators inside training loops — where the learned judge sits in RLHF, RLAIF, RLVR, and DPO.
• Monitoring and control in production — base rates, calibration, and the moment a monitor becomes a training signal.
• Song (2025), "Agent evaluation & project overview," UC Berkeley Agentic AI lecture — slide 5 on what evaluation needs at each stage of a system's life.