Evaluators inside training loops

A training loop turns an evaluator's signal into parameter updates. RLHF, RLAIF, RLVR, and DPO are training pipelines that consume evaluator signals; none of them is an evaluator. Inside each sits a reward model, an external checker, or a set of preference labels, under sustained optimization pressure.

Pipelines, not evaluators#

Placed on the construction stack, RLHF comes apart: the reward model is the evaluator mechanism, its scalar is the signal, and PPO-style policy optimization is the use. "RLHF" names the whole optimization pipeline, not the learned judge alone. The same holds for the other common pipelines:

In self-rewarding training, one model both writes the responses and scores them (Chen, Wang & Qu, 2026, §4). None of these names says whether the final policy is good; each says how evaluator signals were consumed, and the evaluator inside needs its own validity argument. Case E, reasoning-model training, combines evaluators: RLVR on final-answer correctness where exact checking is possible, and optionally a process reward model to localize poor intermediate states.

Reward models learn an ordering#

A reward model is a learned model that maps candidates to scalar or pairwise preference estimates, and it is often optimized against. A common way to train one uses a Bradley–Terry model. For a prompt $x$, a preferred response $y_w$, and a less-preferred response $y_l$:

$$\Pr[y_w \succ y_l \mid x] = \sigma(r_\phi(x, y_w) - r_\phi(x, y_l))$$

where:

• $r_\phi(x, y)$ is the reward model's score for response $y$ to prompt $x$, with parameters $\phi$;
• $y_w \succ y_l$ means that the labeler preferred $y_w$ to $y_l$;
• $\sigma(z) = 1/(1 + e^{-z})$ is the logistic function.

In words: the larger the score gap between two responses, the more confidently the model predicts which one the labeler preferred. Training minimizes $\mathcal{L}_{\text{RM}}(\phi) = -\mathbb{E}\big[\log \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big)\big]$ over the labeled pairs.

Such training learns a preference ordering, not a proof of quality. Two consequences follow. Only score differences enter the model, so adding a constant to every score changes nothing: a reward model's absolute values carry no meaning of their own. And the ordering is learned from data, so the reward model has an empirical validity domain, with errors that depend on training data, prompt, generator distribution, language, domain, and optimization pressure.

Suppose, illustratively, that the team behind DocsAgent, the documentation agent of Case G, trains a reward model on documentation experts' pairwise preferences between answers. It learns which answers the experts prefer, as far as their pairs reveal it; it does not check that each claim is entailed by its cited span, which stays the job of the grounding criterion's checks.

The KL term is an exposure control#

RLHF wraps the reward model in four stages: supervised fine-tuning on human demonstrations, preference data from ranked model outputs, a reward model trained on those preferences, and policy optimization that maximizes reward plus a regularizer. The standard schematic is a KL-regularized objective:

$$\begin{aligned} \max_\theta\; & \mathbb{E}_{x,\, y \sim \pi_\theta}\big[r_\phi(x, y)\big] \\ & - \beta\, \mathbb{E}_x\Big[D_{\mathrm{KL}}\big(\pi_\theta(\cdot \mid x) \,\|\, \pi_{\text{ref}}(\cdot \mid x)\big)\Big] \end{aligned}$$

where:

• $\pi_\theta$ is the policy being trained, with parameters $\theta$;
• $\pi_{\text{ref}}$ is a fixed reference policy, such as the fine-tuned starting point;
• $r_\phi$ is the reward model;
• $D_{\mathrm{KL}}$ is the Kullback–Leibler divergence between the two policies' output distributions for prompt $x$;
• $\beta$ sets the price of that divergence.

In words: seek high reward, but pay for every step away from the reference policy. The KL penalty limits policy drift while the policy searches for high reward.

In regime terms, the KL term is an exposure control. It limits how far the policy can move into regions where the reward model was never validated, and the realized KL from the reference is the natural exposure axis for this optimizer (see Exposure, modification, and access). A KL budget bounds how far the policy may travel; it does not certify that the reward stays valid along the way.

Two variants change the optimizer or the labeler, not the measurement problem. DPO trains the policy directly on preference pairs, without a separately optimized reward model and RL loop (Rafailov et al., 2023). The labels still encode a human or AI judgment process, so the questions remain what they mean, how reliable they are, and where they generalize. RLAIF uses AI-generated feedback instead of, or alongside, human feedback, which makes the labeler a learned judge too. When it shares a model family with the policy, their errors correlate and the loop can drift toward shared blind spots (Chen, Wang & Qu, 2026, §5.3); cross-family judging is the structural defense.

Grounded rewards are still incomplete#

RLVR replaces the learned proxy with an externally checkable signal. In the simplest binary case:

$$r_{\text{RLVR}}(x, y) = \mathbb{1}\big[V_\psi(x, y) = 1\big]$$

where:

• $V_\psi(x, y)$ is the verdict of the external check for specification $\psi$ on response $y$ to task $x$: an exact or symbolic answer checker, a test suite, or a proof checker, of which only the last is a verifier in the strict, soundness-backed sense;
• $\mathbb{1}[\cdot]$ is 1 when the condition holds and 0 otherwise.

In words: the reward is 1 when the check accepts the answer and 0 otherwise. What matters is not that the reward is binary but that the check has an external grounding channel stronger than a learned preference proxy.

Grounded is not complete. A test suite samples behavior: passing it establishes behavior on the tested inputs, not the whole input domain, and an optimizer can exploit untested regions or incidental test structure. A final-answer check rewards a flawed derivation that happens to reach the right answer: keeping training solutions only because their final answers are right lets guesses built on wrong reasoning through (Zhang et al., 2024, via Chen, Wang & Qu, 2026, §4.1). In Case E, a lucky derivation that lands on the right number earns the same reward as a sound one. Nor does a verifiable reward make the loop stable: in one study of REINFORCE post-training for code, single-attempt success rose and then collapsed within a run despite a genuinely verifiable binary reward, and KL and EWC-style regularization did not prevent it (Lin, 2026, via Chen, Wang & Qu, 2026, §4.1).

Denser process rewards help with credit assignment, and step-level supervision can outperform outcome-only supervision for training mathematical reasoning (Lightman et al., 2023). The usual shortcut to step labels scores a step by the fraction of rollouts from it that succeed, which is cheap and quietly circular, since the labels inherit the current policy's own blind spots. Treat rollout-labeled PRM scores as policy-conditioned value estimates, which change whenever the policy does (see Signals and resolution).

The weakest grounding is the policy's own judgment. In self-rewarding RL, rewards tied to the model's own confidence give the most credit to the mistakes it is surest of, so training entrenches them (Tan et al., 2025, via Chen, Wang & Qu, 2026, §5.3). That is a self-confirming loop: unlike reward hacking, it needs no exploit, only correlated bias. Goodhart pressure and self-confirming loops treats such loops in general, including the open question of how much externally grounded signal keeps one improving.

Reward-model overoptimization#

Optimized against, a learned evaluator stops being a fixed instrument: the evaluator becomes a search landscape, so every blind spot becomes a possible optimization direction. Overoptimization is the outcome: continued optimization keeps improving the proxy score after true or reference-anchor quality has plateaued or declined.

Gao et al. (2023) measured it with the reference anchor built in. In a synthetic setup, a fixed reward model played the role of human labelers and supplied the labels on which proxy reward models were trained; policies were then optimized against a proxy, by best-of-n sampling or by RL, while the fixed model scored the results. The proxy score kept rising after the fixed model's score peaked and began to fall (Chen, Wang & Qu, 2026, §5.1), and the relationship took a different functional form for each optimization method.

High validation accuracy does not protect against this. A reward model's accuracy on held-out preference pairs is measured on the distribution its labels came from, before any optimization, and optimizing against it changes the distribution of inputs it sees. A judge calibrated on ordinary outputs may be badly miscalibrated on outputs selected to maximize it. The remedy is an exploitability curve: run the actual optimizer at increasing budgets, score its outputs with an independent reference anchor, and find $b^{*}$ (Adaptive validity).

The widget draws an illustrative best-of-$N$ curve in the functional form Gao et al. report for best-of-n, with invented coefficients; $b$ counts candidates. At a tolerance of 0.10, the proxy stays within tolerance of the independent reference anchor only up to $b^{*} = 8$, the reference anchor's score peaks near $b = 17$ and then falls, and at $b = 1{,}024$ the gap is 0.47. Try two things:

• Set Largest budget tested to $b = 8$. The gap never exceeds the tolerance, so all the study can claim is $b^{*} \ge 8$: a small study cannot see the cliff.
• Raise the tolerance to 0.20. Now $b^{*} = 32$, past the reference anchor's peak: a loose tolerance can certify budgets at which optimization already does harm.

Density for development, independence for certification#

A training loop needs an evaluator that is cheap, dense, and fast; certification needs one the loop has never touched. The training reward and the evaluation instrument should not be identical forever. For reasoning-model training, Case E uses dense RLVR or learned-process feedback for development, keeps a separate certification channel that is not optimized directly, and re-estimates judge error on-policy at checkpoints. Channels, independence, and anchor burn shows how to keep the two channels apart.

In practice#

Run this for every evaluator you put inside a training loop.

1. Map the pipeline. Name the evaluator inside it — reward model, AI labeler, answer checker or test suite, preference labels, or the policy itself — and its signal.
2. Record the regime. Write down the optimizer, its exposure unit (KL from the reference policy, candidates, or training steps), and what the loop can modify and see.
3. Measure natural error first. Report the reward model's or judge's agreement with adjudicated labels by slice, or the checker's coverage of the specification.
4. Keep an independent anchor channel. Hold out an external benchmark and an expert audit that never enter the reward and are never used for tuning.
5. Checkpoint the gap. At each checkpoint, score the policy's outputs with both the training evaluator and the anchor channel, and re-estimate the training evaluator's error on the current policy's outputs.
6. Stop at a predeclared tolerance. When the training evaluator and the anchor channel diverge beyond it, stop or refresh the evaluator, and record $b^{*}$ and the anchor-channel queries spent.

The artifact is a training-evaluator log: the pipeline map, the exposure record, and a checkpoint table of proxy score, anchor-channel score, and on-policy evaluator error. It is the exploitability entry of the validation profile for this regime.

Common failure modes#

• The pipeline offered as evidence — "trained with RLHF" or "rewarded by a verifiable signal" is cited as proof of quality → name the reward model or checker inside and validate it as an instrument.
• Proxy up, anchor channel flat — the reward keeps rising while held-out quality stalls or falls → stop at the predeclared tolerance and refresh the evaluator. See reward hacking.
• Calibration from before training — judge error measured on the starting policy's outputs is trusted at the end of training → re-estimate it on-policy at each checkpoint. See Goodharting.
• Shared blind spots — a same-family labeler or a self-reward drives training → measure generator coupling and add a cross-family or externally grounded channel. See self-confirming loop.

Check yourself#

Analyze. In an RLHF pipeline, place the preference data, the reward model, the RL optimizer, and the final holdout evaluation on the construction stack, and give the regime of each signal.

Hint

The preference pairs represent the specification; the reward model is the mechanism and its scalar the signal; policy optimization is the use. A holdout evaluation that is never trained on stays in the measure regime.

Evaluate. A reward model scores high accuracy on held-out preference pairs, yet the policy trained against it gets worse by expert review. How can both be true?

Hint

The accuracy was measured on the labels' distribution before optimization; training moved the policy's outputs toward where the reward model errs. Ask what an exploitability curve scored by expert review would have shown.

Create. Design an independent holdout channel that would detect a policy exploiting a preference reward model.

Hint

Use a different mechanism from the reward model, such as blinded expert review. Never train on its results, score it at every checkpoint, and fix the stopping tolerance in advance.

Further reading#

• Goodhart pressure and self-confirming loops — the wider failure family.
• Adaptive validity — measuring the exploitability curve and $b^{*}$ with the actual optimizer.
• Case E: reasoning-model training — the training case, end to end.
• Signals and resolution — why PRM scores, value estimates, and robustness margins are different objects.
• Gao et al. (2023), "Scaling laws for reward model overoptimization," ICML — the canonical measurement of a learned proxy's Goodhart curve.
• Rafailov et al. (2023), "Direct preference optimization: your language model is secretly a reward model" — the preference loss that removes the separate reward model.