E: Reasoning-model training

Case E evaluates reasoning-model training: reinforcement learning that improves a model's multi-step reasoning on math, code, and logic problems against verifiable rewards and a learned process reward model. Its evaluator has two jobs that need different instruments, dense feedback for development and independent evidence for certification.

Claim and decision#

The case's lesson is that the training reward and the evaluation instrument should not be identical forever. Development feedback needs density; certification needs independence.

The case uses an illustrative deployment, the one the Studio preset loads: an in-house model trained with RLVR and a rollout-labeled PRM.

• Construct: multi-step reasoning quality on math, code, and logic problems.
• Population: held-out problems with verifiable answers in the three domains, refreshed each training phase.
• Environment and harness: single-turn generation, a code sandbox for code problems, and a fixed sampling configuration per checkpoint.
• Validity domain: problems with checkable answers in the three domains; open-ended reasoning without checkable answers is excluded.
• Decision: promote a checkpoint to the next training phase or to release, or roll it back.

Density for training, independence for certification#

Training may need millions of cheap evaluator calls and certification hundreds of expensive independent audits, so the rule is to choose the strongest affordable mechanism per criterion per regime, then deliberately use different mechanisms across development and certification when independence is valuable.

Two migrations are foreseeable: certification failures added to the training data spend the certification channel's independence, and promoting checkpoints by PRM score would make the PRM the de facto certification signal while the policy is optimized against it. Keeping certification problems and results from the trainer, whose modification surface is the policy alone, is how the design meets the anchor condition, $A \notin S$ with $I(A \to O)$ controlled.

Criteria and mechanisms#

This design uses three criteria. Certification runs on held-out problems the trainer never sees, and never relies on a training signal alone: where training uses the answer checker or the PRM, certification adds the proof checker and expert audit.

An answer checker is grounded but incomplete: intermediate errors can cancel out into a lucky correct answer, which is why step validity is in the specification. Exploits are a fatal gate, though zero found in an audit of n solutions bounds their rate only near 3/n. Step validity is advisory, because outside the formal subset nothing certifies it at scale, and the verified-answer rate is the tradeable score, reported per domain with its interval.

Rollout labels are value estimates#

Step labels are expensive, so the usual workaround labels steps by rollout: the fraction of completions from that step that end in success. Sampled from the current policy $\pi$, that fraction estimates a policy-conditioned value $V^{\pi}(s_t)$, not the step's validity, and such a value changes, by construction, when the policy changes.

Three consequences follow. Labels from last phase's policy describe that policy, so relabel or re-audit each phase. Labels inherit the policy's blind spots: a flawed step from which the policy usually reaches the right answer anyway scores high, which makes the workaround cheap and quietly circular. And a PRM built from the policy's own rollouts is coupled to the generator, where optimization can reinforce the blind spot that produced a mistake, a self-confirming loop (Chen, Wang & Qu, 2026, §5.3).

Re-estimate evaluator error at every checkpoint#

Each checkpoint is a new system trained against the reward, and optimization moves outputs toward the regions the evaluator selects, so error rates measured on the starting policy cannot be assumed to hold later: differential misclassification. Re-estimate judge error on-policy at every checkpoint, meaning the PRM's agreement with expert step labels, and the answer checker's false-positive rate too.

With a share $p$ of rollouts correct and a checker with true- and false-positive rates $t$ and $f$, the benign selection model (independent rollouts, stable $t$ and $f$) puts the share of rewarded rollouts that are correct at $pt/(pt + (1-p)f)$; Adaptive validity develops it, with the scalar-score case for a PRM that picks the top rollout.

The calculator starts at an illustrative hard problem: eight rollouts, each correct with probability 0.3, and a checker with $t = 0.95$ and $f = 0.02$, so 95.3% of rewarded rollouts are correct. Set $f$ to 0.20, an illustrative rate for a later checkpoint that has found a formatting exploit, and precision falls to 67.1%, at any number of rollouts.

Measuring b*#

Measure exploitability with the real optimizer: run the RL trainer against the reward, score each checkpoint's outputs on the anchor channel, and record exposure as KL divergence from the starting policy. $b^{*}$ is the largest tested exposure before divergence exceeds a predeclared tolerance. A learned reward's exploitability curve has a canonical shape: the proxy can keep rising after a stronger reference measure begins to fall (Gao et al., 2023).

The calculator's illustrative curves use best-of-$b$ selection as exposure. It opens on a well-grounded reward standing in for the answer checker, at a tolerance of 0.05: the gap reaches 0.07 by $b$ = 1,024, so $b^{*}$ = 174. Lower the largest budget tested to 128 and $b^{*}$ reads "≥ 128". Switch to Overoptimization, the shape of a learned proxy such as the PRM: $b^{*}$ falls to 3, while the anchor channel's score keeps improving until $b \approx 17$.

Report $b^{*}$ with its tested range, never "robust to Goodhart"; a phase that trains past the last tested checkpoint is outside the validity statement until re-measured.

Validation plan#

Each row is a planned, not yet measured, entry in the validation profile Π.

Channels and tripwires#

Keep three measurement channels apart: the optimized channel, the answer reward and PRM, dense and inside the loop; the anchor channel, held-out problems refreshed each phase and scored by the certification mechanisms above; and production, once a checkpoint ships. The answer checker sits in the first two, so the probes, proof checker, and audit carry the independence.

An anchor burn ledger records every flow of certification information into training: failures used as examples, problems reused across phases, and each scored checkpoint, since every promotion uses certification results to steer training. Tripwires watch for the training reward rising while certification stays flat (proxy corruption), PRM scores rising while audited step validity does not, and the certification suite nearing its ceiling.

Validity statement#

Following the validity-statement template, and stated as a plan until the profile is measured:

The reasoning-model certification evaluator — domain checkers on held-out problems refreshed each phase, a proof checker for the formal subset, exploit probes, and an expert audit of sampled solutions — is designed to support the claim that certified gains reflect real multi-step reasoning improvement rather than reward exploitation, for the training lead's decision to promote or roll back each checkpoint, over math, code, and logic problems with checkable answers under a fixed sampling configuration, under closed-loop RL against the answer reward and a rollout-labeled PRM with no access to certification problems or results, up to a tested exposure $b^{*}$ that has not yet been measured, with a validation profile that is planned rather than measured, relative to an expert panel that audits sampled solutions each phase, whose agreement is still to be measured, and, on the formal subset, the configured proof checker. Step-level rigor beyond the audited sample remains expert judgment.

Common failure modes#

• The training reward as certificate — the checker or PRM that trains the policy also decides promotion → certify with mechanisms the trainer does not optimize. See common-mode assurance.
• Rollout labels read as rigor — rising PRM scores are reported as better reasoning → report them as values under the current policy; certify steps by audit. See self-confirming loop.
• Certification failures in the training mix — failed certification problems become training examples → keep them out, log every flow, and refresh the set. See anchor leakage.

Check yourself#

Analyze. For each criterion, name its training and certification mechanisms, where both exist. Which could fail together, and why?

Hint

The answer checker sits in both channels, so an exploit it misses in training can pass certification too; only the probes, proof checker, and audit are independent of it.

Evaluate. To save expert time, the team proposes promoting checkpoints by their mean PRM score on the certification problems. Argue for or against.

Hint

The PRM is the signal being optimized, and its labels are values under the current policy; development and certification should use different mechanisms.

Create. Design the anchor-burn ledger for one training phase: the events it records, who may query the certification set and how often, and what triggers a refresh.

Hint

List the spends that anchor burn names, then decide whether each promotion decision counts as a probe.

Further reading#

• Evaluators inside training loops — RLHF, RLVR, and DPO as pipelines, and reward-model overoptimization.
• Adaptive validity — selection precision, the scalar-score tail, and exploitability curves.
• Channels, independence, and anchor burn — the optimized, anchor, and production channels.
• Gao et al. (2023), "Scaling laws for reward model overoptimization" — the canonical overoptimization curve.

Open this case in the Studio to start an evaluator dossier with its claim and regimes pre-filled.