Signals and resolution

A signal is what leaves an evaluator: a verdict, score, probability, preference, criterion vector, critique, or error trace. Its resolution is the unit judged: a step, an action, an artifact, a trajectory, or a rate across a population. Neither is fixed by the mechanism, and together they decide what a consumer can do with the judgment.

Signal forms#

The common forms are boolean pass/fail, scalar score, probability, ordinal category, pairwise preference, a vector of criterion scores, critique, confidence, and a structured error trace. Signal form decides what downstream consumers can do and what scalarization loses.

DocsAgent, the documentation agent of Case G, shows what scalarization loses. It is graded on four criteria (factual correctness, citation grounding, procedural faithfulness, and abstention when evidence is absent), not on a single vague "helpfulness" score. One score over the four cannot say which failed, or gate grounding while trading off the rest. At the other end of the table, a critic produces diagnostic critique or error information and need not emit a scalar at all.

Verdicts first, margins beside them#

Prefer binary criteria to composite scores. Binary verdicts on criteria that experts can adjudicate independently are easier to calibrate and diagnose than an unanchored 1–10 score, and they are what the noisy-sensor correction needs. Keep them separate until a decision requires a scalar; Aggregation geometry covers that step.

Abstention is a valid value, not a missing one: forcing a verdict turns uncertainty into false confidence, which is why a criterion card declares an abstention region. Give abstentions a route, such as escalation to a person or a stronger channel, and report their rate. For DocsAgent, abstention runs both ways: the assistant is graded on abstaining when evidence is absent, and its grounding judge may itself abstain on an ambiguous span.

Report margins beside verdicts. A margin keeps apart two trajectories that both satisfy "latency < 10 s", one at 9.8 s and one at 2 s, and it supports partial credit, early warning, and search; Specification shows how to find each check's natural margin.

In this handbook a continuous judge score is acceptable only as a calibrated probability or as a margin, and a score that claims to be a probability needs a reliability diagram behind it (Monitoring and control in production). LLM judges in practice reconciles the rule with the Berkeley slides.

Resolution cuts across mechanisms#

Resolution is the unit judged: a token or span, a reasoning step or state, an action or tool call, a final answer or artifact, a trajectory or episode, or a system-level rate across a population, such as a pass rate. It cuts across mechanism families: "process" tells you when and where the judgment occurs, not how epistemically strong it is.

The execution and deterministic rows use SQLAgent, the text-to-SQL agent of Case C.

So a process verifier is any evaluator that checks intermediate properties, whatever its mechanism family; it is not a synonym for PRM, and a deterministic trajectory checker is process-level without being learned. And a process reward model is usually a learned judge at process resolution: two coordinates, not a new rung of the verification hierarchy. The math case states the consequence: a PRM is not above the answer checker because it inspects more steps; it is usually epistemically weaker but diagnostically denser.

Outcome or process#

Outcome evaluation asks whether the terminal artifact or state satisfies the specification. Process evaluation checks intermediate steps, actions, evidence acquisition, or trajectory constraints. Outcome evaluation usually costs less, intrudes less, and tolerates novel strategies better; process evaluation can catch lucky guesses and forbidden paths and gives denser diagnostic feedback, but a rigid reference process can invalidate legitimate alternatives.

Use process-level evaluation when at least one of these holds:

• the path has independent safety or policy significance, as in "never send without confirmation";
• intermediate errors can cancel out and produce a lucky correct answer;
• the goal is to improve a reusable procedure, not only to rank terminal outputs;
• the evaluator needs localized evidence to support repair or learning;
• the claim explicitly includes competence in planning, evidence gathering, or methodological discipline.

Otherwise process evaluation can hurt validity: when several legitimate strategies exist, forcing one canonical chain can lower construct validity, and a reference trace on file is no reason to grade the path (Specification). Do not demand a reference chain of thought only because one exists; evaluate externally observable process, such as evidence used, tool actions, state transitions, invariants, and checkpoints, without demanding private reasoning traces.

The coding agent of Case B needs both. "Tests pass" is an outcome criterion on the final repository state. "Required approvals observed" is a process criterion, checked deterministically over the event trace for $\neg\,\text{merge}\ \mathsf{U}\ \text{approved}$, which constrains only the ordering the claim includes, not the sequence of commands.

In training, process signals can pay: Lightman et al. (2023) showed on MATH that step-level process supervision can outperform outcome-only supervision, and selecting training data by final answer alone lets through right answers reached by wrong reasoning (Chen, Wang & Qu, 2026, §5.5). For evaluation, the lesson is not "process is always better" but to decide whether the process is part of the construct or a diagnostic channel.

Process evaluators have three legitimate jobs: diagnosis, locating where a trajectory first went wrong; selection and search, preferring trajectories with stronger intermediate evidence; and training, supplying denser credit assignment than a terminal reward, which is where their dangers concentrate. The economics differ too. Outcome checks are operating expenditure: nearly free to produce, but what they buy is per instance, so best-of-N sampling and answer-filtered retraining must be paid for again on every new problem. Process-level evaluation is capital expenditure: step labels are expensive, but a validated process evaluator, like a corrected procedure, is reusable across every future problem that shares the structure (Chen, Wang & Qu, 2026, §5.5).

A correct outcome certifies nothing about how it was reached or reported: agents have reached correct answers while defending a wrong mechanism, which is why outcome, mechanism fidelity, and epistemic honesty should be measured as separate quantities (Eulig, 2026, via Chen, Wang & Qu, 2026, §6.3). Evaluating agents develops the three measurements.

Three objects that look alike#

Three objects can each be written as a number attached to a step, and they should not be conflated.

Here $s_t$ and $a_t$ are the state and action at step $t$, and ctx is the surrounding context. The trap is the standard cheap labeling method: automated labeling by rollout scores a step by the fraction of completions from it that end in success,

$$\hat V^{\pi}(s_t) = \frac{1}{K} \sum_{j=1}^{K} y_j$$

where:

• $s_t$ is the partial solution, or state, after step $t$;
• $K$ is the number of completions sampled from the current policy $\pi$, each starting at $s_t$;
• $y_j$ is 1 if completion $j$ reaches a correct final outcome and 0 otherwise.

In words: the label estimates the probability that this policy, continuing from $s_t$, succeeds, which under a binary outcome reward is the value $V^\pi(s_t)$, not a judgment of the step.

Illustratively, suppose step 3 of a proof contains an error that the current policy usually repairs two steps later, so six of eight completions succeed and the step is labeled 0.75: a fact about the policy's recovery, not about step 3. Retrain the policy and the unchanged step gets a new label, because a policy-indexed value changes by construction when the policy changes. Such labels are cheap but quietly circular, since they inherit the current policy's blind spots. Hence the rule in Case E: treat rollout-labeled PRM scores as policy-conditioned value estimates when that is how the labels were generated, which means relabeling as the policy moves.

A robustness margin is the third object: computed from a formal or quantitative specification and a trajectory, positive when the property holds and negative when it is violated, distinct from learned value estimates, and tied to no policy.

In practice#

Write the signal semantics for each criterion:

1. Name the consumer and the decision the signal feeds: a gate needs a verdict, a search needs a slope.
2. Fix the value set, normally pass, fail, and abstain, and give abstentions a route.
3. Attach the natural margin wherever the specification has one.
4. Admit a continuous score only as a calibrated probability or a margin; otherwise threshold it into a verdict.
5. Choose the resolution from the construct: outcome by default, plus checks on observable process when one of the five conditions holds.
6. Choose the mechanism separately, since process resolution can be formal, executable, learned, or human.
7. Name every step-level score as a PRM score, a value estimate, or a margin, record how its labels were made, and relabel value-type scores when the policy changes.
8. Keep outcome, mechanism fidelity, and honesty as separate criteria when the claim covers them, and keep the vector intact until aggregation.

The artifact is a signal-semantics entry per criterion, which fills the corresponding field of the evaluator dossier.

Common failure modes#

• Rigid trajectory grading — a correct, novel solution fails because it departs from the reference trace → grade the observable constraints the claim includes, not one canonical path. See rigid trajectory grading.
• Outcome-only grading of a path that matters — a right end state reached through a forbidden action passes → add process checks, written as temporal patterns over the trace.
• Rollout labels read as step quality — policy-conditioned values are reported as step correctness and reused after retraining → call them values and relabel on policy change. See self-confirming loop.
• Premature scalarization — criteria collapse into one score before anyone sees the trade-offs → keep the vector and aggregate with an explicit geometry. See one-number reporting.

Check yourself#

Analyze. Classify the coding agent's signals (Case B) by form, resolution, and mechanism family: test outcomes on the final repository state, the approval-before-merge check, the maintainability judge's verdict, and the weekly resolve rate.

Hint

Resolution and mechanism are separate coordinates. The approval check is process-resolution and deterministic; the judge's verdict is outcome-resolution and learned; the resolve rate is a population-level rate estimated from task verdicts.

Evaluate. A team labels each step of its math solutions with the fraction of eight completions from that step that reach the right answer, reports the labels as "step correctness," and reuses them after retraining. Evaluate the report and the reuse.

Hint

The labels estimate $V^\pi(s_t)$: they measure the policy's chance of finishing, change when the policy changes, and inherit its blind spots.

Create. Design a process evaluator for a research agent that inspects observable evidence-gathering actions without requiring private chain of thought.

Hint

List the observable events (searches issued, sources opened, claims linked to cited spans) and write constraints over them as temporal patterns, such as no claim before a supporting source has been retrieved. Give each a margin and a mechanism.

Further reading#

• Specification: what counts as success — temporal patterns and robustness margins.
• Aggregation geometry — how a vector of criterion signals becomes a decision.
• Evaluating agents — the trajectory object and its assessment targets.
• Evaluators inside training loops — PRMs and rollout labels as training signals.
• Lightman et al. (2023), "Let's verify step by step" — the process-supervision result behind the PRM literature.
• Chen, Wang & Qu (2026), "Recursive Self-Improvement in AI" — §5.5 on result-level versus process-level improvement and its costs.