Anchors and evidence views

A reference anchor $A$ is what a judgment is rooted in, such as a canonical answer, a gold state, adjudicated labels, or production outcomes. An evidence view $V$ is what the evaluator may observe while judging. Taxonomies often omit the evidence view and misfile the anchor as a mechanism, yet changing either changes the instrument.

Reference anchors: what the comparison is rooted in#

The framework lists seven kinds of reference anchor:

The quality of a reference anchor and the fidelity of the comparison against it are separate properties: a perfect exact-match scorer against a bad answer key is perfectly faithful to a bad anchor. An ambiguous gold label can invalidate even a closed-ended item, and no better scorer repairs that; the audit has to reach the key itself. Much of the SWE-bench Verified repair was such an audit: professional developers flagged tasks whose unit tests were unfair, and those tasks were filtered out (Song, 2025, slide 78).

Gold is therefore a reference anchor, not a mechanism and not a rung of the verification hierarchy: whatever compares against it, whether exact match, semantic equivalence, execution, or expert adjudication, is classified by its own family. The verification hierarchy, demoted explains why gold answers get no rung of their own.

Adjudicated labels play a second part. The calibration set that measures a judge's error is itself a reference anchor, which the framework calls a validation anchor sample, and its labels trace back to a trust root with its own measurable disagreement.

One word, four objects#

"Anchor" can name four different objects, so this handbook always qualifies it:

Evidence views: what the evaluator may see#

Show one LLM judge the final answer alone, then show it the cited passages, tool traces, hidden environment state, database snapshots, provenance metadata, execution logs, policy constraints, or abstention metadata as well, and you have built different instruments. In the judge signature $z = g_\theta(x, y, \Psi, e)$, the evidence $e$ is an input on a par with the rubric $\Psi$, and changing it can change the verdict (LLM judges in practice).

Consider one judge grading the "no unauthorized writes" criterion of SQLAgent, the text-to-SQL agent of Case C, under three views (illustrative):

The judge's weights are identical in all three rows; the instrument is not. In the last two rows the criterion no longer needs a judge, because a trace scan and a snapshot diff are deterministic checks.

An evidence contract fixes the view for each criterion: it states what observable evidence is admissible. Grounding is judged against citations or retrieved passages; a forbidden action is checked in the tool trace or the state transition, never in the final prose. The grounding card of DocsAgent, the documentation agent of Case G, admits the answer's atomic claims plus the cited document spans.

For agents that act on a world, grade backend state, not what the interface claims: the order exists in the backend, whereas a rendered confirmation page is appearance (Anthropic, 2026). Check what should not have changed as well. The outcome-validity checks for environment state cover both relevant and irrelevant state, and guard against trivial edits that satisfy the checker without doing the task (Zhu et al., 2025; Song, 2025, slide 44).

Better evidence moves criteria up#

Evidence architecture can move criteria upward in evidential strength without changing the underlying model. Research agents can attach claim–evidence pairs, coding agents tests and proof obligations, data agents invariants over state before and after, and policy agents the evidence behind each decision. Verifiability engineering turns this into a method.

Evidence also flows the other way. The framework's anchor condition, $A \notin S$ with information flow $I(A \to O)$ controlled, requires that the reference anchor stay outside the modification surface $S$ and that what flows from it back to the optimizer $O$, whether scores, failure explanations, or revealed tests, be controlled. A coding agent that can write to the repository holding its grading tests fails the first half (Case B); exposure and surfaces treats both.

In practice#

1. Name the reference anchor for each criterion, with its type, its producer, and its version.
2. Audit the reference anchor apart from the comparator. Have experts review a sample of its items for ambiguous keys, unfair tests, and stale references.
3. Write the evidence contract. List what the evaluator may inspect: traces or state for forbidden actions, cited spans for grounding.
4. Grade state, including what should not change. Check the backend, not the interface, and diff irrelevant state too.
5. Check the anchor condition. Confirm each reference anchor lies outside the modification surface, and record what flows from it to the optimizer.

The artifact is the evidence-view-and-anchors field of the evaluator dossier: what each evaluator may inspect, what reference anchor roots each judgment, and how each stays independent.

Common failure modes#

• Faithful to a bad key — the scorer is exact, but the answer key is ambiguous or wrong → audit reference anchors separately from the comparator.
• Appearance verification — the grader trusts a confirmation page or the agent's own report → grade backend state. See grader gaming.
• A reference anchor inside the surface — the system can edit, or repeatedly probe, the reference that grades it → move it out of reach and log every query. See anchor leakage.

Check yourself#

Analyze. For a benchmark you know, name its reference anchor and its comparator. Which of its known problems belong to each?

Hint

Ambiguous labels, unfair tests, and stale references are reference-anchor problems; brittle string matching and lenient judging are comparator problems.

Evaluate. A team shows its answer-only grounding judge the retrieved passages too, and the pass rate drops. Did the system get worse?

Hint

No system changed; the instrument did. Compare both views against adjudicated labels, and version the change as a new measurement.

Create. Write an evidence contract for a customer-service criterion: refunds are issued only after identity verification.

Hint

It is a precedence over actions, so the contract needs the tool trace and the backend's refund records, never the agent's summary.

Further reading#

• Verifiability engineering — redesigning artifacts so quality leaves stronger traces.
• Channels, independence, and anchor burn — keeping the anchor channel independent.
• Task validity and outcome validity — outcome-validity checks by output type, including environment state.