# Mechanism families

> Proof checkers, executors, rule engines, learned judges, intrinsic signals and people: what each can guarantee, and what each can only estimate.

A **mechanism** $M$ is what computes a judgment: a proof checker, an executor, a rule engine, a learned judge, a person, or a portfolio of them. Mechanism families differ in what they can guarantee and what they can only estimate, and every automated one stands in for a human judgment.

> [!KEY] In one sentence
> Know what each mechanism family can guarantee and what it can only estimate, treat every automated evaluator as a substitution claim to be tested, and route each criterion to the strongest affordable mechanism.

## Every evaluator is a substitution claim

Every automated evaluator makes the same commitment. Whenever it replaces a human decision, the system is making an implicit claim: this observable signal is an adequate substitute for the relevant human judgment for this decision, on this validity domain. The job of evaluator engineering is to make that claim explicit, test it, and bound where it fails. This handbook calls it the **substitution claim**. The RSI survey says the same of self-improvement: every loop rests on a signal standing in for human judgment, and no loop rises above the quality of that substitute ([Chen, Wang & Qu, 2026](https://benchmaker.ai/handbook/reference/sources#chen-wang-qu-2026), §5).

The executor of SQLAgent, the text-to-SQL agent of [Case C](https://benchmaker.ai/handbook/cases/sql-data-agent), stands in for an analyst checking whether the rows answer the question; the entailment judge of DocsAgent, the documentation agent of [Case G](https://benchmaker.ai/handbook/cases/docs-agent), stands in for an expert reading the cited page. Each substitution holds, if at all, for a decision and a domain, so it completes naturally into a [validity statement](https://benchmaker.ai/glossary/validity-statement).

Families differ in how the claim can be supported: by an argument from construction or semantics, or only by measurement:

| Family | Examples | What it can guarantee | What it can only estimate |
|---|---|---|---|
| Formal verifier | Proof checker, model checker, type system | The formal property, relative to its configured trusted base | Whether the formalized property is the one that matters |
| Execution or effect checker | Test runner, compiler, simulator, state check | What happened on the inputs it ran | Behavior on inputs it never ran |
| Deterministic checker | Exact match, parser, rule engine | Agreement with its reference anchor under its normalization | Whether the reference anchor and normalization are right |
| Structural control | Read-only credentials, sandboxes, permissions | That a configured bad state is unreachable | Anything outside the configured roles and paths |
| Learned judge | LLM judge, reward model, classifier, PRM | Nothing by construction | Semantic criteria, with error that must be measured |
| Human rater or panel | Expert adjudication, preference panels | Nothing by construction; a panel can serve as trust root | Value-laden criteria, with measurable disagreement |
| Intrinsic signal | Confidence, likelihood, self-consistency | Nothing independent of the generator | Rough confidence, coupled to the generator's errors |
| Hybrid portfolio | Per-criterion routing across families | What each channel guarantees, criterion by criterion | The residue left to its learned and human channels |

The [verification hierarchy](https://benchmaker.ai/handbook/foundations/verification-hierarchy) orders the automated families by expected grounding strength, formal over execution over learned judges over intrinsic signals, with human judgment outside the ladder. The mechanism is where that ordering primarily applies, as a prior that measured error replaces, never as a verdict. In this handbook's map, deterministic checkers take the execution rung's prior, and structural controls sit off the ladder: they remove a failure by construction instead of measuring it.

## Formal, execution, and deterministic checkers

A **formal verifier** does not estimate quality; it proves that a relation holds in a formal system. Its guarantee is [soundness](https://benchmaker.ai/glossary/soundness), always scoped to the property as formalized and to a [configured trusted base](https://benchmaker.ai/glossary/configured-trusted-base); [The verification hierarchy, demoted](https://benchmaker.ai/handbook/foundations/verification-hierarchy#four-rungs) explains both. For a coding agent, syntax, types, memory safety for restricted code, and some authorization invariants in the scaffold can be verified, but whether the patch fixes the user's actual business problem is usually an empirical question.

An **execution checker** observes consequences: it runs code, executes SQL, compiles, simulates, or inspects state. Execution is external to the model's self-assessment, which breaks many self-confirming loops, and cheap enough for repeated use in search and training. It is also a sample: if tests cover only $T \subset X$, passing them establishes behavior on $T$, not on the input domain $X$, and an optimizer can exploit the untested region or incidental test structure. SQLAgent's executor establishes what the query returned on this snapshot, not on snapshots it never saw.

A **deterministic checker** is an exact rule, parser, or comparator without learned semantic judgment, and it can still be invalid or incomplete. It is only as good as its reference [anchor](https://benchmaker.ai/glossary/anchor) and its normalization: comparing SQL strings exactly is a poor test of correctness, since many different queries return the same result. [Constrained decoding](https://benchmaker.ai/glossary/constrained-decoding) offers a related guarantee: a grammar can force every output to parse and every tool call to match its schema, which is a real guarantee about syntax and none at all about meaning ([structural guarantees](https://benchmaker.ai/handbook/practice/structural-guarantees)).

> [!CONVENTION] Two borderline classifications
> - Parsers, schema constraints, and permission invariants count as formal only relative to a syntactic or structural property, never a semantic one.
> - Embedding similarity and lexical overlap compare an output with a reference, so this handbook treats them as weak comparators against a reference anchor, not as intrinsic signals. Intrinsic signals are the generator's own.

## The learned-judge family

A **learned judge** approximates a judgment empirically: the way it maps evidence to a verdict comes from training data or model parameters, not from a formal decision procedure. Its members deserve separate treatment without occupying separate rungs:

- An [LLM-as-a-judge](https://benchmaker.ai/glossary/llm-as-a-judge) is a general LLM prompted to judge, often with a rubric, references, or pairwise comparison, which may never have been trained as a grader ([LLM judges in practice](https://benchmaker.ai/handbook/instrument/llm-judges)).
- A [reward model](https://benchmaker.ai/glossary/reward-model) is trained to map candidates to scalar or pairwise preference estimates, often so that something can be optimized against it ([training loops](https://benchmaker.ai/handbook/regime/training-loops)). By resolution, an [ORM](https://benchmaker.ai/glossary/orm) scores terminal outputs and a [PRM](https://benchmaker.ai/glossary/prm) intermediate steps.
- A classifier or guardrail is a narrow learned detector for one property, such as toxicity.

Two neighbors are often mistaken for members. A **critic** produces diagnostic critique and need not emit a scalar or be learned from preferences; a [value model](https://benchmaker.ai/glossary/value-model) estimates expected future return from a state, conditional on a policy, rather than the quality of what is in front of it. A rubric is not a member at all: it is a specification laid over whichever mechanism applies it, from a person to a deterministic checker.

What the members share is an empirical validity domain. Training data, prompt, generator distribution, language, domain, and optimization pressure all shape their errors, so every member needs calibration and external validation. What they do not share is how they were built, which matters for use even though the hierarchy groups them together. A PRM, for instance, is typically a learned judge operating on steps, not a rung above outcome checkers ([signals and resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution)), and [the judge as a noisy sensor](https://benchmaker.ai/handbook/validation/judge-as-noisy-sensor) turns a binary judge's measured error rates into corrected estimates.

## Intrinsic signals

An [intrinsic signal](https://benchmaker.ai/glossary/intrinsic-signal), such as likelihood, confidence, self-consistency, or ungrounded self-critique, comes from the generator itself. It is cheap but not an independent observation of truth, and in a self-improvement loop the generator and the evaluator can share the same blind spots. The evidence that such signals cannot carry improvement on their own, from failed intrinsic self-correction to the Mirror Loop, is in [The verification hierarchy, demoted](https://benchmaker.ai/handbook/foundations/verification-hierarchy#a-regularity-not-a-law); the field's response has been to tie critique to something outside the model, such as execution or retrieval (Chen, Wang & Qu, 2026, §3.1).

Execution shows why grounding helps. A failing test hands the model a concrete counterexample it can run, and a placebo-controlled study of code self-repair credits the value of feedback to that falsification, not to seeing the problem again (Chen, Wang & Qu, 2026, §3.2). A self-consistency vote over a model's own math answers has no reference anchor at all; a final-answer checker has one. When the evaluator shares the generator's errors, optimization amplifies them ([Goodhart pressure and self-confirming loops](https://benchmaker.ai/handbook/pressure/goodhart-and-self-confirming-loops)).

## People: a mechanism and a trust root

People play two roles that must not be conflated: a measurement mechanism, with the error, inconsistency, cost, and bias of any instrument, and, as an adjudicating panel, the [trust root](https://benchmaker.ai/glossary/trust-root) at which the chain of evaluating evaluators stops for value-laden constructs. [The verification hierarchy, demoted](https://benchmaker.ai/handbook/foundations/verification-hierarchy#human-judgment-sits-beside-the-ladder) explains why that places human judgment beside the ladder rather than on it; this section treats people as a mechanism.

As a mechanism, a **human rater or panel** is slow and expensive, disagrees across annotators and within one annotator over time, and is hard to reproduce ([Song, 2025](https://benchmaker.ai/handbook/reference/sources#song-2025), slide 20). Characterize raters' reliability, expertise, and disagreement rather than assuming them infallible. DocsAgent's judges are validated against double-labeled SME examples: the SMEs supply the reference labels, and double-labeling lets their own agreement be measured.

Where no cheaper oracle exists, people remain the frontier: tasks without an oracle or clear criteria often need human evaluation or an LLM judge (Song, 2025, slides 17–18). But "non-verifiable" usually means that ground truth is expensive, the criterion is plural, or no sound oracle exists at the required granularity, not that the property is unverifiable in principle; research direction-setting is the paradigmatic case.

> [!SOURCE] Raters, not "the gold standard"
> The Berkeley lecture says human evaluations "are regarded as gold standards" (Song, 2025, slide 20). This handbook instead measures the human mechanism's error, and names a panel as trust root only with measured reliability and a governance mandate. People who score are raters, panels, or adjudicators, never judges.

> [!OPEN] Running human annotation
> This handbook says what to measure about raters (reliability, expertise, disagreement) but not how to run the operation that produces their labels: recruiting and training raters, sizing panels, budgeting adjudication, and monitoring raters over time.

## Meta-evaluation and hybrid portfolios

**Meta-evaluation**, the evaluation of an evaluator, adds assurance only when it contributes information not already coupled to the primary evaluator's failure mode; a meta-evaluator that shares the first evaluator's blind spots merely deepens the loop. By the RSI survey's account, meta-evaluation has not yet reached the reliability of execution feedback (Chen, Wang & Qu, 2026, §5.4), and the claim that "the evaluator is smarter" is not a measurement argument. Checking DocsAgent's entailment judge with a second prompt to the same model family adds little; checking it against SME-adjudicated labels adds a lot.

A **hybrid portfolio** routes each criterion to its strongest appropriate channel instead of seeking one universal evaluator. A production agent often needs four channels at once: formal and structural checks for permissions, execution for backend state, learned judgment for semantic quality, and human oversight for novel high-impact decisions. Case C compresses the design into one principle: formalize access policy, execute semantic consequences, and reserve learned judgment for the residual language and intent problem.

Cost and regime amend the routing rule: choose the [strongest affordable mechanism](https://benchmaker.ai/glossary/strongest-affordable-mechanism) per criterion per regime, and deliberately use different mechanisms for development and certification when independence matters. [Verifiability engineering](https://benchmaker.ai/handbook/practice/verifiability-engineering) develops the rule and the questions to ask of each criterion, in order. A portfolio is also only as strong as its independence: if its channels' blind spots line up, it adds little assurance, so measure [channel coupling](https://benchmaker.ai/glossary/channel-coupling).

## In practice

Choose and record a mechanism for every criterion.

1. **Write the substitution claim.** Name the human judgment the mechanism replaces, the decision, and the domain.
2. **Ask the strongest-oracle questions in order.** Can the criterion be guaranteed by construction or formal verification, checked by execution or state inspection, or decomposed into deterministic subclaims? Only then use a calibrated learned judge, and then people.
3. **Record the guarantee and the estimate.** Write what the mechanism guarantees, under which configuration or trusted base, and what it only estimates.
4. **Plan the error measurement.** For every learned or human mechanism, plan error rates against adjudicated labels, by slice and by compared system.
5. **Check independence.** A second judge, a meta-evaluator, or a certification channel should differ from what it checks in model family, data, prompts, or evidence.
6. **Split development from certification.** Where independence matters, gate releases with a different mechanism from the dense development signal.

The artifact is the mechanisms field of the evaluator dossier: the mechanism per criterion, its hierarchy family, its configuration or trusted base, and the model, prompt, and version if it is learned.

## Common failure modes

- **Syntax reported as semantics** — schema validity or constrained decoding is reported as correctness → state each guarantee's scope next to the result.
- **A judge where an executor could answer** — an LLM reads a query to decide whether it returns the right rows → execute it against the snapshot.
- **Correlated cross-checks** — two judges from one model family are counted as independent → measure channel coupling and diversify mechanisms. See [common-mode assurance](https://benchmaker.ai/handbook/practice/anti-patterns#common-mode-assurance).
- **Self-grading** — the generator's confidence or self-critique drives an improvement loop → ground it in an external check. See [self-confirming loop](https://benchmaker.ai/handbook/practice/anti-patterns#self-confirming-loop).

## Check yourself

**Analyze.** Classify Case C's mechanisms by family, from execution on pinned snapshots and read-only credentials to the rubric-guided judge and SME adjudication. What does each guarantee, and what does it only estimate?

> [!DETAILS] Hint
> Read-only credentials are structural control; the judge and the SMEs guarantee nothing by construction, so their error must be measured.

**Evaluate.** To save expert time, a team proposes to meta-evaluate its DocsAgent grounding judge with a second judge from the same model family. Evaluate the proposal.

> [!DETAILS] Hint
> Ask what information the second judge adds that is not coupled to the first judge's failure modes, and what an independent channel would look like.

**Create.** Design a hybrid portfolio for an agent that sends external messages on a user's behalf (Case F): one structural control, one trajectory check, one learned monitor, and the role of people.

> [!DETAILS] Hint
> Make unauthorized destinations unreachable, check $\neg\,\text{send}\ \mathsf{U}\ \text{confirmed}$ over the trace, and leave the learned monitor only residual policy interpretation.

## Further reading

- [The verification hierarchy, demoted](https://benchmaker.ai/handbook/foundations/verification-hierarchy) — the families ordered by grounding strength, as a prior.
- [LLM judges in practice](https://benchmaker.ai/handbook/instrument/llm-judges) — building, characterizing, and versioning the commonest learned judge.
- [Verifiability engineering](https://benchmaker.ai/handbook/practice/verifiability-engineering) — the strongest-oracle questions.
- [Guarantee before you estimate](https://benchmaker.ai/handbook/practice/structural-guarantees) — structural controls, constrained decoding, and the subtraction principle.
- Chen, Wang & Qu (2026) — §3.1–3.2 on self-critique and execution-feedback repair; §5 on evaluators as substitutes for human judgment.

---
Source: https://benchmaker.ai/handbook/instrument/mechanisms