# The evaluator system

> Meet the master object E_sys = (C, I, R, V, L), and why a proof checker is a mechanism but a release gate built on it is a system.

An **evaluator system** is the complete arrangement by which a claim about an AI system is operationalized, measured, consumed, and maintained. It has five coordinates — claim, instrument, regime, validation profile, and lifecycle — and the mechanism that computes a score is only one part of one of them.

> [!KEY] In one sentence
> A proof checker is a mechanism; a release gate built on it is an evaluator system, $\mathcal{E}_{\text{sys}} = (\mathcal{C}, \mathcal{I}, \mathcal{R}, \mathcal{V}, \mathcal{L})$, and the whole system is what you design, validate, and maintain.

## Three questions, and the one that gets lost

Three questions organize the field:

1. **Creation:** how does human intent become an operational signal?
2. **Validation:** why should we believe the signal means what we claim?
3. **Use:** what causal role does the signal play, and what happens when a system adapts to it?

The third is the one most likely to be lost when evaluation is treated as static measurement. It gets lost because the usual unit of discussion is the mechanism — "we grade with unit tests", "we use an LLM judge" — and a mechanism has no use of its own. Its use is decided elsewhere: by whoever reads the score, gates on it, or optimizes against it.

Take SQLAgent, a text-to-SQL agent graded by executing its queries against a pinned enterprise database snapshot and checking the results. The executor is the same code whether its verdict feeds a quarterly report, a release gate, or a nightly loop in which developers tune prompts until the number rises. The meaning of the verdict is not the same in the three cases. To keep use in view, the unit of analysis has to be the whole arrangement around the executor.

## The master object

The framework names that arrangement and gives it five coordinates:

$$
\mathcal{E}_{\text{sys}} = (\mathcal{C}, \mathcal{I}, \mathcal{R}, \mathcal{V}, \mathcal{L})
$$

where:

- $\mathcal{C} = (C, P, E, H, D, \delta)$ is the **claim**: the [construct](https://benchmaker.ai/glossary/construct) to be inferred, over which population, environment, and harness, within which validity domain, for which decision;
- $\mathcal{I} = \langle \Psi, A, V, M, Z, R, \Gamma \rangle$ is the **instrument**: the specification, the anchor it is rooted in, the evidence view, the mechanism, the signal it emits, the resolution it judges at, and the aggregation geometry;
- $\mathcal{R} = (\kappa, O, b, S, A_{\text{access}}, K)$ is the **regime**: who consumes the signal, who optimizes against it, with how much exposure, what they can modify, what they can see, and the cost and latency envelope;
- $\mathcal{V}$ is the **validation profile**, $\Pi = (N, E, T, G, C, D)$: measured natural error, exploitability curve, transport, generator coupling, channel coupling, and drift profile;
- $\mathcal{L}$ is the **lifecycle and governance state**: versioning, independent channels, anchor-burn accounting, drift and saturation monitoring, and frame review.

In plain words: an evaluator system is a question asked for someone, answered by some machinery, inside some causal loop, on the strength of some evidence, for some period of time.

Each layer answers its own question, and each has its own part of this handbook:

| Layer | The question it answers | Where to read |
|---|---|---|
| Claim $\mathcal{C}$ | What property are we trying to infer, over what world, and for what decision? | [Claim](https://benchmaker.ai/handbook/claim) |
| Instrument $\mathcal{I}$ | How is the judgment actually produced? | [Instrument](https://benchmaker.ai/handbook/instrument) |
| Regime $\mathcal{R}$ | What causal role does the signal play, and what happens when a system adapts to it? | [Regime](https://benchmaker.ai/handbook/regime) |
| Validation $\mathcal{V}$ | Why should we believe the signal means what we claim, before and after the system adapts? | [Validation](https://benchmaker.ai/handbook/validation) and [Validity under pressure](https://benchmaker.ai/handbook/pressure) |
| Lifecycle $\mathcal{L}$ | Does the validity survive over time, and are we still evaluating the right thing? | [Lifecycle](https://benchmaker.ai/handbook/lifecycle) |

## Mechanism versus system

A **mechanism** is the procedure that computes a judgment: a proof checker, a test runner, a rule engine, an LLM judge, a human panel. A mechanism is not an evaluator system. A proof checker shows where the line falls: the checker is a mechanism, while a benchmark whose release gate depends on that checker — under a particular theorem statement, environment, version, trusted base, and decision threshold — is an evaluator system.

Each of those additions is a place where the system can fail while the mechanism works perfectly:

- **Theorem statement.** A sound checker certifies whatever was formalized. Formality improves fidelity to the specified property; it does not validate the choice of property.
- **Environment.** A proof that a controller never exceeds a speed limit holds for the formal model of the world. If a real sensor violates the model, the theorem stays true while the physical system fails.
- **Version and trusted base.** A formal guarantee belongs to a [configured trusted base](https://benchmaker.ai/glossary/configured-trusted-base): permitted axioms, theorem statements, checker version, kernel configuration, environment assumptions, and whether the optimizer can modify any of them. The Goodhart boundary moves to these surfaces rather than disappearing.
- **Threshold.** The gate turns a verdict into ship or hold. Where to put the threshold is a product decision, not a property of the checker.

Most evaluators are less formal than a proof checker, so the gap between mechanism and system is wider. For SQLAgent the mechanism is an executor plus a result-set comparator. The system is everything in this table:

| Layer | SQLAgent's rollout evaluator ([Case C](https://benchmaker.ai/handbook/cases/sql-data-agent)) |
|---|---|
| Claim | Answers enterprise analysts' questions correctly and safely over the target schema; finance analysts' logged questions; a pinned snapshot of the finance warehouse at schema v12; roll the agent out or hold it *(illustrative)* |
| Instrument | Execution against the snapshot; semantic result-set equality with a reference anchor; metamorphic tests; read-only credentials and row-level permissions; a rubric-guided judge for ambiguity handling; SME adjudication of semantic edge cases |
| Regime | The head of analytics' rollout gate consumes the verdict; developers iterate against a separate development split *(illustrative)* |
| Validation | Evidence still to gather: rank stability across alternate snapshots, agreement with an SME-reviewed workflow sample, a check that the score does not mostly track memorized schema names, and analyst acceptance in shadow deployment |
| Lifecycle | Snapshot, rubric, judge, and threshold versioned together; certification tasks kept out of development; revalidation when the schema changes *(illustrative)* |

The claim, instrument, and validation rows follow Case C; entries marked illustrative are this chapter's assumptions.

Swap the snapshot, show the certification failures to the developers, or let a new judge version in without recalibration, and you have a different evaluator system, even though the executor has not changed by a byte. That is why "the benchmark is valid" is never a complete sentence. Evidence supports an interpretation for a population, a decision, and a regime within a validity domain; change any of them and the argument may need to be rebuilt.

## Six objects, not one

The same point holds at a finer grain. Consider six statements about a single evaluation:

- "The model answers A, B, C, or D." — a response format.
- "Every factual claim must be supported by a primary source." — a specification criterion.
- "An LLM judge scores the answer against that criterion." — an evaluator mechanism.
- "The judge emits 0.87." — a signal.
- "We keep the best of 64 answers by that score." — a use of the signal.
- "We update the policy to maximize that score." — a training regime.

They are causally connected but ontologically different. Six objects have to be kept separate — construct, specification, evaluator mechanism, signal, resolution, and use — and they form a construction stack in which resolution and deployment context cut across every stage. Merging them produces the category errors this handbook keeps correcting: a [rubric](https://benchmaker.ai/glossary/rubric) is a specification, not a rung of the verification hierarchy, and RLHF is a training pipeline that consumes evaluator signals, not an evaluator.

The evaluator system keeps every one of these objects and gives each a place. The construct moves into the claim, next to the population and the decision. Specification, mechanism, signal, and resolution join the anchor, the evidence view, and the aggregation geometry in the instrument. Use becomes the regime, together with its optimizer and surfaces. Validation and lifecycle are layers of their own: a six-axis description says how an evaluator is built and used, not whether the inference holds or for how long. [Vocabulary without category errors](https://benchmaker.ai/handbook/foundations/vocabulary) puts each common term on the axis it describes.

## The lifecycle loop

The framework draws the evaluator system as a lifecycle with feedback, not a one-way pipeline. Read forward, the claim is operationalized into an instrument, the instrument's signal is put to use in a regime, validation tests the inference in that regime, and lifecycle controls maintain what validation established. Two arrows run backward into the instrument: validation feeds back into redesign, and lifecycle controls preserve validity over time.

The forward arrows show how the layers depend on one another, not the order in which to build them. The One Spine, the framework's ten-step workflow, states the claim and the regime first, before any mechanism is chosen, because the right mechanism depends on the regime: training may need millions of cheap evaluator calls, and certification hundreds of expensive independent audits.

The map below draws that loop. Select each layer to see its coordinates and dominant failures, then switch to **One spine** and highlight Regime: it appears in step 1, before anything is built.

```widget
framework-map
view: layers
```

## A discipline, not a tool

Because the object has five layers, no single field covers it. A mature practice needs eight disciplines at once:

- **Specification engineering** — translating constructs into operational criteria. See [Specification](https://benchmaker.ai/handbook/instrument/specification) and [Rubrics and criterion cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards).
- **Verification engineering** — pushing criteria toward stronger evidence channels. See [Verifiability engineering](https://benchmaker.ai/handbook/practice/verifiability-engineering).
- **Psychometrics and measurement theory** — building validity and reliability arguments. See [Validation](https://benchmaker.ai/handbook/validation).
- **Statistics** — quantifying uncertainty and rare events. See [Estimands and intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals) and [Rare events](https://benchmaker.ai/handbook/validation/rare-events).
- **Optimization and RL analysis** — understanding Goodhart pressure and adaptive validity. See [Validity under pressure](https://benchmaker.ai/handbook/pressure).
- **Systems engineering** — controlling harnesses, environments, versions, trusted bases, and information flow. See [Harness and reproducibility](https://benchmaker.ai/handbook/lifecycle/harness-and-reproducibility) and [Exposure and surfaces](https://benchmaker.ai/handbook/regime/exposure-and-surfaces).
- **Production assurance** — maintaining calibration, coverage, and intervention quality under drift. See [Production monitoring](https://benchmaker.ai/handbook/regime/production-monitoring).
- **Governance and frame review** — asking the logically prior question: are we still evaluating the right thing? See [Frame review](https://benchmaker.ai/handbook/lifecycle/frame-review).

Nor do their questions collapse into one scale. The framework keeps grounding strength, validity, reliability, optimization robustness, and production assurance as separate questions with separate evidence. That is why one evaluator system can be strong on one question and broken on another: a formal verifier can be perfectly sound and still certify the wrong property.

## In practice

Write a coordinate sheet for one evaluator you already rely on.

1. **Name the decision and its consumer.** Write down the decision the score changes and who makes it. If you cannot name one, you have found eval theater.
2. **Write the claim.** Give one line each to the construct, population, environment, harness, and validity domain.
3. **List the instrument per criterion.** Record the specification, anchor, evidence view, mechanism, signal, and resolution, then the aggregation rule that turns criteria into the decision.
4. **State the regime.** Name who optimizes against the signal, with how much exposure, what they can modify, and what they can see.
5. **Inventory the validation evidence.** For each component of the validation profile, write what was measured, with n and an interval, or mark it "planned".
6. **Record the lifecycle state.** Note which versions are pinned together, which channels are kept separate, what triggers revalidation, and who may revise the construct.
7. **Walk the arrows.** At each arrow of the map, write one sentence on why the inference is justified and one on what changes when the system adapts to the instrument. Every blank is a work item.

The artifact is a one-page coordinate sheet. It is a compressed form of the 16-field evaluator dossier (see [Templates](https://benchmaker.ai/handbook/reference/templates)), and its claim, regime, and validation rows become the core of a [validity statement](https://benchmaker.ai/glossary/validity-statement).

## Common failure modes

- **Mechanism mistaken for system** — "we grade with a proof checker, so the eval is sound" → name the theorem statement, environment, version, trusted base, and threshold, and validate the system that contains the checker. See [the verification hierarchy](https://benchmaker.ai/handbook/foundations/verification-hierarchy).
- **Eval theater** — scores appear in dashboards and reviews, but no decision cites them → write the decision first, then ask when the eval last changed one. See [eval theater](https://benchmaker.ai/handbook/practice/anti-patterns#eval-theater).
- **Unnoticed regime migration** — the instrument is unchanged, but developers now hill-climb it or training now reacts to its alerts → classify the signal by the behavior it changes, not the dashboard it appears on, and move its validation obligations with it. See [Goodharting](https://benchmaker.ai/handbook/practice/anti-patterns#goodharting).
- **Judge rot** — the judge's prompt, model, or settings changed without a new version or recalibration → version the whole measurement together and revalidate on change. See [judge rot](https://benchmaker.ai/handbook/practice/anti-patterns#judge-rot).

## Check yourself

**Analyze.** Take an evaluator from a recent paper and classify it along every coordinate: claim, specification, anchor, evidence view, mechanism, signal, resolution, aggregation, and regime. Which parts of what the paper calls "the evaluation" belong to different layers?

> [!DETAILS] Hint
> Mark each coordinate as stated, inferable, or missing, and look first for the decision, the regime, and the validity domain.

**Evaluate.** For a formal verifier you know, identify the specification gap and the trusted-base assumptions. Then describe a failure that preserves the verifier's soundness while the evaluator system no longer supports its claim.

> [!DETAILS] Hint
> Start with the theorem statement: does it formalize the property the decision depends on? Then check the environment assumptions, and whether anything under optimization can modify axioms, statements, or checker configuration.

**Create.** Place "a rubric-conditioned LLM judge used to keep the best of 64 answers" into every coordinate of the evaluator system. Then rewrite only the coordinates that change when the same judge becomes an RL reward, and list the evidence the new system needs.

> [!DETAILS] Hint
> The instrument does not change; the regime's optimizer, exposure budget, and modification surface do. The validation profile then needs an exploitability curve measured with the actual optimizer.

## Further reading

- [Vocabulary without category errors](https://benchmaker.ai/handbook/foundations/vocabulary) — reward model, rubric, PRM, RLHF, and verifier, each placed on the axis it describes.
- [Use regimes: same judge, four jobs](https://benchmaker.ai/handbook/regime/use-regimes) — the regime coordinate in depth, and how signals migrate between regimes.
- [The validity argument](https://benchmaker.ai/handbook/validation/validity-argument) — the six claims a validation profile has to support.
- [The One Spine](https://benchmaker.ai/handbook/practice/one-spine) — the ten-step workflow that builds an evaluator system in a safe order.

---
Source: https://benchmaker.ai/handbook/foundations/evaluator-systems