The Evaluator Engineering Handbook

Evaluator engineering is the discipline of turning a construct into an instrument, validating the inference that instrument supports, and ensuring that the validity survives the causal regime in which its signal is used. This handbook is a practical map of that discipline: the objects, the evidence, and the failure modes.

Why this handbook exists#

Evaluation vocabulary has grown by accretion. Benchmark builders talk about graders, pass rates, and leaderboards. Psychometrics talks about constructs, validity, reliability, and rater error. Reinforcement learning talks about rewards, critics, value models, and optimization. Formal methods talk about specifications, soundness, and trusted computing bases. Agent evaluation adds environments, trajectories, tools, permissions, and side effects. Work on recursive self-improvement adds evaluators that co-evolve with the systems they judge.

These are not rival descriptions. They describe different coordinates of the same engineered system. The trouble starts when the coordinates are collapsed: when "the benchmark is valid" stands in for an argument, when a reward model is called a verifier, or when a judge that was fine for reporting is quietly promoted to a training reward.

AI evaluation now serves at least three causally different purposes:

1. Measure. Infer a property of a system for a report, a comparison, or a release decision.
2. Optimize. Supply the signal that selects candidates, guides search, or trains a policy.
3. Monitor or control. Watch a deployed system and trigger alerts, blocks, rollbacks, or escalation.

The same evaluator mechanism — a test suite, a proof checker, an LLM judge, a rubric-guided panel — can occupy any of these roles. Its meaning changes with the role. A judge can be excellent for retrospective reporting and unsafe as a reward; a test suite that began as a benchmark becomes an optimization target after months of hill-climbing; a production monitor becomes a training signal the moment policy updates react to it. The mechanism has not changed; the causal regime has.

That is why this handbook is organized around three questions rather than a list of tools:

• Creation: how do we turn human intent into an operational signal?
• Validation: why should we believe the signal means what we claim it means?
• Use: what causal role does the signal play, and what happens when a system adapts to it?

The third question is the one most often lost when evaluation is treated as static measurement. Modern AI systems increasingly cannot be trusted by inspection. The evaluator is what we trust instead, so it has to be engineered as carefully as the system it judges.

The shortest map#

Everything in this handbook hangs on one object, the evaluator system:

$$\mathcal{E}_{\text{sys}} = (\mathcal{C}, \mathcal{I}, \mathcal{R}, \mathcal{V}, \mathcal{L})$$

where:

• $\mathcal{C}$ is the claim: the construct, population, environment, harness, validity domain, and the decision the evidence should change;
• $\mathcal{I}$ is the instrument: specification, anchor, evidence view, mechanism, signal, resolution, and aggregation;
• $\mathcal{R}$ is the regime: who consumes the signal, who optimizes against it, how hard, and what they can change or see;
• $\mathcal{V}$ is the validation profile: measured natural error, exploitability, transport, coupling, and drift;
• $\mathcal{L}$ is the lifecycle: versioning, independent channels, anchor burn, tripwires, and frame review.

In plain words: decide what you are trying to infer and why, build the instrument that produces the judgment, be explicit about what the signal will be used for, gather evidence that the inference holds in that use, and keep it holding over time.

A proof checker is a mechanism. A release gate that depends on that checker — under a particular theorem statement, environment, version, trusted base, and threshold — is an evaluator system. Most evaluation failures live in the gap between the two.

At every arrow in the map, ask two questions:

1. Why is this inference justified?
2. What changes when the system adapts to the instrument?

When those questions become reflexive, evaluation stops being benchmark plumbing and becomes an engineering discipline.

What the framework does not collapse#

The framework is unified, but not one-dimensional. It keeps five questions distinct, because each has its own evidence and its own failure modes:

A formal verifier can be perfectly sound and still certify the wrong property. A judge can agree with humans 90% of the time and still be exploited by best-of-256 selection. A monitor can look accurate on a balanced test set and drown operators in false alarms at a 0.1% base rate. No single number answers all five questions.

Who this is for#

This handbook is for people who build, buy, or rely on evaluations of AI systems:

• Engineers shipping models and agents who need release gates and monitors they can defend.
• Researchers who report benchmark results and want the claims to survive scrutiny.
• Evaluation builders designing benchmarks, rubrics, LLM judges, and reward models.
• Reviewers and decision-makers who need to know what a score does and does not support.

You need basic probability — rates, conditional probability, confidence intervals. You do not need a background in psychometrics, formal methods, or reinforcement learning; each idea is introduced where it is first used, with a worked example.

How to use this handbook#

You can read it start to finish or treat it as a lookup table. The parts follow the framework:

• Foundations sets up the vocabulary and the evaluator system.
• The five layers — Claim, Instrument, Regime, Validation (with Validity under pressure), and Lifecycle — each get their own part.
• Applying the framework covers agents and self-improvement, practical workflows such as the One Spine, and seven end-to-end case studies.
• Reference collects the glossary, the notation and formula card, copy-ready templates, and exercises.

Most chapters follow the same shape: a short lead, the key idea in one sentence, the concepts with worked examples, an In practice procedure, the common failure modes, an interactive tool where one helps, and Check yourself prompts that ask you to analyze, evaluate, and create.

Every page is also available as plain Markdown: add .md to any URL, or give your agent /llms.txt. The case studies open directly in the Studio, which walks you through building an evaluator dossier for your own system.

House conventions#

This handbook gives a few words fixed meanings and uses them the same way in every chapter.

Numbers are reported with their uncertainty: "72% (95% Wilson CI 63–80%, n = 100)", never a bare percentage for a sampled estimate.

Notation at a glance#

You will meet a handful of symbols repeatedly. They always appear with their names, and the notation card collects them in one place.

• Claim: $\mathcal{C} = (C, P, E, H, D, \delta)$ — construct, population, environment, harness, validity domain, decision.
• Instrument: $\mathcal{I} = \langle \Psi, A, V, M, Z, R, \Gamma \rangle$ — specification, anchor, evidence view, mechanism, signal, resolution, aggregation.
• Regime: $\mathcal{R} = (\kappa, O, b, S, A_{\text{access}}, K)$ — consumer, optimizer, exposure budget, modification surface, information access, cost envelope.
• Validation profile: $\Pi = (N, E, T, G, C, D)$ — natural error, exploitability, transport, generator coupling, channel coupling, drift.

Further reading#

• The evaluator system — the master object, mechanism versus system, and the lifecycle loop.
• Use regimes: same judge, four jobs — why the same judge needs different evidence in different loops.
• Auditing an eval — seven questions to ask of any benchmark or leaderboard.