# Introduction

> Evaluation is an engineered chain from intent to evidence to action. What this handbook covers, who it's for, and which path fits what you're trying to do.

**Evaluator engineering** is the discipline of turning a construct into an instrument, validating the inference that instrument supports, and ensuring that the validity survives the causal regime in which its signal is used. This handbook is a practical map of that discipline: the objects, the evidence, and the failure modes.

> [!KEY] In one sentence
> Evaluation is an engineered chain from intent to evidence to action, and every link in it can be designed, measured, and broken.

## Why this handbook exists

Evaluation vocabulary has grown by accretion. Benchmark builders talk about graders, pass rates, and leaderboards. Psychometrics talks about constructs, validity, reliability, and rater error. Reinforcement learning talks about rewards, critics, value models, and optimization. Formal methods talk about specifications, soundness, and trusted computing bases. Agent evaluation adds environments, trajectories, tools, permissions, and side effects. Work on recursive self-improvement adds evaluators that co-evolve with the systems they judge.

These are not rival descriptions. They describe different coordinates of the same engineered system. The trouble starts when the coordinates are collapsed: when "the benchmark is valid" stands in for an argument, when a reward model is called a verifier, or when a judge that was fine for reporting is quietly promoted to a training reward.

AI evaluation now serves at least three causally different purposes:

1. **Measure.** Infer a property of a system for a report, a comparison, or a release decision.
2. **Optimize.** Supply the signal that selects candidates, guides search, or trains a policy.
3. **Monitor or control.** Watch a deployed system and trigger alerts, blocks, rollbacks, or escalation.

The same [evaluator](https://benchmaker.ai/glossary/evaluator) mechanism — a test suite, a proof checker, an LLM judge, a rubric-guided panel — can occupy any of these roles. Its meaning changes with the role. A judge can be excellent for retrospective reporting and unsafe as a reward; a test suite that began as a benchmark becomes an optimization target after months of hill-climbing; a production monitor becomes a training signal the moment policy updates react to it. The mechanism has not changed; the causal regime has.

That is why this handbook is organized around three questions rather than a list of tools:

- **Creation:** how do we turn human intent into an operational signal?
- **Validation:** why should we believe the signal means what we claim it means?
- **Use:** what causal role does the signal play, and what happens when a system adapts to it?

The third question is the one most often lost when evaluation is treated as static measurement. Modern AI systems increasingly cannot be trusted by inspection. The evaluator is what we trust instead, so it has to be engineered as carefully as the system it judges.

## The shortest map

Everything in this handbook hangs on one object, the [evaluator system](https://benchmaker.ai/glossary/evaluator-system):

$$
\mathcal{E}_{\text{sys}} = (\mathcal{C}, \mathcal{I}, \mathcal{R}, \mathcal{V}, \mathcal{L})
$$

where:

- $\mathcal{C}$ is the **claim**: the construct, population, environment, harness, validity domain, and the decision the evidence should change;
- $\mathcal{I}$ is the **instrument**: specification, anchor, evidence view, mechanism, signal, resolution, and aggregation;
- $\mathcal{R}$ is the **regime**: who consumes the signal, who optimizes against it, how hard, and what they can change or see;
- $\mathcal{V}$ is the **validation profile**: measured natural error, exploitability, transport, coupling, and drift;
- $\mathcal{L}$ is the **lifecycle**: versioning, independent channels, anchor burn, tripwires, and frame review.

In plain words: decide what you are trying to infer and why, build the instrument that produces the judgment, be explicit about what the signal will be used for, gather evidence that the inference holds in that use, and keep it holding over time.

```widget
framework-map
```

A proof checker is a mechanism. A release gate that depends on that checker — under a particular theorem statement, environment, version, trusted base, and threshold — is an evaluator system. Most evaluation failures live in the gap between the two.

At every arrow in the map, ask two questions:

1. **Why is this inference justified?**
2. **What changes when the system adapts to the instrument?**

When those questions become reflexive, evaluation stops being benchmark plumbing and becomes an engineering discipline.

## What the framework does not collapse

The framework is unified, but not one-dimensional. It keeps five questions distinct, because each has its own evidence and its own failure modes:

| Lens | The question it asks |
|---|---|
| The verification hierarchy | How strongly is a signal grounded, conditional on a target property? |
| Validity theory | Do the target, task, evidence, scorer, generalization, and use support the intended inference? |
| Reliability theory | How stable is the measurement across legitimate replications? |
| Optimization analysis | How does the evaluator behave after the evaluated system adapts to it? |
| Production assurance | Does the instrument stay calibrated, available, timely, and actionable under real base rates and drift? |

A formal verifier can be perfectly sound and still certify the wrong property. A judge can agree with humans 90% of the time and still be exploited by best-of-256 selection. A monitor can look accurate on a balanced test set and drown operators in false alarms at a 0.1% base rate. No single number answers all five questions.

## Who this is for

This handbook is for people who build, buy, or rely on evaluations of AI systems:

- **Engineers** shipping models and agents who need release gates and monitors they can defend.
- **Researchers** who report benchmark results and want the claims to survive scrutiny.
- **Evaluation builders** designing benchmarks, rubrics, LLM judges, and reward models.
- **Reviewers and decision-makers** who need to know what a score does and does not support.

You need basic probability — rates, conditional probability, confidence intervals. You do not need a background in psychometrics, formal methods, or reinforcement learning; each idea is introduced where it is first used, with a worked example.

## How to use this handbook

You can read it start to finish or treat it as a lookup table. The parts follow the framework:

- **Foundations** sets up the vocabulary and the evaluator system.
- **The five layers** — [Claim](https://benchmaker.ai/handbook/claim), [Instrument](https://benchmaker.ai/handbook/instrument), [Regime](https://benchmaker.ai/handbook/regime), [Validation](https://benchmaker.ai/handbook/validation) (with [Validity under pressure](https://benchmaker.ai/handbook/pressure)), and [Lifecycle](https://benchmaker.ai/handbook/lifecycle) — each get their own part.
- **Applying the framework** covers agents and self-improvement, practical workflows such as [the One Spine](https://benchmaker.ai/handbook/practice/one-spine), and seven end-to-end [case studies](https://benchmaker.ai/handbook/cases).
- **Reference** collects the [glossary](https://benchmaker.ai/handbook/reference/glossary), the [notation and formula card](https://benchmaker.ai/handbook/reference/notation-and-formulas), copy-ready [templates](https://benchmaker.ai/handbook/reference/templates), and [exercises](https://benchmaker.ai/handbook/reference/exercises).

Most chapters follow the same shape: a short lead, the key idea in one sentence, the concepts with worked examples, an **In practice** procedure, the **common failure modes**, an interactive tool where one helps, and **Check yourself** prompts that ask you to analyze, evaluate, and create.

Every page is also available as plain Markdown: add `.md` to any URL, or give your agent [/llms.txt](https://benchmaker.ai/llms.txt). The case studies open directly in the [Studio](https://benchmaker.ai/studio), which walks you through building an evaluator dossier for your own system.

## House conventions

This handbook gives a few words fixed meanings and uses them the same way in every chapter.

> [!CONVENTION] Five conventions used throughout
> - **Specification is Ψ.** A [rubric](https://benchmaker.ai/glossary/rubric) is a specification — an intermediate representation of judgment — not a rung of the verification hierarchy.
> - **Gold is an anchor.** A gold answer is what a comparison is rooted in; the comparator is classified by its own mechanism.
> - **The hierarchy is a prior.** Formal verification > execution feedback > learned judges > intrinsic signals describes expected grounding strength, not validity. Human judgment sits outside the ladder as the external frontier and trust root.
> - **pass@k is not pass^k.** pass@k means at least one of k attempts succeeds; pass^k means all k succeed.
> - **Three regimes.** Signals are used to measure, to optimize, or to monitor and control.

Numbers are reported with their uncertainty: "72% (95% Wilson CI 63–80%, n = 100)", never a bare percentage for a sampled estimate.

## Notation at a glance

You will meet a handful of symbols repeatedly. They always appear with their names, and the [notation card](https://benchmaker.ai/handbook/reference/notation-and-formulas) collects them in one place.

- Claim: $\mathcal{C} = (C, P, E, H, D, \delta)$ — construct, population, environment, harness, validity domain, decision.
- Instrument: $\mathcal{I} = \langle \Psi, A, V, M, Z, R, \Gamma \rangle$ — specification, anchor, evidence view, mechanism, signal, resolution, aggregation.
- Regime: $\mathcal{R} = (\kappa, O, b, S, A_{\text{access}}, K)$ — consumer, optimizer, exposure budget, modification surface, information access, cost envelope.
- Validation profile: $\Pi = (N, E, T, G, C, D)$ — natural error, exploitability, transport, generator coupling, channel coupling, drift.

## Further reading

- [The evaluator system](https://benchmaker.ai/handbook/foundations/evaluator-systems) — the master object, mechanism versus system, and the lifecycle loop.
- [Use regimes: same judge, four jobs](https://benchmaker.ai/handbook/regime/use-regimes) — why the same judge needs different evidence in different loops.
- [Auditing an eval](https://benchmaker.ai/handbook/practice/auditing-an-eval) — seven questions to ask of any benchmark or leaderboard.

---
Source: https://benchmaker.ai/handbook