Evaluation is measurement

An evaluation is a measurement instrument: it supports an inference from observable behavior to a claim about a latent capability, under a defined distribution of tasks, for a particular decision. The score it produces is an observation of that capability, not the capability itself.

Observations, not capabilities#

Classical software can often be trusted through inspection, deterministic tests, and type systems. A modern AI system breaks those conveniences: its policy lives in opaque learned parameters, its inputs cannot be enumerated, its outputs are stochastic, and an agent's behavior depends on tools, memory, environment state, and a long trajectory. The question becomes: what evidence justifies the claim that this deployed system will satisfy the properties we care about, often enough, over the situations we expect it to encounter, and with enough certainty for the decision we must make?

That is a measurement question. What you care about is a construct: a latent capability, risk, behavior, quality, or preference. What you can observe is behavior and its products. The system under test (SUT) is the model, policy, or agent, together with the observation and action pipeline whose behavior is measured; an artifact is the output presented for evaluation, such as an answer, proof, program, report, edited file, or database state. Artifacts, judge scores, and test logs are observable; none of them is the construct, and Constructs, facets, and nuisances shows how to define one so that the difference stays visible.

DocsAgent, the documentation agent of Case G, makes the gap concrete. Its system under test is a model with search and read tools over a documentation corpus, and its artifacts are answers to user questions. The construct is closer to procedurally correct, grounded answers to help requests within the supported product corpus. A judge's verdict on one answer observes that construct through a particular question, document version, rubric, and judge prompt; change any of them and the observation can change while the construct does not. The Berkeley agent-evaluation lecture defines evaluation as "the systematic, repeatable measurement of models and agents" (Song, 2025, slide 4). Repeatability is necessary but not enough: a perfectly repeatable evaluator can reliably measure the wrong construct.

The instrument chain#

The measurement is a chain: target construct, sampled tasks and environments, agent trials and trajectories, grader readings, an estimate with its uncertainty, and a decision. Write $S$ for the system under test and $\psi$ for the success property that the specification defines. A simple deployment estimand is

$$p = \mathbb{E}_{x \sim P}\big[\psi(S(x))\big]$$

where:

• $P$ is the target task distribution, the population of situations the claim is about, and $x$ is a task drawn from it;
• $S(x)$ is the system's behavior on $x$: a response for a bare model, a trajectory $\tau$ for an agent;
• $\psi$ is the specification's success property, 1 when the behavior satisfies it and 0 otherwise;
• $p$ is the expected success rate over the target distribution.

In words: $p$ is how often the system would succeed across the situations the claim is about, and nobody observes it directly.

In practice you sample tasks, generate stochastic trajectories, apply imperfect graders, aggregate the readings, and decide on the estimate. Each step is a link at which the reported number can part company with $p$: a test case is one observation; the instrument is the entire chain. Here is the chain for DocsAgent:

The rows, the decision included, follow Case G.

At every arrow of the chain, ask why the inference is valid and how stable it is under legitimate replication. The framework adds a third question: what changes when the system adapts to the instrument? The evaluator system answers it with a regime layer, which records what the score is used for once the system under test can respond to it.

In the map below, select Claim to find the chain's construct, population, environment, and decision, then Instrument to find the parts of the grader.

Four families of error#

Error can enter at every link. This handbook keeps four error families apart:

Each family answers to different evidence: how the tasks were sampled, repeated rollouts, grader error rates against adjudicated labels, and the estimand, n, and interval.

A second decomposition, from the validation of autonomous systems, separates the system under test, the environment it acts in, and the specification its behavior is checked against (Kochenderfer et al., 2026). This validation triad exposes two ways an eval can be wrong before one even considers sample size or grader noise:

• Wrong world. The eval faithfully measures a distribution or environment other than the deployment world.
• Wrong property. The eval faithfully checks a specification other than the quality, capability, or risk that decision makers intended.

SQLAgent, the text-to-SQL agent of Case C, can fail both ways. Graded against a database snapshot whose schema has since changed in production, it is measured in the wrong world. Graded by exact match against a reference SQL string, it is held to the wrong property, because many semantically equivalent queries return the correct result.

Adding instrument error and estimation error to the two structural failures gives a second set of four: environment, specification, instrument, and estimate. The two sets line up as follows:

Probabilistic links, not equivalences#

Zhu et al. (2025) name the chain's two middle links: task validity, whether success requires the target capability, and outcome validity, whether a positive result means the task was actually done. They state both as equivalences: the task should be solvable if, and only if, the agent has the capability, and the evaluator should return a positive score exactly when the task was completed.

As an engineering ideal this is clarifying; as a theory of measurement it is too strong. A capable agent can fail through stochastic decoding, a transient tool error, or one particular environment realization; a weaker agent can succeed by chance or by a legitimate strategy nobody anticipated; and every grader has a domain of applicability and an error process. The general model is probabilistic:

$$\begin{aligned} &\Pr[T = 1 \mid C, E, H] \\ &\Pr[G = 1 \mid T, \hat g, H] \end{aligned}$$

where:

• $T = 1$ means the task was actually completed, and $G = 1$ means the grader returned a positive grade;
• $C$ is the target capability, $E$ the environment, and $H$ the harness and administration conditions;
• $\hat g$ is the grader.

In words: capability makes success likely rather than certain, and success makes a pass likely rather than certain. Good design makes both probabilities strong in the intended direction and audits the remaining routes by which irrelevant variables can move the result. Because either link can fail while the score itself is computed perfectly, validity attaches to an interpretation and a use of the score, not to the test object. The validity argument places the two links among the six claims a score has to survive, with a SQLAgent example of each link weakening.

Four words, five questions#

The validation literature fixes four words that practice uses loosely (Kochenderfer et al., 2026). Verification gives guarantees with respect to a formal specification and is largely unavailable for the learned core of an LLM system beyond narrow properties: a check that SQLAgent's queries parse against the schema guarantees syntax, not correct answers. Validation asks the empirical, statistical question of whether the system behaves as desired in the real world: do analysts accept SQLAgent's answers in shadow deployment? Evaluation is the concrete act of measurement: input, system, grading logic, number. Testing evaluates the system on a discrete set of cases, so an eval suite is testing, minus the everyday connotation that one deterministic run settles the matter.

That system-level validation is broader than this handbook's validation layer, which asks whether evidence supports the interpretation and use of an instrument's scores. The framework also keeps five questions apart, each with its own evidence: grounding (the verification hierarchy), validity, reliability, optimization robustness, and production assurance. The instrument chain touches all five, and an evaluator can answer one well and another badly.

In practice#

Draw the instrument chain for one number you report.

1. Write the sentence the number supports. Name the construct, population, environment, and decision. If all you can write is "the model scored 72", there is no claim yet.
2. Write the estimand. State $P$, $\psi$, and $S$ precisely enough that someone else could tell whether a new task belongs to $P$ and whether a behavior satisfies $\psi$.
3. Fill in each link. Name the task source, environment and harness versions, trials per task, grader and version, estimator, and decision rule.
4. Assign the error families. At each link, write the error that enters and your evidence about its size: task mix against deployment, repeated rollouts, grader error against adjudicated labels, n with an interval.
5. Check the structural failures. Is this the deployment world? Would decision makers accept $\psi$ as the quality they meant?
6. Ask the arrow questions. Why is each inference justified, how stable is it under replication, and what changes when the system adapts?

The artifact is an instrument-chain sheet, one line per link with its error family and evidence; it seeds the claim, task-population, specification, and statistics fields of the evaluator dossier (see Templates).

Common failure modes#

• Score mistaken for capability — "the model has 72% reasoning" appears in a release review → state the population, environment version, policy, and time window the number holds for. See one-number reporting.
• A precise estimate of the wrong world — a large convenience sample gives a tight interval around a population nobody deploys to → reweight where the sample has support and add tasks where it has none. See Task distributions and dataset engineering.
• A better grader for the wrong property — effort goes into judge accuracy while the specification checks something nobody intended → fix the construct and specification first; a perfect grader cannot repair a task that measures the wrong capability. See Constructs, facets, and nuisances.
• Correlated trials counted as independent — reruns share state but enter the interval as separate samples → isolate trials and resample at the task level. See correlated trials.

Check yourself#

Analyze. For an agent benchmark you know, draw the causal chain from environment initialization through the trajectory, evaluator observation, score computation, and estimator to the decision, and mark every place construct-irrelevant variance can enter.

Hint

Walk the four error families link by link, then the two structural failures. Timeouts, retries, and tool availability often hide between environment and trajectory.

Evaluate. SQLAgent's pass rate rose sharply in the week its team replaced exact SQL-string matching with execution and semantic result-set comparison. Did SQLAgent improve?

Hint

The system did not change; the grading link did. Because $\psi$ changed, the two numbers estimate different estimands. String matching rejected semantically equivalent queries: the new number is closer to the intended property, not evidence of a better agent.

Create. SQLAgent was certified on a pinned snapshot at schema v12, and production has since moved to v13. Design the evidence that would tell you whether the eval now measures the wrong world, and say what you would change in the chain if it does.

Hint

Compare population and environment link by link: which tables, joins, and question types changed, and do v12 results predict results on a fresh v13 sample? Where v13 has regions the old suite never sampled, reweighting cannot help; only new tasks can.

Further reading#

• The evaluator system — the five layers that organize the chain, including the regime the chain leaves out.
• Task validity and outcome validity — the chain's two middle links, with checks by output type.
• Estimands, pass rates, and intervals — the estimand, n, and the intervals that carry estimation error.
• Zhu et al. (2025), "Establishing best practices for building rigorous agentic benchmarks" — the chains behind task and outcome validity.
• Kochenderfer, Katz, Corso, and Moss (2026), Algorithms for Validation — the system–environment–specification decomposition.