Five questions behind every score

SQLAgent answers finance analysts' questions by writing SQL and running it against the company's data warehouse. On a test set of 200 questions it returns the right answer to 192 of them: 96%. Should the team roll it out? Five questions stand between that number and the decision.

The agent, the warehouse, and every number on this page are a worked example, the one the handbook develops in full as the SQL data agent case.

Each question is one layer of an evaluator system: the whole arrangement that turns a claim about an AI system into a number someone acts on. Evaluator engineering is the discipline of designing that arrangement on purpose: turning what you care about into a measurement, checking that the measurement supports the inference you draw from it, and keeping that true in the loop where the number is used. Each layer is a place where the inference from score to decision can break while every component works as built.

Claim: 96% of what?#

A score is evidence about something no test observes directly. The SQLAgent team cares about a construct, a property of the agent: it answers analysts' questions correctly, without reading or writing data it should not touch. A test observes 200 answers. The claim says how far those answers are allowed to speak.

So the claim fixes four things before anything is measured:

• Population: the questions the score speaks for. Here, questions finance analysts actually logged over two quarters, sampled by kind and difficulty.
• Environment: where the agent runs. A frozen copy of the finance warehouse at schema version 12, under read-only analyst credentials.
• Validity domain: where the claim stops. English questions about this warehouse at the analyst permission level; not other teams' data, and not after the schema changes.
• Decision: what changes when the number moves. The head of analytics rolls the agent out to the finance team, or holds it.

The population does more work than it seems to. A score speaks only for the kinds of question it sampled. If the 200 test questions are mostly single-table lookups, 96% says little about joins, fiscal calendars, or questions that are ambiguous as asked, which is where the job gets hard.

Write the decision down first. If no decision would change when the number moves, the evaluation is eval theater: a score on a dashboard that nobody acts on.

Instrument: Measured how?#

The instrument is the machinery that turns the claim into a verdict on each question. It has more parts than "the metric":

• a specification, written Ψ, of what counts as a correct answer;
• a reference anchor, the external object the verdict is rooted in: here, a result table from a query an analyst has checked;
• an evidence view: what the checker may look at, from the final table to the full tool trace;
• a mechanism that computes the verdict;
• the signal it emits, here pass or fail for each question;
• and an aggregation rule that turns 200 verdicts into a decision.

The obvious mechanism compares the agent's SQL with the reference SQL, and it fails at once: the same question can be answered by many different queries. The better mechanism runs the agent's query on the frozen snapshot and compares the rows it returns. It grades what the query does, not how it is spelled. Even that can be fooled, and a metamorphic test, a check of a relation the right answer must satisfy, is how you catch it.

SELECT r.name AS region,       SUM(i.amount) AS revenueFROM invoices iJOIN regions r ON r.region_id = i.region_idWHERE i.invoiced_on >= '2024-01-01'  AND i.invoiced_on <  '2025-01-01'GROUP BY r.nameORDER BY r.name;

WITH sales_2024 AS (  SELECT region_id, amount  FROM invoices  WHERE EXTRACT(YEAR FROM invoiced_on) = 2024)SELECT regions.name, SUM(amount)FROM sales_2024JOIN regions USING (region_id)GROUP BY 1;

Correctness is not the only thing the claim promises, and different promises need different checks. The design rule is to route each criterion to the strongest check you can afford:

That ordering is the verification hierarchy: formal verification, then execution feedback, then learned judges, then intrinsic signals such as the model's own confidence. A rung is a prior to start from, not a verdict. It says how strongly a signal is expected to be grounded; only measurement says how well it is.

Regime: Used for what?#

This is the question most evaluations leave out. SQLAgent's checks serve three jobs. The result-set check that certifies a rollout also scores developers' prompt experiments, and the access-policy check that grades certification traces also runs on every live query:

• a rollout gate (measure): the head of analytics reads aggregate scores from a hidden certification set and decides;
• development feedback (optimize): developers keep or drop prompt and tool changes depending on whether the score on a separate development split goes up;
• an access monitor (monitor-control): every live query is checked within a second, and a suspect one is blocked while a human is paged.

The code is the same in each job. The meaning of its verdict is not. As a gate, the question is whether the score means what the claim says. As development feedback, the agent is being shaped by the score, and a score that is optimized against can keep rising after the agent stops getting better. As a monitor, base rates dominate: when real violations are rare, even a small false-alarm rate means most pages are false.

This is the use regime: the causal role a signal plays, written measure / optimize / monitor-control. Classify a signal by the behavior it changes, not by the dashboard it sits on, and watch for regime migration. If monitor alerts start feeding prompt changes, the monitor has become an optimization target, and its validation has to move with it.

Why monitor at all, when credentials already make writes unreachable? Because the guarantee holds only for the roles and connection paths that were configured. The monitor checks the same rule on every live query, in case something escapes that configuration.

Validation: Why believe it?#

Validation asks whether the evidence supports reading the score as the claim, and it asks twice. Natural validity concerns ordinary measurement: does the signal track the claim when nobody is optimizing against it? Adaptive validity concerns pressure: does it still track the claim after developers have tuned the agent against it?

Start with the plainest evidence there is, the uncertainty of a sample. 192 correct out of 200 is 96% (95% Wilson CI 92–98%, n = 200). The rollout bar is 95%. The point estimate clears the bar; the interval does not. A gate that reads only the point estimate lets sampling noise make the call. Gate on the interval, or size the certification set to the decision.

Then check the checker. Every part of the instrument has an error rate of its own, and the validation profile collects them as measurements. For SQLAgent it includes:

• Natural error: how often result-set comparison accepts a wrong query that returns the right rows by accident, or rejects a right one over column order or aliases, judged against expert adjudication; and how well the judge agrees with the finance experts, as true- and false-positive rates and Cohen's κ.
• Exploitability: each development-tuned version of the agent scored on the hidden certification set, to see how much tuning the evaluator absorbs before the two disagree.
• Transport: whether rankings hold on alternate snapshots of the same schema.
• Generator coupling: whether the judge errs exactly where the agent errs, which is more likely when both come from one model family.

For SQLAgent, every one of these is planned, not yet measured, and an honest report says so. The hierarchy rung was a prior; the validation profile replaces it with measurements.

Lifecycle: Still true next quarter?#

Validity is not something an evaluator has once and keeps. The evidence above was gathered at schema version 12, on last quarter's questions, with one version of the judge, against a certification set nobody had tuned on. Each of those changes: the schema moves past version 12, the question mix shifts, the judge model is updated, certification questions leak into development, or the easy questions saturate and stop telling versions apart.

The lifecycle layer keeps the claim true while that happens:

• Separate channels. Keep the development split, the hidden certification set, and production signals apart, so tuning cannot quietly use up the test. Each look at the hidden set spends some of its independence, and an anchor-burn ledger counts the looks.
• Version together. The snapshot, rubric, judge, and threshold are versioned as one instrument, and a schema change triggers revalidation.
• Tripwires. Declare the warning patterns in advance. Development rising while certification stays flat suggests tuning is chasing the proxy. Both rising while analysts correct more answers suggests a construct or deployment mismatch. A kind of question at its ceiling signals saturation.
• Frame review. Every so often, ask the prior question: are correct rows still what the finance team needs from the agent?

The whole argument in one sentence#

Everything above compresses into a validity statement: one sentence that says what the evidence supports, for whom, where, under what pressure, and where that support stops. For SQLAgent, stated as a plan until the profile is measured:

The lifecycle has no clause of its own. Its job is to keep the sentence true, and to say when it has expired. A clause you cannot fill marks a hole in the evaluation, and the statement shows exactly where it is.