Auditing an eval

Auditing an eval means reconstructing the argument that connects its score to a claim, then testing each link against the evidence the eval actually ships with. The same questions serve for someone else's benchmark and for your own evaluator; they are the positive form of the anti-pattern catalog.

What an audit checks#

A score earns trust only through an argument that someone has written down: what it measures, how much it moves under legitimate replication, where the inference holds, and which decision it can support. An audit rebuilds that argument from what the eval ships with and marks every link the evidence does not carry.

An eval should ship with a compact, versioned argument for what its score means and where it does not. In this handbook that record is the sixteen-field evaluator dossier, with a fifteen-item minimum reporting checklist as its short form; both are on Templates.

Take SQLAgent, the text-to-SQL agent of Case C. A leaderboard line saying that it scores 72% is incomplete until a population, an environment version, a policy, and a time window are attached, along with the decision it should inform. The audit finds out which of those the evidence pins down.

The shortest audit: seven questions#

With an hour, ask seven questions, each compressing part of the full audit below. The third column maps each question onto the full audit, and the last audits a minimal SQLAgent write-up: one that describes the construct, tasks, grader, and replications, but no decision or optimization plan.

Questions 4 and 5 are task validity and outcome validity, the two local links that agentic benchmark work emphasizes (checks by output type in Task validity and outcome validity), and the example offers evidence for both. It leaves questions 1 and 7 open, where the framework predicts gaps: use is the question measurement-only thinking most often drops. A gap is a finding, not a verdict: the example may well support a scientific comparison, but as written it does not support a release gate.

The full audit: twenty-five questions#

With a week, work through the full audit. The five parts follow the inference from claim to consequence.

Part A: claim and construct#

1. Precisely what does the score measure: which construct, behavior, risk, or quality?
2. How is the score supposed to be read?
3. Which decision or scientific comparison depends on it?
4. Which other constructs could plausibly explain a high score?

Read with Constructs and Decision claims.

Part B: task and environment evidence#

5. Which population are the tasks supposed to stand for?
6. Where did the tasks come from, and how were they stratified and weighted?
7. Which facets of the construct are covered, and which are missing?
8. What shows that the tasks call on the intended capability instead of a shortcut?
9. Can anything the system reaches through its tools give the answer away?
10. Which parts of the environment are frozen, which simulated, and which live?

Read with Task distributions and Validity domain.

Part C: scorer evidence#

11. For each criterion, what is the strongest available oracle, and is it the one in use?
12. Where grading relies on tests or state matching, what evidence shows their coverage and soundness?
13. Where it relies on learned judges or human raters, whose reference labels were their errors measured against?
14. What are the scorer's sensitivity, specificity, agreement, and calibration on the slices that matter?
15. Has anyone tried to fool the evaluator with adversarial or reward-hacking tests, and what did they find?

Read with Mechanisms and Noisy judges.

Part D: reliability and statistics#

16. What counts as a legitimate replication?
17. How much of the variance was traced to tasks, rollouts, judges, environment snapshots, and harness versions?
18. What are n, the intervals, and each slice's own n?
19. When systems are compared, are they compared on the same tasks?
20. Does the evidence support an effect as large as the one claimed, or only some difference?

Read with Reliability, Estimands and intervals, and Comparing systems.

Part E: external and longitudinal validity#

21. Do independently built measures of the same construct agree with it?
22. Can it be told apart from nuisance constructs it should not track?
23. Does it predict an outcome that matters in deployment or in science?
24. Where does the claim stop, and how would you notice drift?
25. What happens to the score's meaning once people or models optimize against it?

Read with Validity evidence, Adaptive validity, and Living eval suites.

What the framework adds#

Five questions from the evaluator dossier and the validity statement have no direct counterpart above:

• Is each criterion written down, with its allowed evidence, exemplar anchors, abstention rule, and owner, and what may the evaluator inspect?
• Up to what optimization exposure was the evaluator tested, and where does its $b^{*}$ sit?
• Which channels are kept separate for development, certification, and production, how coupled are they, and what does the anchor burn ledger show?
• Where does the evaluate-the-evaluator chain stop: which trust root, with what measured reliability?
• What triggers revalidation and frame review, and who has the authority to revise the construct?

Two drawings and the weakest link#

Questions find gaps one at a time; two drawings show how the gaps connect.

• The coordinate pass. Place each component on its coordinate (claim, specification, reference anchor, evidence view, mechanism, signal, resolution, aggregation, regime), and see which parts of what the paper calls "the evaluation" sit in different layers. SQLAgent's "grader" splits into a specification (result-set equivalence), a reference anchor (the expected result on a pinned snapshot), a mechanism (executor and comparator), and a signal (a verdict per task): four places to be wrong.
• The causal chain. Sketch the path from environment initialization through the trajectory, the evaluator's observation, score computation, and the estimator to the decision, and flag each point where variance unrelated to the construct can get in. For SQLAgent: snapshot and tool versions, retries and timeouts, how the comparator normalizes equivalent result sets, whether rollouts of one task count as independent, and where the threshold sits.

Then walk the One Spine, the framework's ten-step workflow, and name the weakest step. The mapping below shows which audit questions test each step:

The framework's additions cover the uncovered steps. The weakest link is the step whose failure would most change the decision, which is not always the one with the least evidence.

Adopting a public benchmark#

Adopting someone else's benchmark is an audit with a deliverable. The Berkeley course asks teams that integrate an existing benchmark for a quality analysis alongside the integration (Song, 2025, slides 74–77):

• Manual validation. Check a sample of tasks by hand for errors, ambiguity, and difficulty.
• Evaluator check. Confirm that what the metrics or judges reward is what actually counts as success.
• Bias and limitation notes. Write down the gaps and weaknesses you found.

Then correct what is wrong, expand thin coverage, and check faithfulness: your integration should reproduce the original results, apart from the flaws you fixed (slide 88). A strong model scoring near zero in your harness implicates the task or the harness before the model (Anthropic, 2026). SWE-bench Verified shows what such an analysis can find: of 1,699 samples screened by professional developers, 68.3% were filtered out, and GPT-4o resolved 33.2% of the verified tasks against 16% on the original set with the best scaffold, so the invalid tasks had understated capability (slide 78; see the SWE-bench card).

Two checks are specific to adoption. A public benchmark samples its authors' task distribution, not your deployment's, and more tasks only make an estimate of the wrong population more precise. And once your team hill-climbs on a benchmark it used only for reporting, the benchmark becomes an optimization target and its validation obligations move with it (regime migration); its public tasks may also be in pretraining data already (contamination).

What the audit produces#

Every audit ends with three artifacts.

1. Dossier gaps. Mark each of the sixteen dossier fields stated, inferable, or missing. A missing field is a work item for the eval's owner, not automatically a defect; say which decision it blocks.
2. A draft validity statement. Fill the template with what the evidence supports, not with what the paper claims. Mark the tested exposure $b^{*}$ and the validation profile Π as not yet measured wherever the evidence lacks them, and say what remains human judgment.
3. A risk register. One row per finding: the anti-pattern, the tell observed, the layer, the consequence for the decision, the response, and an owner. The format is this handbook's convention; the diagnostic pass in Anti-patterns and failure vocabulary produces its rows.

The builder opens in audit mode, which phrases each field as an auditor's question, on an illustrative SQL-agent statement that, unlike the worked example, names its decision: the head of analytics' rollout decision. All eight slots are filled, but $b^{*}$ and $\Pi$ are marked planned, so the statement reads "not yet tested" and "not yet measured". Switch the example to Blank and write the validity statement your own audit supports.

In practice#

Audit one eval end to end.

1. Name your decision and the claim the eval would need to support. If you cannot, stop: you are auditing eval theater.
2. Ask the seven questions of the eval's paper, documentation, and code, marking each answer stated, inferable, or missing.
3. Draw the coordinate pass and the causal chain.
4. Open the full audit where the seven found gaps, answer each question with evidence or record it as missing, and add the framework's five questions.
5. Name the weakest link: the spine step whose failure would most change your decision.
6. For a public benchmark, run the quality analysis: validate a sample of tasks by hand, check the evaluator against true success, and reproduce the published numbers.
7. Write the three deliverables.

The artifact is an audit report: the seven answers, the gap list, the weakest link, the draft statement, and the risk register.

Common failure modes#

• Face validity as the stopping rule — "the tasks look like real work" → ask for task-validity and outcome-validity evidence; face validity is a smoke test.
• Trusting the headline — a leaderboard number without n, intervals, or slices → ask question 18 first. See one-number reporting.
• Trusting a grader because it is code — a deterministic scorer is accepted as correct because it never varies → audit its reference anchor and test coverage: a perfect exact-match scorer is perfectly faithful to a bad answer key. See Anchors and evidence views.
• Auditing for the wrong use — cleared for reporting, then used to gate releases or to hill-climb → rerun questions 3, 15, and 25 and the framework's additions for the new use. See Goodharting.

Check yourself#

Analyze. Take an agent benchmark you rely on. Using only its paper and repository, answer the seven questions, marking each stated, inferable, or missing, and sketch its causal chain from environment initialization to decision.

Hint

Start with the decision and the behavior under optimization, the two answers the SQLAgent example lacked. On the chain, look hardest at the evaluator's observation and the estimator.

Evaluate. What evidence would you demand before letting a learned judge gate autonomous deployment? Rank it by how much each item could change your decision.

Hint

Work through Parts C and E and the framework's additions: error rates on the gated system's own outputs, adversarial tests, a tested $b^{*}$, an independent certification channel, and a named trust root.

Create. Build your team's audit kit: a one-page template for the seven answers, the gap list, the draft statement, and the risk register, plus a rule for when an audit must be redone.

Hint

Tie the rule to the One Spine's revalidation triggers: material changes to the model, judge, harness, tasks, environment, or users, and metrics that improve while real outcomes do not.

Further reading#

• Templates — the sixteen-field dossier, the fifteen-item checklist, and the validity statement, ready to copy.
• The validation profile and validity statement — what each term of the statement commits you to.
• Anti-patterns and failure vocabulary — the failure modes the audit questions are designed to catch.
• Reading agent benchmarks — six agent benchmarks read in the framework's coordinates.
• Song (2025), "Agent evaluation & project overview," UC Berkeley lecture slides — slides 74–78 and 88 for the quality analysis of an existing benchmark.