Decision claims

A decision claim names the decision an evaluation's evidence is meant to change: which model to deploy, whether a release gate passes, which of N candidates to keep, whether to terminate or escalate a trajectory, which policy update earns reward, or whether an alert pages a human. A score acquires operational meaning only through such a decision.

Validity attaches to a use#

The same measurement can be adequate for one decision and inadequate for another. That is why validity belongs to an interpretation and proposed use rather than to a benchmark in the abstract. The decision is the last coordinate of the claim, $\delta$.

One judge shows how far the same error rates stretch. Give it a true-positive rate of 90% and a false-positive rate of 10% (illustrative). Reporting on a system that truly passes 60% of tasks, it reads 58%, and correcting for its known error rates recovers 60% (The judge as a noisy sensor). Gating 100,000 production requests a day, of which one in 1,000 is a real incident, the same judge raises about 10,080 alerts, and only about 1 in 112 of them is real. Nothing about the judge changed; the decision did.

The widget opens on the reporting job with these settings; switch to Production gate, then to Select best-of-N, and watch the same two error rates serve very different decisions.

Use regimes develops all four jobs.

From a score to a decision claim#

A description is not yet a claim. "Model A scored 72 on Benchmark B" is descriptive. A validity claim starts with an inference such as "Therefore Model A can reliably resolve a target class of customer-service requests under the deployment conditions represented by this benchmark, and the system is suitable for release under policy threshold T." Read it coordinate by coordinate: "reliably resolve" names the construct, "customer-service requests" the population, "the deployment conditions represented by this benchmark" the environment, harness, and validity domain, and "suitable for release under policy threshold T" the decision and its threshold. Any of these can fail even when the score itself was computed perfectly.

This is Marr's computational level: what property, over which population, for which decision. It carries a warning for tooling debates: a more sophisticated grader cannot repair an incorrectly posed computational-level question (see Marr's levels for evaluation).

Each stage needs a different evaluation#

The Berkeley agent-evaluation course lays out where evaluation is needed — training, development, model selection, publication, and deployment — and asks different things of it at each stage (Song, 2025, slide 5). Training and development need very fast, very cheap evaluations free of shortcuts (training signals must also be differentiable), and model selection needs fast, cheap ones. Publication needs standardized, reproducible evaluations with broad coverage, good difficulty, and fine-grained separation between systems, where crude metrics may be fine. Deployment needs evaluations that are trustworthy, task-specific, and absolute.

The contrast matters: a number built to separate systems for publication need not say whether any of them is good enough for a particular deployment. Shipping SQLAgent, the text-to-SQL agent of Case C, needs an absolute statement about its analysts' questions on its warehouse, not a leaderboard rank.

Thresholds and the cost of errors#

A threshold turns a metric into a specification, and choosing it is a product decision, made at the exact point where the computational level touches the algorithmic. Specification develops the difference between a metric and a demand.

A gate also needs an uncertainty rule: a gate such as "pass rate ≥ 95%" is incomplete until it also binds the uncertainty, for example by requiring the interval to exclude the threshold. An illustrative release gate at 95%, read with Wilson intervals, shows why:

The point estimate is the same in every row; the evidence is not. A Beta-binomial posterior can instead give the probability that the true pass rate exceeds 95% (Estimands, pass rates, and intervals).

Where the threshold sits depends on which error costs more, a false pass or a false fail. At a release gate, a false pass ships a regression to users and a false fail holds back a good release. For a monitor that pages a human, a miss lets an incident through, while false alarms cost reviewer attention and, at production base rates, can dominate the alerts. The answer sets the operating point. Moving a threshold trades one error for the other rather than reducing both, and rewording a judge's prompt often shifts its operating point without increasing its fundamental resolution (Usami et al., 2026). When both errors cost too much, you need a better instrument, not a better threshold.

When did this eval last change a decision?#

An evaluation connected to no decision is eval theater: a suite that is run before launches and quoted in reviews but changes nothing. The diagnostic is one question: when did this eval last change a decision? If the answer is "never," it is set dressing.

In a validity statement, the decision enters through the claim and the consumer. The illustrative safety-monitor statement below names its consumer, the block-or-allow decision of the permission layer and on-call, and a claim that carries its own error budget, "at an acceptable false-alarm load". Switch it to Audit mode, where the consumer field asks who or what actually consumes the signal; if the honest answer is a dashboard, you have found theater.

In practice#

Write a decision record before you build or adopt an evaluation.

1. Name the decision and its owner. Write the action that changes, who takes it, and how often.
2. Write the computational-level sentence. State the property, the population and conditions, and the decision with its threshold before you name a metric.
3. Match the evaluation to the stage. Fast, cheap, and shortcut-free for training and development; standardized and discriminating for publication; trustworthy, task-specific, and absolute for deployment.
4. Price both errors. Write what a false pass and a false fail cost, and who bears each.
5. Set the threshold and its uncertainty rule. Record the threshold, its owner, and how uncertainty enters, such as "the 95% interval excludes the threshold".
6. Check for theater. Record when the evaluation last changed the decision; if never, connect it to one or retire it.

The artifact is a decision record holding all six answers. It supplies $\delta$ in the claim and the consumer in the validity statement.

Common failure modes#

• Eval theater — scores are displayed and quoted but never change an action → write the decision first, and ask when the evaluation last changed one. See eval theater.
• Point-estimate gates — the gate compares a bare pass rate with its threshold → bind the uncertainty: require the interval to exclude the threshold, or a posterior probability to clear a stated bar. See one-number reporting.
• A leaderboard as a launch gate — a publication benchmark doubles as the deployment decision → keep it for comparison, and build a task-specific, absolute evaluation for the launch. See Use regimes.

Check yourself#

Analyze. Rewrite "Model A scored 72 on Benchmark B" as a decision claim for a decision you know, and mark which words introduce the construct, the population and conditions, and the decision.

Hint

Compare it with the customer-service claim above. If no phrase names an action and a threshold, you have written an interpretation, not a decision claim.

Evaluate. For a production monitor that pages a human, which costs more, a false alarm or a miss? For a judge that gates a weekly release? Argue how each answer should move the threshold.

Hint

Name who bears each error, then ask whether moving the threshold is enough or whether you need a better instrument. At production base rates, false alarms can swamp real alerts.

Create. Write the decision record for an evaluation your team runs, including the last date the evaluation changed the decision.

Hint

If that last field stays blank, start with the eval-theater diagnostic.

Further reading#

• Use regimes: same judge, four jobs — one judge as reporter, selector, reward, and gate.
• Estimands, pass rates, and intervals — the intervals and posteriors a gate needs.
• Aggregation geometry — gates and weights when several criteria feed one decision.
• Song (2025), slide 5 — what each stage, from training to deployment, needs from an evaluation.