# Specification: what counts as success

> Turn a latent construct into a checkable demand, outcome or trajectory, boolean or margin, without mistaking formality for validity.

A **specification** $\Psi$ is the operational statement of what counts as success: the demand an artifact or trajectory must meet. It is where a latent construct becomes something a mechanism can check, and the first coordinate of the instrument that produces every judgment.

> [!KEY] In one sentence
> A specification turns a latent construct into a checkable demand, on an outcome or a trajectory, as a verdict or a margin, and its formality buys fidelity to the property you wrote, never validity for the property you chose.

## The instrument at a glance

The [instrument](https://benchmaker.ai/framework#instrument) layer answers one question: how is the judgment actually produced? It has seven coordinates:

$$
\mathcal{I} = \langle \Psi, A, V, M, Z, R, \Gamma \rangle
$$

where:

- $\Psi$ is the specification: what counts as success;
- $A$ is the reference [anchor](https://benchmaker.ai/glossary/anchor): what the comparison is rooted in;
- $V$ is the [evidence view](https://benchmaker.ai/glossary/evidence-view): what the evaluator may observe;
- $M$ is the [mechanism](https://benchmaker.ai/glossary/mechanism): what computes the judgment;
- $Z$ is the [signal](https://benchmaker.ai/glossary/signal): what leaves the evaluator;
- $R$ is the [resolution](https://benchmaker.ai/glossary/resolution): what unit is judged;
- $\Gamma$ is the [aggregation geometry](https://benchmaker.ai/glossary/aggregation-geometry): how criterion-level judgments become a decision.

In plain words: say what success is, what it is checked against, what the checker may see, what does the checking, what comes out, at what grain, and how the pieces add up to a decision.

Most coordinates are chosen per criterion rather than once per benchmark. SQLAgent, the text-to-SQL agent of [Case C](https://benchmaker.ai/handbook/cases/sql-data-agent), graded against a pinned snapshot of a finance warehouse, shows the range:

| Coordinate | SQLAgent | Where to read |
|---|---|---|
| Specification $\Psi$ | The query executes, the result is semantically correct, and permission and retry constraints are respected | This chapter; [Rubrics and criterion cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards) |
| Reference anchor $A$ | A reference result set on the pinned snapshot | [Anchors and evidence views](https://benchmaker.ai/handbook/instrument/anchors-and-evidence-views) |
| Evidence view $V$ | The returned rows, the tool trace, and the database state before and after the episode | [Anchors and evidence views](https://benchmaker.ai/handbook/instrument/anchors-and-evidence-views) |
| Mechanism $M$ | Execution with semantic result-set comparison; a rubric-guided judge for ambiguity handling; SME adjudication of edge cases | [Mechanism families](https://benchmaker.ai/handbook/instrument/mechanisms); [LLM judges in practice](https://benchmaker.ai/handbook/instrument/llm-judges) |
| Signal $Z$ | A pass, a fail, or an abstention per criterion, with a margin where one exists | [Signals and resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution) |
| Resolution $R$ | The final result set for correctness; every statement in the trajectory for write permission | [Signals and resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution) |
| Aggregation $\Gamma$ | A write outside the granted scope fails the trial, however good the answer | [Aggregation geometry](https://benchmaker.ai/handbook/instrument/aggregation-geometry) |

The specification, reference anchor, mechanism, and resolution entries follow Case C; the other entries are illustrative. The rest of this chapter is about the first row.

## Forms a specification can take

A specification can be written in many forms, most of them met in the case studies:

| Form | Example |
|---|---|
| A formal theorem or logical property | The theorem statement a proof assistant checks, for a math benchmark's formalized subset (Case A) |
| A unit, integration, or property-based test suite | Reference tests from the fix linked to a repository issue (Case B) |
| A database-state invariant | No data outside the agent's granted scope changes during an episode (Case C) |
| A rubric | Separate criteria for correctness, grounding, procedure, and abstention (DocsAgent) |
| A set of acceptable states | Backend states in which the order exists, whatever the interface displays |
| A preference relation | "Which response do sampled users prefer?", when preference is the construct |
| A temporal property | Never send a message before the user confirms it (Case F) |

Whatever its form, a specification carries more than the positive goal: invariants and forbidden actions, temporal ordering, resource, latency, and cost limits, abstention or escalation requirements, and requirements on uncertainty and evidence.

The diagnosis is plain: many apparent grader problems are actually specification problems. If two domain experts cannot agree on what success means from the task and rubric, improving the judge model only automates the ambiguity. Suppose reviewers of DocsAgent, the documentation agent of [Case G](https://benchmaker.ai/handbook/cases/docs-agent), split on answers that are correct for the current product version but cite a page written for the previous one. No judge can be calibrated against that split until the specification says which version counts. The fix is a sentence in the specification, not a new prompt, which is why the calibration protocol in [LLM judges in practice](https://benchmaker.ai/handbook/instrument/llm-judges) revises the specification before blaming the judge.

Formality does not settle the deeper question. Formality improves fidelity to the specified property; it does not validate the choice of property. A proof checker can certify the wrong theorem flawlessly, and a unit test can pass without ever exercising the behavior users care about. Whether the property is the right one is a question for the [construct](https://benchmaker.ai/glossary/construct) and the [validity argument](https://benchmaker.ai/handbook/validation/validity-argument); rigor inside the instrument cannot answer it.

## Metrics are not specifications

Two objects that evaluation practice often conflates need separate names ([Kochenderfer et al., 2026](https://benchmaker.ai/handbook/reference/sources#kochenderfer-2026)). A **metric** measures: it maps a trajectory to a quantity, such as latency, tool calls used, or a judge score. Because the system is stochastic, a metric has a distribution over the task population, and whichever summary you report, whether mean, median, p95, or worst case, is one functional chosen from it. A specification demands: it evaluates to true or false, per trajectory ("this trial cited a real document") or as a system-level requirement such as $\Pr[\text{latency} < 10\ \text{s}] \ge 0.95$.

A metric without an attached demand is a dashboard; a demand without a measured metric is a wish. Thresholding bridges from metric to specification, and where to put the threshold is a product decision, not a property of the metric ([decision claims](https://benchmaker.ai/handbook/claim/decision-claims#thresholds-and-the-cost-of-errors)). Robustness bridges back, from a verdict to a graded quantity, as the last section shows.

The distinction has practical consequences. Choosing the functional is choosing whom to protect: an average serves typical users, a p95 or a worst case serves the unlucky ones, and safety properties call for tail functionals rather than means ([aggregation geometry](https://benchmaker.ai/handbook/instrument/aggregation-geometry)). And a gate such as "pass rate at least 95%" ignores sampling error until it also says where the interval must lie relative to the threshold ([estimands and intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals)). For SQLAgent, "correct-result rate of at least 95%, with the lower bound of the 95% Wilson interval above 95%" is a specification; a bare 95% on a dashboard is a metric *(illustrative)*.

## Outcome or trajectory

An agent produces a [trajectory](https://benchmaker.ai/glossary/trajectory), $\tau = (s_0, a_0, s_1, a_1, \dots, s_T)$, and the specification must state whether the path is part of the construct. Sometimes only the end state matters: any valid derivation should earn a math solver credit, and any semantically equivalent query should earn a database agent credit. Sometimes the path is part of quality: a customer-service agent has to obtain authorization, protect privacy, and follow policy on its way to the right end state.

Both directions fail. A terminal success flag can miss unsafe trajectories entirely: an agent that edits ledgers can make an unsafe intermediate write and still end with the correct balance. Conversely, a reference trace invites a grader that demands one canonical path. Reference traces are demonstrations, not necessarily specifications, so having a trace on file is no reason to constrain the process.

SQLAgent needs both kinds. Correctness is an outcome specification: any query that returns the right rows passes. Write permission is a trajectory specification, because a destructive statement followed by a repair can leave the final state intact. Who checks the trajectory part is a separate choice of mechanism and resolution: a deterministic trajectory checker is process-level without being learned, so a process verifier is not a PRM ([signals and resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution)).

## Trajectory properties as temporal patterns

Trajectory assertions reduce to a small set of **temporal patterns**: state predicates joined by boolean connectives and extended through time with $\mathsf{G}$ (always), $\mathsf{F}$ (eventually), $\mathsf{X}$ (next), and $\mathsf{U}$ (until), plus requirements on which states are reachable. The value lies in the vocabulary, not in a model checker: a named pattern is an unambiguous assertion, and checking it over a transcript takes a short program.

| Pattern | Formula | Example |
|---|---|---|
| Safety invariant | $\mathsf{G}\,\neg\,\text{destructive\_call}$ | At no step does the agent issue a statement other than a SELECT |
| Precedence | $\neg\,\text{run\_query}\ \mathsf{U}\ \text{schema\_checked}$ | No query against an unfamiliar table until its schema has been fetched |
| Liveness (goal) | $\mathsf{F}\,(\text{answer} \land \text{has\_citation})$ | The trial eventually produces a cited answer |
| Bounded response | $\mathsf{G}\,(\text{error} \rightarrow \mathsf{F}_{\le 3}\,(\text{fixed} \lor \text{escalated}))$ | Every database error is fixed or escalated within three retries |
| Reachability (unsafe set) | A set of forbidden states | Personal data never leaves the tool boundary |

Two properties make the vocabulary worth learning. First, one property serves twice: an invariant that grades evaluation trials offline can watch production traffic online, so [production monitoring](https://benchmaker.ai/handbook/regime/production-monitoring) can reuse the specification. Second, patterns separate the constraints a claim includes from incidental features of a demonstration. The coding agent's approval rule, $\neg\,\text{merge}\ \mathsf{U}\ \text{approved}$, says nothing about the order in which files are edited ([Case B](https://benchmaker.ai/handbook/cases/coding-agent)). Where a pattern can be enforced by construction, it leaves the sampling budget altogether ([structural guarantees](https://benchmaker.ai/handbook/practice/structural-guarantees)).

## Margins, not only verdicts

A boolean specification says whether $\psi$ held; a **robustness margin** says by how much. Validation theory attaches to a specification a signed number $\rho(\psi, \tau)$ whose sign says whether $\tau$ satisfies $\psi$ and whose size says by how much. For a latency requirement:

$$
\rho(\psi, \tau) = 10\ \text{s} - \text{latency}(\tau)
$$

where:

- $\psi$ is the predicate one check evaluates, here $\text{latency}(\tau) < 10\ \text{s}$;
- $\tau$ is one trajectory;
- $\rho(\psi, \tau)$ is the margin, in seconds.

In words: the margin is how far inside or outside the requirement a trajectory landed. Trajectories that finish in 9.8 s and in 2 s both pass, with margins of 0.2 s and 8 s: the verdicts agree, the facts do not, and $\rho$ keeps the difference.

Margins have three uses:

- **Partial credit.** Graded continua, such as three of four subgoals solved, keep "failed immediately" and "almost succeeded" apart.
- **Early warning.** If the pass rate holds while the average margin collapses, a regression has been caught one release before the verdicts show it.
- **Search guidance.** Search needs a slope, and a pass/fail grader is a flat step; a margin gives adversarial testing something to descend ([falsification](https://benchmaker.ai/handbook/pressure/falsification)).

Whenever you design a binary check, ask for its natural margin; one is usually close at hand, such as a score minus its threshold, a budget minus its usage, or a distance from the unsafe set. SQLAgent's bounded response has one in plain sight: three permitted retries minus the retries used. A margin is a property of one trajectory relative to the specification, not a learned process score or a policy-conditioned value estimate, and the three should never be conflated ([Signals and resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution#three-objects-that-look-alike)).

## In practice

Write the specification for one criterion before you choose anything that checks it.

1. **Start from the claim.** Name the construct facet the criterion represents and the decision it serves ([constructs](https://benchmaker.ai/handbook/claim/constructs)).
2. **Choose the form.** Prefer a form a stronger mechanism can check, but not at the cost of narrowing the construct.
3. **Write the demand.** State it as true or false, per trajectory or system-wide. If it thresholds a metric, name the functional, the threshold, and the uncertainty rule.
4. **Decide outcome or trajectory.** Write one sentence on whether the path is part of the construct; if it is, name the temporal pattern.
5. **Add negative and abstention requirements.** List forbidden actions and limits, and when "cannot determine" or escalation is the correct output.
6. **Name the margin.** Write down each binary check's natural $\rho$, and log it beside the verdict.
7. **Test it on experts.** Have two experts apply it to the same hard cases, and revise it where they split.

The artifact is a specification sheet per criterion: demand and form, outcome or trajectory, temporal pattern, margin, negative and abstention requirements, and why this property operationalizes the construct. It becomes the definition field of a [criterion card](https://benchmaker.ai/glossary/criterion-card) and the specification entry of the evaluator dossier.

## Common failure modes

- **A dashboard without a demand** — a metric is charted, but no threshold, functional, or decision consumes it → attach a specification or retire the number. See [eval theater](https://benchmaker.ai/handbook/practice/anti-patterns#eval-theater).
- **The demonstration as the specification** — valid strategies fail because they deviate from a reference trace → grade only the constraints the claim includes. See [rigid trajectory grading](https://benchmaker.ai/handbook/practice/anti-patterns#rigid-trajectory-grading).
- **Terminal state only** — an unsafe path that ends in the right state passes → write temporal patterns for the forbidden paths and check them over the trace.
- **A better judge for an ambiguous specification** — experts disagree, and the team tunes the judge prompt → revise the specification until experts agree, then calibrate.

## Check yourself

**Analyze.** Pick a benchmark's headline number. Is it a metric or a specification? Write the demand it would need and the decision that would consume it.

> [!DETAILS] Hint
> Start from the decision. If nothing consumes the number, it is a dashboard; if something does, write the functional, the threshold, and where the interval must sit.

**Evaluate.** A team grades a customer-service agent by comparing its tool calls with an expert's reference trace. Which parts of the path belong in the specification?

> [!DETAILS] Hint
> Authorization, privacy, and policy steps are part of the construct and become temporal patterns; the order of harmless lookups belongs to one demonstration.

**Create.** An agent sends external messages on a user's behalf. Write a safety invariant, a precedence, and a bounded response for it, each with a natural margin.

> [!DETAILS] Hint
> The precedence is Case F's $\neg\,\text{send}\ \mathsf{U}\ \text{confirmed}$. Margins can count retries or steps left; a property enforced structurally may need none.

## Further reading

- [Rubrics and criterion cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards) — the specification in rubric form, one criterion per card.
- [Signals and resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution) — why margins, PRM scores, and value estimates are different objects.
- [Falsification and failure regions](https://benchmaker.ai/handbook/pressure/falsification) — margins as the slope that failure search descends.
- [Monitoring and control in production](https://benchmaker.ai/handbook/regime/production-monitoring) — the same temporal property as a runtime monitor.
- Kochenderfer et al. (2026), *Algorithms for Validation* — the validation-theory source of the metric–specification distinction, trajectory patterns, and robustness.

---
Source: https://benchmaker.ai/handbook/instrument/specification