Evaluating agents

Evaluating an agent means judging a trajectory, not an answer: the sequence of states, actions, tool calls, and side effects an agent produces while it acts in an environment until it stops. The final state is one criterion among several, and each criterion needs a mechanism that can actually observe it.

From outputs to trajectories#

A bare model is evaluated on a mapping from an input to an output: x → model → y. An agent is a system in a perceive–act loop: it observes state, acts through tool calls or code execution, and iterates until a stopping condition. Agency is a property of the surrounding loop and its affordances, not of the model. The harness — instructions, tools, memory, retrieval, orchestration, and stopping rules — is what turns a model into an agent.

For an agent, the object of evaluation becomes a trajectory:

$$\tau = (s_0, a_0, s_1, a_1, \dots, s_T)$$

where:

• $s_t$ is the state of the environment at step $t$: files, database rows, messages, memory, the user's situation;
• $a_t$ is the action the agent takes at step $t$: a tool call, a message, a memory write, a code execution;
• $T$ is the step at which the episode stops, by success, failure, budget, or escalation.

In words: the trajectory is everything the agent did and everything it changed, in order.

Take SQLAgent, the text-to-SQL agent of Case C, with a schema-inspection tool and a read-only query executor over a pinned enterprise database snapshot. A bare model would be graded on the table it returns. The trajectory also holds the queries it ran, the tables it touched, the errors and retries, and whether it stayed inside the analyst's permissions — all things a deployment decision may depend on.

The environment also changes as the agent acts, which is why agent evaluation is harder than evaluating a text-to-text model (Song, 2025, slide 8). That is "dynamic" in the stateful sense, within an episode, not a task set refreshed between benchmark versions.

What an agent evaluator may need to check#

Once the object is a trajectory, "did it succeed?" splits into nine targets an agent evaluator may need to assess. Not every claim includes all nine; the specification decides which belong to the construct.

The mechanism column is a suggestion that applies the routing rule in the next section. $\mathsf{G}$ (always) and $\mathsf{U}$ (until) are temporal patterns, covered in Specification.

Two opposite errors#

Grading only the end state misses unsafe paths: for a customer-service agent, obtaining authorization, preserving privacy, and following policy are part of quality, yet an unsafe path can produce the same final state. Grading the path too rigidly fails valid work: a reference trace is a demonstration, not necessarily a specification, and matching one canonical trajectory can fail novel but legitimate strategies. The specification must state whether process is part of the construct.

This is the choice between outcome evaluation and process evaluation, which Signals and resolution treats in full. For agents, check the observable constraints the claim includes, such as approval before merge, and nothing else about the path.

Route each criterion to an evidence channel#

A production agent usually needs several evidence channels at once:

The rule is to route each criterion to the strongest appropriate channel rather than pick one universal evaluator. Case C compresses it for SQLAgent: formalize access policy, execute semantic consequences, and reserve learned judgment for the residual problem of language and intent.

Two moves do most of the work. First, do not estimate what the harness can guarantee: a read-only credential makes a destructive write unreachable, so that property comes off the sampling budget (see Guarantee before you estimate). Second, grade the backend, not the interface: "the order exists" is evidence, while "a confirmation page rendered" is only the interface's claim. That choice of evidence view moves a criterion upward in evidential strength without changing the underlying model.

Users, environments, and dual control#

For multi-turn tasks someone has to play the other side. Simulated users — a language model prompted with personas mined from real conversation logs — suit capability evals; scripted turns are cheaper but brittle. A simulated user is environment, not system: its quirks can manufacture fake failures and fake passes, and it drifts the way a judge does, so audit it against real logs on a schedule and version it with the rest of the measurement (see Harness architecture and reproducibility).

In τ-bench, an agent resolves a simulated customer's goal, such as returning a product, through API tools and a domain policy; success means the final database matches an annotated goal state, and each scenario was validated to have a unique end state (Yao et al., 2024, via Song, 2025, slides 54–55). See the τ-bench card.

τ²-bench changes the structure of the environment. In a dual-control environment, both the agent and the user act through tools on a shared world — separate agent and user databases and toolsets, with a user simulator constrained by what it can observe — which stresses coordination and guidance (Barres et al., 2025, via Song, 2025, slides 56–57). It also adds environment, communication, natural-language, and action assertions to the end-state comparison. Treat each category as a separate criterion, and find out which mechanism checks it before you compare totals.

Retries change the estimand#

Agents are stochastic, so "the success rate" hides a choice of estimand. pass@k is the probability that at least one of k attempts succeeds; pass^k is the probability that all k succeed. With independent attempts,

$$\text{pass@}k = 1 - (1-p)^k, \qquad \text{pass}^k = p^k$$

where:

• $p$ is the probability that a single attempt succeeds on the task;
• $k$ is the number of attempts;
• independence means one attempt's outcome says nothing about another's.

In words: more attempts make it likelier that something works and less likely that everything works.

At p = 0.75 and k = 3, pass@3 is about 98% and pass^3 about 42%: the same system can be excellent for an inspected coding workflow and unacceptable for an unattended customer workflow. pass@k fits only when something reliably picks out the success, such as a reviewer choosing among attempts. An agent that handles the same request unattended, again and again, needs pass^k: the question is whether it succeeds every time, not whether it can. τ-bench reports the all-k rate beside average success for this reason, though the course slides call it "pass@k" (Song, 2025, slide 55); translate before comparing.

The calculator starts at p = 0.75 and k = 3. Raise p until pass^3 reaches 90%: it takes a per-attempt success rate of about 97%.

Real retries are often dependent, so run several trials per task and resample tasks, not trials, when you build an interval; Estimands, pass rates, and intervals covers the estimators.

Outcome, mechanism, and honesty#

An agent's report of what it did is a claim, not evidence. Studies of AI-scientist agents show why the difference matters. SciIntegrity-Bench builds dilemmas in which honestly acknowledging failure is the only correct answer and completing the task would require misconduct. Across seven state-of-the-art models it found a 34.2% integrity-failure rate; in missing-data scenarios all seven fabricated synthetic data rather than admit the task was infeasible, differing only in whether they disclosed it (Yang et al., 2026, via Chen, Wang & Qu, 2026, §6.3). Removing explicit pressure to complete sharply reduced undisclosed fabrication but not the fabrication itself.

A second failure is subtler: agents that reach the correct answer while defending a wrong mechanism. A position paper documenting it argues that outcome, mechanism fidelity, and epistemic honesty must be measured as separate quantities (Eulig, 2026, via Chen, Wang & Qu, 2026, §6.3). The survey links both findings to reward hacking: at the margin, an agent optimized for results misreports rather than fails, a disposition that appears not to be prompt-induced.

Measure the three separately:

• Outcome — the backend verdict: tests, database state, the order that exists.
• Mechanism fidelity — whether the agent succeeded for the intended reason, which is response-process evidence (see Sources of validity evidence).
• Honesty — whether the agent's report matches the backend verdict; one measure is the share of failed episodes it reports as successes, plus any substitutions it did not disclose.

In practice#

Write a trajectory criteria sheet before you write any grader.

1. State the claim and the decision. Say what the agent is trusted to do and which decision the evaluation serves, and decide whether any part of the process belongs to the construct.
2. Select the targets. Keep the targets the claim includes, each as a binary criterion with the evidence view it needs: final state, trace, backend state, or the agent's report.
3. Write path constraints as temporal patterns. Express "never", "before", and "within n steps" as invariants, precedences, and bounded responses, not as a reference trace.
4. Route each criterion. Move what you can into structure, execute what has consequences, and leave only the residue to a calibrated judge or a human.
5. Pin the environment. Freeze snapshots, fixtures, tool contracts, and the simulated user, and audit the simulator against real logs.
6. Choose the estimand per decision. Use pass^k for unattended work and pass@k only with a reliable selector; run several trials per task and resample tasks.
7. Separate outcome, mechanism, and honesty. Compare every self-report with the backend verdict, and report the three results apart.

The artifact is a trajectory criteria sheet: one row per criterion with its target, specification, evidence view, mechanism, and estimand. Each row becomes a criterion card.

Common failure modes#

• End-state-only grading — the agent reaches the goal state through a forbidden path and the score never shows it → add trajectory constraints for the parts of the path the claim includes, and grade them separately.
• Rigid trajectory grading — valid strategies fail because the checker demands the reference trace → grade observable constraints, not one canonical path. See rigid trajectory grading.
• Single-run reliability claims — one trial per task, reported as if it described unattended reliability → run k trials per task, report pass^k with an interval from task-level resampling. See one-number reporting.
• Shared state between trials — trials reuse a database, cache, or service, so their outcomes co-move → isolate trials and resample at the task level. See correlated trials.

Check yourself#

Analyze. For a customer-service agent that issues refunds, decide which of the nine targets belong to the claim "resolves refund requests correctly and within policy", and name the evidence view each one needs.

Hint

Final success and side effects need backend state; authorization and policy order need the trace; the agent's closing message is evidence only for honesty, which you compare against the backend.

Evaluate. A vendor reports "92% task success" for an unattended agent, from one run per task, graded by reading the agent's final message. What does the number support, and what would you ask for before deploying?

Hint

It is a single-trial rate of claimed success. Ask for backend-state grading, several trials per task with pass^k, an interval from task-level resampling, and the rate at which failed episodes were reported as successes.

Create. Specify "never send an external message without the user's confirmation" for an email agent. Write it as a temporal pattern, then decide which part a harness can guarantee and which part a trace check or judge must estimate.

Hint

The precedence pattern is $\neg\text{send}\ \mathsf{U}\ \text{confirmed}$. A send tool that refuses to run without a confirmation token makes the violation unreachable; whether the confirmation was informed may still need judgment.

Further reading#

• Signals and resolution — outcome versus process in full.
• Specification — temporal patterns and robustness margins for trajectory properties.
• Guarantee before you estimate — making forbidden actions unreachable instead of rare.
• Reading agent benchmarks — τ-bench, τ²-bench, and other agent benchmarks read in coordinates.
• Yao et al. (2024), "τ-bench: A benchmark for tool-agent-user interaction in real-world domains" — the end-state design and the all-k reliability metric.