Reading agent benchmarks

An agent benchmark is a task population, an environment, and an instrument that together turn an agent's behavior into a score. Reading one means recovering its coordinates — the scope of claim it supports, its place on four design axes, and how it grades — before you trust what its leaderboard says.

Three scopes of claim#

Agent benchmarks differ first in the scope of the claim they can carry. The Berkeley agentic AI course (Song, 2025, slides 25–38) sorts them into three groups, using a taxonomy from arXiv:2503.16416↗ (opens in a new tab):

The scope fixes the population a score speaks for. A capability-scoped benchmark supports inferences about a construct facet, an application-scoped one about a deployment domain, and a general-purpose one makes the strongest and therefore most fragile claim, because it must defend representation across many domains at once. Population, environment, harness, and validity domain develops claim scope in full.

Scope belongs to the claim, not to the benchmark. The taxonomy files CRMArena with the generalist evaluations, while the course presents it as a benchmark of professional customer-relationship-management workflows (Song, 2025, slides 38, 60). The same tasks license different sentences under each reading, so state the scope you are claiming.

Four design axes#

Scope says whom a score speaks for. Four design axes locate how the benchmark is built:

The axes are independent. Open does not mean subjective: a program is open-ended but verifiable by execution, while a closed-ended label can still be invalid if the gold label is ambiguous. Oracle strength is the most consequential axis, and "non-verifiable" usually means that ground truth is expensive, plural, or unavailable at the needed granularity, not that it is unverifiable in principle. Maintenance and interaction are both called "dynamic"; keep them apart. A refreshed corpus can still be run as a frozen, reproducible measurement — versioned dynamism, covered in Living eval suites.

Where a benchmark sits predicts what to check first. For an interactive, verifiable benchmark the primary risks are simulator fidelity, hidden side effects, and confusion of path with outcome. An interactive, judgment-based benchmark adds correlated judge and system bias, user-simulator validity, and long-horizon credit assignment. A versioned dynamic benchmark risks losing longitudinal comparability or silently changing what a score means.

The cards on this site show the axes at work. WebArena runs on self-hosted replicas of real web applications, so its stateful tasks can be reproduced, while AssistantBench runs on the live web, so its scores are tied to the date they were collected (WebArena, AssistantBench). A site can be realistic without being the live public site. LiveCodeBench keeps adding problems and compares models on release-date windows that can start after their training cutoffs, a versioned dynamic design (LiveCodeBench).

Six benchmarks in coordinates#

The course's case studies (Song, 2025, slides 47–63) read the same way once you ask three questions: what scope, what specification and reference anchor, and what mechanism observing what.

The rows follow Wang et al. (2025), Yao et al. (2024), Barres et al. (2025), OpenAI (2025), Huang et al. (2025), and LegalAgentBench (arXiv:2412.17259), all as presented by Song (2025, slides 51–63). How each builds its tasks is compared in Task distributions and dataset engineering; here the question is how each grades. Each teaches one thing:

• CyberGym — the oracle is a differential execution check: the same proof-of-concept runs against the unpatched and patched builds, and sanitizers decide, with no subjective judging. The overall success rate pools two routes, reproducing the target vulnerability before the patch and triggering any vulnerability after it, so read them separately if your claim concerns one. Contamination is checked by scoring vulnerabilities found after model knowledge cutoffs.
• τ-bench — a reference end state is only as good as its uniqueness, which is why each scenario was validated with many agent runs. The customer is played by a language model, so its fidelity is part of the benchmark's validity.
• τ²-bench — dual control changes the construct from acting alone to coordinating with a user who also acts. Four assertion categories make the score a composite; find out what checks each before comparing totals.
• GDPval — the mechanism is a human panel and the reference anchor is an expert's work, so the signal is a pairwise preference whose quality rests on the graders' expertise and agreement, both of which can be measured. The course cites it as low in contamination risk and hard to saturate, because its tasks need domain experts and are tied to concrete work products.
• CRMArena — exact match on IDs is a strict deterministic check against a fixed key, while F1 on knowledge answers measures overlap with a reference, a weak comparator. It reports turns, tokens, and dollar cost beside the score. The environment is a real CRM product, but its data is LLM-synthesized with planted latent variables, so the population is synthetic even where the environment is real.
• LegalAgentBench — keyword overlap and BERTScore are weak comparators against a reference anchor, not semantic verification. Keyword matching is exposed to the failures the outcome-validity checks for text target, such as negation and answers that list many possibilities. Its progress rate measures the path with the same weak comparator.

Check a benchmark before you adopt it#

The course asks anyone integrating an existing benchmark to analyze its quality first (Song, 2025, slides 74–77): validate a sample by hand for correctness, clarity, and difficulty; check that metrics and judges align with true task success; record biases and limitations; then correct and expand the benchmark where it is wrong. SWE-bench Verified is the worked example: after professional developers screened 1,699 samples, 68.3% were filtered out, and GPT-4o resolved 33.2% of the 500 verified tasks against 16% on the original set with the best scaffold (OpenAI, 2024, via Song, 2025, slide 78). Task validity and outcome validity covers that repair, and the SWE-bench card records what the verified set supports.

The course's project grading rubric doubles as a benchmark-quality checklist (Song, 2025, slides 87–88). Read in this handbook's coordinates:

Read as a checklist, the rubric covers the claim, the instrument, and the tasks. It asks little about the regime and the lifecycle — who will optimize against the benchmark, how much hidden material each submission exposes, and when its scores stop discriminating. Ask those questions separately: anchor burn and saturation are two ways benchmark scores lose their meaning over time.

The evidence view changes the verdict#

What a grader may observe is part of the instrument: the same judge shown traces, hidden state, or provenance is a different instrument (see Anchors and evidence views). Research agents supply a sharp case. ResearchArena runs frontier coding agents through the full research loop and asks whether their papers are any good. Under manuscript-only automated review, the best agent's papers match the average human ICLR submission score; artifact-aware review and human meta-review show that picture to be overstated (Zhang et al., 2026, via Chen, Wang & Qu, 2026, §6.3). The papers did not change; what the reviewers could see, and who reviewed, did.

Agent benchmarks differ in what they let the grader see. GAIA grades only the final answer, WebArena and τ-bench grade final state, and SWE-bench observes test outcomes but not the agent's trajectory (see each card). None of them can see a forbidden action that leaves no trace in what it observes, so a claim about how an agent behaves along the way needs a benchmark whose evidence view includes the path.

In practice#

Fill in an evaluator card before you cite a benchmark's leaderboard.

1. Write the sentence you want to say. State the claim, its scope, and the decision it serves.
2. Locate the benchmark on the four axes. Note the primary risk its location predicts.
3. Recover the instrument. From the paper or card, write down the specification, reference anchor, evidence view, mechanism, signal, resolution, and aggregation.
4. Analyze its quality on a sample. Check tasks for correctness, clarity, and difficulty, check the grader against hand-verified verdicts, and note biases and limitations.
5. Recover the estimand. Find n, the trials per task, and whether "pass@k" means at least one or all k; compute intervals where the paper gives none.
6. Recover the regime. Note what is public and what is hidden, who tunes against the benchmark, and whether contamination was checked.
7. Write the supported sentence. Phrase it as "supports X for Y; does not support Z", and compare it with the sentence from step 1.

The artifact is an evaluator card in the form this site uses for its benchmarks: a claim headline, the instrument's coordinates, the regime, and claim-scoped evidence notes.

Common failure modes#

• Scope inflation — an application-scoped score is quoted as general capability → write the scope into the claim and stop at its edge. See Population, environment, harness, and validity domain.
• Pooled outcomes — one success rate merges different routes or criteria, such as CyberGym's two success conditions or τ²-bench's assertion categories → report the components, each with n and an interval. See one-number reporting.
• Weak comparators read as verification — keyword overlap, token F1, or embedding similarity treated as correctness → validate the comparator against hand-graded samples and say what it can miss. See grader gaming.
• Public tasks, unchecked exposure — results on public tasks and answers quoted with no contamination check → prefer hidden or post-cutoff splits, and report the check. See contamination.

Check yourself#

Analyze. Choose one benchmark card on this site. Place the benchmark on the three scopes and the four axes, and name the risk its location predicts you should check first.

Hint

Match the benchmark's location to the primary risks listed with the four design axes: an interactive, verifiable benchmark points you to simulator fidelity, hidden side effects, and path-versus-outcome confusion.

Evaluate. A team cites a 15-point lead on an application benchmark graded by keyword overlap as evidence that its agent is "better at legal research". What does the evidence support?

Hint

Check four things: the scope (one application, not research in general), the comparator (weak, so validate it by hand), n and the interval on the difference, and the evidence view (the answer only, not the research process).

Create. Plan the quality analysis you would run before adopting τ-bench as a release gate for your own customer-service agent: what you would sample, how you would check the reference end states and the simulated user, and which estimand you would report.

Hint

Confirm that each sampled task has a unique acceptable end state, audit the simulator against your real conversation logs, compare the policy documents with yours, and report pass^k with an interval from task-level resampling.

Further reading#

• Task validity and outcome validity — the outcome-validity checks and the SWE-bench Verified repair.
• Estimands, pass rates, and intervals — sample sizes, intervals, and pass@k versus pass^k.
• Anchors and evidence views — why what the grader can see is part of the instrument.
• Benchmarks — evaluator cards for the benchmarks named in this chapter.
• Song (2025), "Agent evaluation & project overview," UC Berkeley lecture slides — the lecture behind the scope taxonomy, the six case studies, and the quality-analysis workflow.