Skip to content
Benchmaker

Agents · Evaluator card

AssistantBench

2024Ori Yoran et al.Paper (opens in a new tab)

A benchmark of challenging, realistic web tasks that require an assistant to navigate the open web to find precise, verifiable answers. Tasks span planning, fact-checking, data collection, and complex information retrieval.

View as Markdown

Evidence notes pending review. The verdicts on this card are drafts derived from the benchmark's documentation and published sources; no reviewer has checked them against those sources yet.

Headline claim

AssistantBench scores support comparing web agents on realistic, time-consuming open-web tasks with verifiable answers, as a dated snapshot; the small task set and a changing web limit precision and reproducibility.

At a glance

Mechanism
Deterministic checkerPrior: execution feedback, rung 2 of 4
Item format
Short answer
Size
214 tasks
Typical uses
Measure
Saturation
Active
Contamination
Unknown
Year
2024

1Claim

What property are we trying to infer, over what world, and for what decision?

Construct
Completing realistic multi-site web research and planning tasks to a precise answer
Population
214 realistic assistant tasks (planning, comparison, fact-checking, data collection) that require navigating multiple live websites
Measures
  • Ability to complete complex web-based information retrieval and reasoning tasks
  • End-to-end performance on tasks requiring multi-site navigation and information synthesis
  • Accuracy of extracted factual information verified against human-annotated gold answers
  • Performance on tasks that mirror realistic assistant use cases like trip planning and price comparison
Does not measure
  • Offline reasoning or knowledge recall — tasks explicitly require live web interaction
  • Subjective quality of responses, as evaluation is based on factual accuracy against gold answers
  • Task completion efficiency, speed, or resource consumption
  • Generative tasks like writing, summarization, or creative content production

2Instrument

How is the judgment actually produced?

Mechanism

Deterministic checker

Rule-based comparison with the gold answer plus a normalized answer-similarity score for partial matches: a weak comparator for answers phrased differently.

Hierarchy prior: execution feedback, rung 2 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.

Item format
Short answer
Specification
A task's final answer is compared with the human-verified gold answer and scored by accuracy and by a normalized similarity that gives partial credit.
Anchor
Human-verified gold answers, fixed at collection time
Evidence view
The final answer only; navigation traces are not graded
Signal
Correctness and a similarity score per task
Resolution
Final answer per task
Aggregation
Mean accuracy and mean answer similarity over tasks
Metrics
accuracy, answer similarity
Size
214 tasks
Methodology
Contains 214 tasks sourced from realistic web assistant scenarios. Each task requires navigating multiple websites to find a specific, verifiable answer. Tasks span categories like navigation, data extraction, comparative analysis, and multi-constraint planning. Evaluation compares model outputs against human-verified gold answers using accuracy and a normalized answer similarity score.

3Regime

What causal role does the signal play, and what happens when a system adapts to it?

Typical uses
  • Measure · supports inference and decision
Exposure
Public task set since 2024. Because tasks run on the live web, the environment changes underneath fixed gold answers, so scores are tied to the date they were collected.

4ValidationDraft

Why should we believe that the signal means what we claim it means — before and after the system adapts to it?

AssistantBench's evidence supports claims about how web agents handle the complex, multi-site tasks people want assistants to do; tasks are chosen to require genuine multi-step navigation rather than simple lookups. The price of realism is reproducibility: live web content and gold answers drift, so results are best read as dated snapshots.

Evidence notes1 partially supported · 2 not supported

  1. Not supported

    Scores are reproducible over time

    Websites change and gold answers can become outdated, so the same system can score differently on different dates.

  2. Not supported

    Small accuracy differences are reliable

    Binomial arithmetic: with 214 tasks, a 95% interval around 50% accuracy spans about ±7 points.

  3. Partially supported

    The task mix represents real assistant use

    Tasks reflect the authors' conception of realistic use cases, which may not match a given deployment.

5Lifecycle

Does the validity survive over time, and are we still evaluating the right thing?

Saturation
Active

Whether top systems still differ, or crowd the ceiling.

Contamination
Unknown

How likely test items, or close variants, have leaked into training data.

Successors
None listed
Last reviewed
Not yet reviewed · card updated October 12, 2026

Limitations

  • Small dataset size of 214 tasks limits statistical power for fine-grained comparisons
  • Web-dependent tasks are inherently non-reproducible as websites change over time
  • Gold answers may become outdated as real-world information evolves
  • Task distribution reflects the authors' conception of realistic use cases, which may not be representative

When to use

  • Evaluating web-based AI assistants on realistic, complex information tasks
  • Assessing whether an agent can handle real-world planning and research queries
  • Comparing web agent architectures on tasks that require multi-site navigation and synthesis

When not to use

  • Evaluating pure language model capabilities without web access
  • When reproducibility is critical, as web content changes make exact replication difficult

Related benchmarks