# AssistantBench

> A benchmark of challenging, realistic web tasks that require an assistant to navigate the open web to find precise, verifiable answers. Tasks span planning, fact-checking, data collection, and complex information retrieval.

## Claim

- **headline**: AssistantBench scores support comparing web agents on realistic, time-consuming open-web tasks with verifiable answers, as a dated snapshot; the small task set and a changing web limit precision and reproducibility.
- **construct**: Completing realistic multi-site web research and planning tasks to a precise answer
- **population**: 214 realistic assistant tasks (planning, comparison, fact-checking, data collection) that require navigating multiple live websites
- **measures**: Ability to complete complex web-based information retrieval and reasoning tasks; End-to-end performance on tasks requiring multi-site navigation and information synthesis; Accuracy of extracted factual information verified against human-annotated gold answers; Performance on tasks that mirror realistic assistant use cases like trip planning and price comparison
- **doesNotMeasure**: Offline reasoning or knowledge recall — tasks explicitly require live web interaction; Subjective quality of responses, as evaluation is based on factual accuracy against gold answers; Task completion efficiency, speed, or resource consumption; Generative tasks like writing, summarization, or creative content production

## Instrument

- **itemFormat**: short-answer
- **specification**: A task's final answer is compared with the human-verified gold answer and scored by accuracy and by a normalized similarity that gives partial credit.
- **anchor**: Human-verified gold answers, fixed at collection time
- **evidenceView**: The final answer only; navigation traces are not graded
- **mechanism**: deterministic
- **mechanismDetail**: Rule-based comparison with the gold answer plus a normalized answer-similarity score for partial matches: a weak comparator for answers phrased differently.
- **signal**: Correctness and a similarity score per task
- **resolution**: Final answer per task
- **aggregation**: Mean accuracy and mean answer similarity over tasks
- **metrics**: accuracy; answer similarity
- **size**: 214 tasks
- **methodology**: Contains 214 tasks sourced from realistic web assistant scenarios. Each task requires navigating multiple websites to find a specific, verifiable answer. Tasks span categories like navigation, data extraction, comparative analysis, and multi-constraint planning. Evaluation compares model outputs against human-verified gold answers using accuracy and a normalized answer similarity score.

## Regime

- **typicalUses**: measure
- **exposure**: Public task set since 2024. Because tasks run on the live web, the environment changes underneath fixed gold answers, so scores are tied to the date they were collected.

## Validation

- **summary**: AssistantBench's evidence supports claims about how web agents handle the complex, multi-site tasks people want assistants to do; tasks are chosen to require genuine multi-step navigation rather than simple lookups. The price of realism is reproducibility: live web content and gold answers drift, so results are best read as dated snapshots.
- **evidence**: Scores are reproducible over time; Small accuracy differences are reliable; The task mix represents real assistant use
- **status**: draft

## Lifecycle

- **saturation**: active
- **contamination**: unknown
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/assistantbench