Agents · Evaluator card
WebArena
2023Shuyan Zhou et al.Paper (opens in a new tab)
A realistic web environment benchmark where agents must complete tasks on self-hosted replicas of real websites (Reddit, GitLab, shopping sites, maps, Wikipedia). It evaluates autonomous web navigation and task completion.
Evidence notes pending review. The verdicts on this card are drafts derived from the benchmark's documentation and published sources; no reviewer has checked them against those sources yet.
Headline claim
WebArena success rates support comparing autonomous web agents on end-to-end task completion in self-hosted replicas of real web applications; they do not support claims about the live web or about partial progress.
At a glance
- Mechanism
- Hybrid portfolioPrior: mixed, one per channel
- Item format
- Interactive
- Size
- 812 tasks across 5 web environments
- Typical uses
- Measure
- Saturation
- Active
- Contamination
- Medium risk
- Year
- 2023
1Claim
What property are we trying to infer, over what world, and for what decision?
- Construct
- Autonomous task completion through a browser in realistic web applications
- Population
- 812 natural-language tasks on self-hosted e-commerce, content-management, forum, code-hosting, and map applications
- Measures
- Autonomous web navigation and interaction in realistic website environments
- Ability to complete multi-step web tasks like posting, searching, configuring settings, and managing content
- Understanding of web UI elements, forms, navigation patterns, and web application workflows
- End-to-end task success rate on fully functional, self-hosted web applications
- Does not measure
- Performance on the open, live web — WebArena uses controlled replicas of fixed websites
- Ability to handle dynamic, adversarial, or rapidly changing web content
- API or programmatic interaction skills — tasks are designed for browser-based UI interaction
- Partial task completion or quality of intermediate steps, as evaluation is binary success/failure
2Instrument
How is the judgment actually produced?
- Mechanism
Hybrid portfolio
Tasks that change a site are checked by querying the applications' final state (execution); information-seeking tasks compare the answer with reference strings by exact or must-include matching (deterministic), with an LLM-based fuzzy match for some answers (learned judge).
Hierarchy prior: mixed. Each channel carries the prior of its own mechanism. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.
- Item format
- Interactive
- Specification
- A task succeeds when the final environment state, or the returned answer for information-seeking tasks, passes the task's programmatic check.
- Anchor
- Task-specific reference checks written by the authors: expected database or page state, URLs, or reference answers
- Evidence view
- The final state of the web applications and the agent's final answer; intermediate actions are not graded
- Signal
- Binary success/failure per task
- Resolution
- Final state or answer per episode
- Aggregation
- Task success rate over the 812 tasks
- Metrics
- task success rate
- Size
- 812 tasks across 5 web environments
- Methodology
- Deploys self-hosted replicas of five real websites (a Reddit forum, GitLab instance, e-commerce site, map service, and content management system). Agents receive a natural language task instruction and must interact with the websites through browser actions (click, type, navigate) to complete the task. Evaluation uses programmatic checks on the final state of the web environment (e.g., checking if a post was created, a setting was changed) to determine binary success or failure across 812 tasks.
3Regime
What causal role does the signal play, and what happens when a system adapts to it?
- Typical uses
- Measure · supports inference and decision
- Exposure
- Task intents and evaluation checks are public and the website snapshots are fixed, so agent scaffolds can be tuned to the known task set over repeated runs; there is no hidden split.
4ValidationDraft
Why should we believe that the signal means what we claim it means — before and after the system adapts to it?
WebArena's evidence supports reproducible claims about web-agent task completion in realistic settings, because it deploys working web applications and checks the resulting state programmatically. Self-hosted replicas cannot capture the full complexity of the live web — a deliberate trade for reproducibility — and binary success misses partial progress.
Evidence notes2 partially supported · 1 untested
- Partially supported
Tasks are achievable, so agent failures reflect the agent
At release, human annotators completed about 78% of tasks against about 14% for the best GPT-4 agent; the human failures leave room for ambiguous or broken tasks.
Source: Zhou et al., 2023
- Partially supported
Success checks capture task completion
Programmatic state checks ground the verdict in consequences, but binary success misses near-misses and partial progress, and results depend heavily on how the observation and action spaces are implemented.
- Untested
Results transfer to the live web
The replicas have no CAPTCHAs, rate limits, real users, or novel layouts.
5Lifecycle
Does the validity survive over time, and are we still evaluating the right thing?
- Saturation
- Active
Whether top systems still differ, or crowd the ceiling.
- Contamination
- Medium risk
How likely test items, or close variants, have leaked into training data.
- Successors
- None listed
- Last reviewed
- Not yet reviewed · card updated October 12, 2026
Limitations
- Self-hosted environment differs from the real web in important ways — no CAPTCHAs, no rate limits, no real users
- Binary success/failure evaluation does not capture near-misses or partial progress
- Agent performance is heavily dependent on the observation and action space implementation
- Environment setup complexity creates reproducibility challenges
- Fixed website snapshots mean agents cannot encounter novel web patterns or layouts
When to use
- Evaluating web agents in a controlled, reproducible environment with realistic web interfaces
- Comparing different web agent architectures on standardized navigation and interaction tasks
- Assessing readiness of an agent system for autonomous web-based task completion
When not to use
- When you need to evaluate performance on the live, open web with all its complexity and unpredictability
- When partial credit or reasoning quality matters more than binary task completion
Related benchmarks
- GAIAAgents · 2023General AI Assistants benchmark evaluates AI systems on real-world tasks requiring multi-step reasoning, web browsing, tool use, and file manipulation. Questions have unambiguous, verifiable answers but require complex workflows to solve.
- AssistantBenchAgents · 2024A benchmark of challenging, realistic web tasks that require an assistant to navigate the open web to find precise, verifiable answers. Tasks span planning, fact-checking, data collection, and complex information retrieval.
- τ-benchAgents · 2024A benchmark for evaluating language model agents as customer service representatives in simulated retail and airline domains. Agents must handle complex customer requests by navigating policies, databases, and multi-turn conversations.