Agents · Evaluator card
GAIA
2023Gregoire Mialon et al. (Meta)Paper (opens in a new tab)
General AI Assistants benchmark evaluates AI systems on real-world tasks requiring multi-step reasoning, web browsing, tool use, and file manipulation. Questions have unambiguous, verifiable answers but require complex workflows to solve.
Evidence notes pending review. The verdicts on this card are drafts derived from the benchmark's documentation and published sources; no reviewer has checked them against those sources yet.
Headline claim
GAIA accuracy supports comparing complete assistant systems (model, tools, and scaffold) on short-answer questions that require web browsing, file handling, and multi-step reasoning; it does not isolate model capability or grade the process.
At a glance
- Mechanism
- Deterministic checkerPrior: execution feedback, rung 2 of 4
- Item format
- Short answer
- Size
- 466 questions across 3 difficulty levels
- Typical uses
- Measure
- Saturation
- Active
- Contamination
- Medium risk
- Year
- 2023
1Claim
What property are we trying to infer, over what world, and for what decision?
- Construct
- General assistant capability: multi-step reasoning combined with tool use to reach a verifiable answer
- Population
- Human-written questions at three difficulty levels, conceptually simple for people but requiring browsing, document and spreadsheet handling, and calculation; agent systems with tools
- Measures
- End-to-end task completion requiring multi-step reasoning combined with tool use
- Ability to decompose complex questions into actionable sub-tasks across web search, file processing, and calculation
- Real-world assistant capability on questions that are easy for humans but hard for AI systems
- Robustness of agent workflows across three calibrated difficulty levels
- Integration of multiple capabilities — reasoning, retrieval, code execution — in a single problem-solving loop
- Does not measure
- Performance on tasks with subjective or ambiguous answers, since GAIA requires exact verifiable outputs
- Creative problem-solving or open-ended generation ability
- Ability to handle adversarial inputs, deception, or tasks requiring human judgment
- Long-horizon planning over hours or days — GAIA tasks are solvable in minutes to tens of minutes
2Instrument
How is the judgment actually produced?
- Mechanism
Deterministic checker
Quasi-exact match after type-aware normalization of numbers, strings, and lists.
Hierarchy prior: execution feedback, rung 2 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.
- Item format
- Short answer
- Specification
- A question is answered correctly when the system's final answer, after normalization, matches the single reference answer (a number, a short string, or a list).
- Anchor
- One human-written reference answer per question; test-split answers are withheld
- Evidence view
- The final answer only; browsing, tool calls, and reasoning traces are not inspected
- Signal
- Binary correct/incorrect per question
- Resolution
- Final answer per question
- Aggregation
- Accuracy overall and per difficulty level
- Metrics
- accuracy
- Size
- 466 questions across 3 difficulty levels
- Methodology
- Contains 466 questions at three difficulty levels, each with a single verifiable answer (a number, name, date, or short phrase). Questions are designed to require multiple steps including web browsing, document analysis, spreadsheet manipulation, and multi-hop reasoning. Some questions include attached files. Evaluation is exact-match accuracy on the final answer after normalization.
3Regime
What causal role does the signal play, and what happens when a system adapts to it?
- Typical uses
- Measure · supports inference and decision
- Exposure
- Questions are public. Validation-split answers are public and widely used for development, so validation scores are exposed to tuning; test-split answers are withheld and scored through a leaderboard, where each submission reveals only aggregate scores. Web-dependent answers can drift as online content changes.
4ValidationDraft
Why should we believe that the signal means what we claim it means — before and after the system adapts to it?
GAIA's evidence supports system-level claims about AI assistants: verifiable answers combined with genuinely multi-step workflows give a meaningful signal, and three difficulty levels spread current systems. It measures model plus tools plus scaffolding rather than isolating the model — arguably the right target for agents — and it grades only the final answer, not the process.
Evidence notes1 supported · 2 partially supported · 1 not supported
- Supported
Tasks are easy for people and hard for AI systems
At release, human respondents answered 92% of questions correctly versus 15% for GPT-4 with plugins.
Source: Mialon et al., 2023
- Partially supported
Answers are unambiguous and stay verifiable
Single short answers make scoring deterministic, but web-dependent answers can go stale as online content changes.
- Not supported
Scores isolate model capability
Agent infrastructure and tool access vary widely between evaluations, so scores describe systems.
- Partially supported
Separates systems at the top of the range
Level 3 has few questions, so comparisons at the top end carry high variance.
5Lifecycle
Does the validity survive over time, and are we still evaluating the right thing?
- Saturation
- Active
Whether top systems still differ, or crowd the ceiling.
- Contamination
- Medium risk
How likely test items, or close variants, have leaked into training data.
- Successors
- None listed
- Last reviewed
- Not yet reviewed · card updated October 12, 2026
Limitations
- Exact-match evaluation on short factual answers does not capture the quality of the reasoning process
- Questions are static, allowing potential overfitting of agent scaffolding to known problem patterns
- Web-dependent questions may become stale as online content changes over time
- Agent infrastructure and tool access vary widely between evaluations, making pure model comparison difficult
- Level 3 (hardest) questions are few in number, increasing variance at the top end
When to use
- Evaluating AI agent systems on multi-step real-world tasks requiring tool use and reasoning
- Comparing different agent architectures and scaffolding approaches on a standardized task set
- Assessing whether an assistant system is ready for complex real-world question answering
When not to use
- Evaluating raw model capability without tools — GAIA is designed for agent systems
- Assessing open-ended generation, creative writing, or tasks without clear right answers
Related benchmarks
- AssistantBenchAgents · 2024A benchmark of challenging, realistic web tasks that require an assistant to navigate the open web to find precise, verifiable answers. Tasks span planning, fact-checking, data collection, and complex information retrieval.
- WebArenaAgents · 2023A realistic web environment benchmark where agents must complete tasks on self-hosted replicas of real websites (Reddit, GitLab, shopping sites, maps, Wikipedia). It evaluates autonomous web navigation and task completion.
- ScienceAgentBenchAgents · 2024A benchmark for evaluating AI agents on data-driven scientific discovery tasks drawn from real published research papers. Agents must write code to process datasets, run experiments, and produce results that match the original paper's findings.
- τ-benchAgents · 2024A benchmark for evaluating language model agents as customer service representatives in simulated retail and airline domains. Agents must handle complex customer requests by navigating policies, databases, and multi-turn conversations.