# GAIA

> General AI Assistants benchmark evaluates AI systems on real-world tasks requiring multi-step reasoning, web browsing, tool use, and file manipulation. Questions have unambiguous, verifiable answers but require complex workflows to solve.

## Claim

- **headline**: GAIA accuracy supports comparing complete assistant systems (model, tools, and scaffold) on short-answer questions that require web browsing, file handling, and multi-step reasoning; it does not isolate model capability or grade the process.
- **construct**: General assistant capability: multi-step reasoning combined with tool use to reach a verifiable answer
- **population**: Human-written questions at three difficulty levels, conceptually simple for people but requiring browsing, document and spreadsheet handling, and calculation; agent systems with tools
- **measures**: End-to-end task completion requiring multi-step reasoning combined with tool use; Ability to decompose complex questions into actionable sub-tasks across web search, file processing, and calculation; Real-world assistant capability on questions that are easy for humans but hard for AI systems; Robustness of agent workflows across three calibrated difficulty levels; Integration of multiple capabilities — reasoning, retrieval, code execution — in a single problem-solving loop
- **doesNotMeasure**: Performance on tasks with subjective or ambiguous answers, since GAIA requires exact verifiable outputs; Creative problem-solving or open-ended generation ability; Ability to handle adversarial inputs, deception, or tasks requiring human judgment; Long-horizon planning over hours or days — GAIA tasks are solvable in minutes to tens of minutes

## Instrument

- **itemFormat**: short-answer
- **specification**: A question is answered correctly when the system's final answer, after normalization, matches the single reference answer (a number, a short string, or a list).
- **anchor**: One human-written reference answer per question; test-split answers are withheld
- **evidenceView**: The final answer only; browsing, tool calls, and reasoning traces are not inspected
- **mechanism**: deterministic
- **mechanismDetail**: Quasi-exact match after type-aware normalization of numbers, strings, and lists.
- **signal**: Binary correct/incorrect per question
- **resolution**: Final answer per question
- **aggregation**: Accuracy overall and per difficulty level
- **metrics**: accuracy
- **size**: 466 questions across 3 difficulty levels
- **methodology**: Contains 466 questions at three difficulty levels, each with a single verifiable answer (a number, name, date, or short phrase). Questions are designed to require multiple steps including web browsing, document analysis, spreadsheet manipulation, and multi-hop reasoning. Some questions include attached files. Evaluation is exact-match accuracy on the final answer after normalization.

## Regime

- **typicalUses**: measure
- **exposure**: Questions are public. Validation-split answers are public and widely used for development, so validation scores are exposed to tuning; test-split answers are withheld and scored through a leaderboard, where each submission reveals only aggregate scores. Web-dependent answers can drift as online content changes.

## Validation

- **summary**: GAIA's evidence supports system-level claims about AI assistants: verifiable answers combined with genuinely multi-step workflows give a meaningful signal, and three difficulty levels spread current systems. It measures model plus tools plus scaffolding rather than isolating the model — arguably the right target for agents — and it grades only the final answer, not the process.
- **evidence**: Tasks are easy for people and hard for AI systems; Answers are unambiguous and stay verifiable; Scores isolate model capability; Separates systems at the top of the range
- **status**: draft

## Lifecycle

- **saturation**: active
- **contamination**: medium
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/gaia