Agents · Evaluator card

τ-bench

2024Shunyu Yao et al.Paper (opens in a new tab)

A benchmark for evaluating language model agents as customer service representatives in simulated retail and airline domains. Agents must handle complex customer requests by navigating policies, databases, and multi-turn conversations.

View as Markdown

Evidence notes pending review. The verdicts on this card are drafts derived from the benchmark's documentation and published sources; no reviewer has checked them against those sources yet.

Headline claim

τ-bench pass rates support comparing tool-using agents on policy-compliant, multi-turn customer-service tasks in simulated retail and airline domains, including consistency across repeated trials; they do not support claims about other domains or real customers.

At a glance

Mechanism: Execution / effect checkerPrior: execution feedback, rung 2 of 4
Item format: Interactive
Size: ~200 tasks across retail and airline domains
Typical uses: Measure
Saturation: Active
Contamination: Medium risk
Year: 2024

1Claim

What property are we trying to infer, over what world, and for what decision?

Construct: Reliable, policy-compliant task completion through tool calls and conversation
Population: Simulated retail and airline customer requests, with fixed policy documents and databases and a language model playing the customer
Measures: Ability to act as an autonomous customer service agent handling realistic user requests
Policy adherence — correctly following complex business rules while serving customers
Tool use competence for querying databases, modifying orders, and executing business operations
Multi-turn conversation management with appropriate information gathering and action execution
Robustness across diverse customer scenarios including edge cases and policy-boundary situations
Does not measure: General-purpose agent capabilities outside the customer service domain
Performance on tasks requiring creative problem-solving or open-ended reasoning
Ability to handle truly adversarial or abusive customer interactions
Latency, cost efficiency, or operational feasibility in production settings

2Instrument

How is the judgment actually produced?

Mechanism

Execution / effect checker

The agent's tool calls change state in a simulated backend while a language model plays the customer; at the end, the database is compared with the goal state.

Hierarchy prior: execution feedback, rung 2 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.

Item format

Interactive

Specification

An episode succeeds when the database at the end of the conversation matches the goal state for the task.

Anchor

Annotated goal database state, produced by executing the task's ground-truth action sequence

Evidence view

The final database state, compared with the annotated goal state

Signal

Binary success per episode

Resolution

Final database state per episode

Aggregation

Average pass rate over tasks and trials; pass^k is the probability that all k independent trials of a task succeed (pass@k, by contrast, needs at least one of k)

Metrics

pass rate, pass^k

Size

~200 tasks across retail and airline domains

Methodology

Provides two simulated environments — a retail store and an airline — each with a database of customers, products/flights, and detailed policy documents. A user simulator generates customer requests that range from simple (checking order status) to complex (multi-item returns with policy exceptions). Agents interact through tool calls (database queries, order modifications) and natural language. Evaluation automatically compares the final database state with the goal state produced by the annotated action sequence.

3Regime

What causal role does the signal play, and what happens when a system adapts to it?

Typical uses: Measure · supports inference and decision
Exposure: Task instructions, policies, and annotated goal actions are public, so agents and prompts can be tuned to the known task set. It is a headline agentic tool-use number in model releases. The simulated user is itself a language model, so environment behavior changes if the simulator model changes.

4ValidationDraft

Why should we believe that the signal means what we claim it means — before and after the system adapts to it?

τ-bench's evidence supports claims about practical business-automation capability: realistic policies, database operations, and multi-turn conversation make for a demanding test, and end-state checking grounds the verdict. Its domain specificity is both its strength — high relevance for customer-service agents — and its limit, since generalization beyond the two domains is unclear.

Evidence notes2 partially supported · 1 not supported · 1 untested

Partially supported
The end-state check captures task completion
Comparing the final database with the goal state grounds the verdict in consequences, but binary success misses partial resolution and the quality of the customer interaction.
Not supported
Single-trial pass rates indicate deployment reliability
The authors introduced pass^k and found success on all k repeated trials falling well below single-trial success, so one run overstates reliability.
Source: Yao et al., 2024
Partially supported
Simulated users stand in for real customers
A language-model user cannot replicate the unpredictability of real customers, and simulator mistakes add noise to the agent's score.
Untested
Results generalize beyond retail and airline
Only two domains, with fixed, self-consistent policies and standardized tool interfaces.

5Lifecycle

Does the validity survive over time, and are we still evaluating the right thing?

Saturation: Active
Whether top systems still differ, or crowd the ceiling.
Contamination: Medium risk
How likely test items, or close variants, have leaked into training data.
Successors: None listed
Last reviewed: Not yet reviewed · card updated October 12, 2026

Limitations

Limited to two domains (retail and airline), constraining generalization claims
Simulated users cannot fully replicate the unpredictability of real customer interactions
Policy documents are fixed and self-consistent, unlike the messy reality of real business rules
Binary success evaluation does not capture partial resolution or quality of customer interaction
Agent tool interfaces are standardized, which may not reflect real-world API diversity

When to use

Evaluating AI agents for customer service automation or similar structured workflow tasks
Assessing policy adherence and tool-use reliability in multi-turn conversational settings
Comparing agent architectures on tasks requiring both reasoning and action execution

When not to use

Evaluating general-purpose agent capability across diverse domains
When you need to assess performance with real users rather than simulated interactions