# τ-bench

> A benchmark for evaluating language model agents as customer service representatives in simulated retail and airline domains. Agents must handle complex customer requests by navigating policies, databases, and multi-turn conversations.

## Claim

- **headline**: τ-bench pass rates support comparing tool-using agents on policy-compliant, multi-turn customer-service tasks in simulated retail and airline domains, including consistency across repeated trials; they do not support claims about other domains or real customers.
- **construct**: Reliable, policy-compliant task completion through tool calls and conversation
- **population**: Simulated retail and airline customer requests, with fixed policy documents and databases and a language model playing the customer
- **measures**: Ability to act as an autonomous customer service agent handling realistic user requests; Policy adherence — correctly following complex business rules while serving customers; Tool use competence for querying databases, modifying orders, and executing business operations; Multi-turn conversation management with appropriate information gathering and action execution; Robustness across diverse customer scenarios including edge cases and policy-boundary situations
- **doesNotMeasure**: General-purpose agent capabilities outside the customer service domain; Performance on tasks requiring creative problem-solving or open-ended reasoning; Ability to handle truly adversarial or abusive customer interactions; Latency, cost efficiency, or operational feasibility in production settings

## Instrument

- **itemFormat**: interactive
- **specification**: An episode succeeds when the database at the end of the conversation matches the goal state for the task.
- **anchor**: Annotated goal database state, produced by executing the task's ground-truth action sequence
- **evidenceView**: The final database state, compared with the annotated goal state
- **mechanism**: execution
- **mechanismDetail**: The agent's tool calls change state in a simulated backend while a language model plays the customer; at the end, the database is compared with the goal state.
- **signal**: Binary success per episode
- **resolution**: Final database state per episode
- **aggregation**: Average pass rate over tasks and trials; pass^k is the probability that all k independent trials of a task succeed (pass@k, by contrast, needs at least one of k)
- **metrics**: pass rate; pass^k
- **size**: ~200 tasks across retail and airline domains
- **methodology**: Provides two simulated environments — a retail store and an airline — each with a database of customers, products/flights, and detailed policy documents. A user simulator generates customer requests that range from simple (checking order status) to complex (multi-item returns with policy exceptions). Agents interact through tool calls (database queries, order modifications) and natural language. Evaluation automatically compares the final database state with the goal state produced by the annotated action sequence.

## Regime

- **typicalUses**: measure
- **exposure**: Task instructions, policies, and annotated goal actions are public, so agents and prompts can be tuned to the known task set. It is a headline agentic tool-use number in model releases. The simulated user is itself a language model, so environment behavior changes if the simulator model changes.

## Validation

- **summary**: τ-bench's evidence supports claims about practical business-automation capability: realistic policies, database operations, and multi-turn conversation make for a demanding test, and end-state checking grounds the verdict. Its domain specificity is both its strength — high relevance for customer-service agents — and its limit, since generalization beyond the two domains is unclear.
- **evidence**: The end-state check captures task completion; Single-trial pass rates indicate deployment reliability; Simulated users stand in for real customers; Results generalize beyond retail and airline
- **status**: draft

## Lifecycle

- **saturation**: active
- **contamination**: medium
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/tau-bench