# Chatbot Arena

> A crowdsourced platform where users have blind conversations with two anonymous models and vote for the one they prefer. Ratings on an Elo-like scale are fitted to a large and growing pool of pairwise human preference votes.

## Claim

- **headline**: Chatbot Arena ratings support claims about which models the platform's self-selected users prefer, in blind pairwise comparison, on the prompts they choose to bring; they do not support claims about accuracy, safety, or any specific capability.
- **construct**: Aggregate human preference in open-ended conversation
- **population**: Prompts and votes from the platform's self-selected, mostly tech-savvy, English-speaking users, with an organic task mix weighted toward conversational requests
- **measures**: Aggregate human preference between models in head-to-head blind comparisons; Overall conversational quality as judged by a diverse, self-selected user population; Relative model ranking on the mix of tasks that users organically bring to the platform; Broad user satisfaction across creative writing, coding, analysis, and general chat
- **doesNotMeasure**: Any specific capability — the Elo score is a black box that collapses all quality dimensions into one number; Factual accuracy or truthfulness, which users often cannot evaluate during brief interactions; Performance on hard, well-defined tasks where correctness can be objectively verified; Consistency, reliability, or safety — users judge single conversations, not deployment-level behavior; Performance for specific use cases, as the task distribution reflects the crowd's interests, not any particular application

## Instrument

- **itemFormat**: pairwise
- **specification**: In each battle, the user's vote between two anonymous responses to the same prompt (or a tie) is the outcome.
- **anchor**: No reference answers: the voters' preferences are the trust root, and ratings are relative to the other models in the pool
- **evidenceView**: The prompt and the two responses as displayed; voters see no references, sources, or tool traces and cannot check correctness beyond what is shown
- **mechanism**: human
- **mechanismDetail**: Crowdsourced pairwise votes from anonymous users, with model identities hidden until after the vote.
- **signal**: Pairwise preference: A, B, tie, or both bad
- **resolution**: One conversation (one or a few turns) per battle
- **aggregation**: Bradley–Terry/Elo: a Bradley–Terry model fitted to all votes, reported on an Elo-like scale with bootstrap confidence intervals; category and style-controlled leaderboards re-fit on subsets or with style covariates
- **metrics**: Elo rating; win rate
- **size**: 1M+ pairwise human votes (continuously growing)
- **methodology**: Users visit the Chatbot Arena website, enter a prompt, and receive responses from two randomly selected anonymous models. They vote for their preferred response (or declare a tie). Ratings are computed with the Bradley–Terry model from accumulated pairwise votes and reported on an Elo-like scale. Category-specific Elo ratings (coding, harder prompts, etc.) are also computed from user-annotated subsets. Over one million votes have been collected as of 2024.

## Regime

- **typicalUses**: measure; optimize
- **exposure**: A live public leaderboard under heavy pressure from model providers. Providers can test unreleased variants and publish the best, and prompts flow back to the providers whose models serve them, so rankings are exposed to selection and to overfitting on the arena's prompt distribution. Users' preference for verbose, polished responses rewards style tuning.
- **notes**: Renamed LMArena in 2025. Style-controlled ratings separate some stylistic effects, such as length and formatting, from substance; read them alongside the default ratings.

## Validation

- **summary**: Chatbot Arena's evidence supports a claim no automated benchmark captures: which models real users prefer in blind, head-to-head comparison on their own prompts. The preference signal is real but shallow — first impressions on self-selected tasks — so a higher rating does not show that a model is more accurate, safer, or better at a specific task; treat it as one input, not the final word on model quality.
- **evidence**: Ratings reflect what the platform's users prefer on the prompts they bring; A higher rating means a more accurate or more capable model; Rankings are robust to provider strategy; Small rating gaps between frontier models are meaningful
- **status**: draft

## Lifecycle

- **saturation**: active
- **contamination**: medium
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/chatbot-arena