Chatbot Arena

CConstruct

Aggregate human preference in open-ended conversation

PPopulation

Prompts and votes from the platform's self-selected, mostly tech-savvy, English-speaking users, with an organic task mix weighted toward conversational requests

Measures

• Aggregate human preference between models in head-to-head blind comparisons
• Overall conversational quality as judged by a diverse, self-selected user population
• Relative model ranking on the mix of tasks that users organically bring to the platform
• Broad user satisfaction across creative writing, coding, analysis, and general chat

Does not measure

• Any specific capability — the Elo score is a black box that collapses all quality dimensions into one number
• Factual accuracy or truthfulness, which users often cannot evaluate during brief interactions
• Performance on hard, well-defined tasks where correctness can be objectively verified
• Consistency, reliability, or safety — users judge single conversations, not deployment-level behavior
• Performance for specific use cases, as the task distribution reflects the crowd's interests, not any particular application

MMechanism

Human rater or panel

Crowdsourced pairwise votes from anonymous users, with model identities hidden until after the vote.

Outside the ladder: human judgment is the external frontier and trust root, with measurable rater error. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.

Item format

Pairwise

ΨSpecification

In each battle, the user's vote between two anonymous responses to the same prompt (or a tie) is the outcome.

AAnchor

No reference answers: the voters' preferences are the trust root, and ratings are relative to the other models in the pool

VEvidence view

The prompt and the two responses as displayed; voters see no references, sources, or tool traces and cannot check correctness beyond what is shown

ZSignal

Pairwise preference: A, B, tie, or both bad

RResolution

One conversation (one or a few turns) per battle

ΓAggregation

Bradley–Terry/Elo: a Bradley–Terry model fitted to all votes, reported on an Elo-like scale with bootstrap confidence intervals; category and style-controlled leaderboards re-fit on subsets or with style covariates

Metrics

Elo rating, win rate

Size

1M+ pairwise human votes (continuously growing)

Methodology

Users visit the Chatbot Arena website, enter a prompt, and receive responses from two randomly selected anonymous models. They vote for their preferred response (or declare a tie). Ratings are computed with the Bradley–Terry model from accumulated pairwise votes and reported on an Elo-like scale. Category-specific Elo ratings (coding, harder prompts, etc.) are also computed from user-annotated subsets. Over one million votes have been collected as of 2024.

Typical uses

• Measure · supports inference and decision
• Optimize · shapes the candidate or policy distribution

Exposure

A live public leaderboard under heavy pressure from model providers. Providers can test unreleased variants and publish the best, and prompts flow back to the providers whose models serve them, so rankings are exposed to selection and to overfitting on the arena's prompt distribution. Users' preference for verbose, polished responses rewards style tuning.

Notes

Renamed LMArena in 2025. Style-controlled ratings separate some stylistic effects, such as length and formatting, from substance; read them alongside the default ratings.

Chatbot Arena's evidence supports a claim no automated benchmark captures: which models real users prefer in blind, head-to-head comparison on their own prompts. The preference signal is real but shallow — first impressions on self-selected tasks — so a higher rating does not show that a model is more accurate, safer, or better at a specific task; treat it as one input, not the final word on model quality.

Evidence notes1 supported · 1 partially supported · 1 contested · 1 not supported

1. Supported

   Ratings reflect what the platform's users prefer on the prompts they bring

   Blind pairwise votes at very large scale, with confidence intervals on the fitted ratings, estimate that population's preferences.

   Source: Chiang et al., 2024

2. Not supported

   A higher rating means a more accurate or more capable model

   Users tend to prefer verbose, confident, polished responses regardless of accuracy and often cannot check correctness in a brief interaction; style control was added to separate part of this effect.

3. Contested

   Rankings are robust to provider strategy

   An analysis reported that undisclosed private testing of many variants, selective disclosure, and unequal access to arena data inflate some providers' ratings; the platform disputed parts of the analysis.

   Source: Singh et al., 2025 (The Leaderboard Illusion)

4. Partially supported

   Small rating gaps between frontier models are meaningful

   Ratings compress at the top; gaps within overlapping confidence intervals do not support an ordering.

Saturation

Active

Whether top systems still differ, or crowd the ceiling.

Contamination

Medium risk

How likely test items, or close variants, have leaked into training data.

Successors

None listed

Last reviewed

Not yet reviewed · card updated October 12, 2026

• AlpacaEvalPreference · 2023An automated evaluation framework that uses a strong LLM judge (GPT-4 Turbo) to compare model outputs against a reference model on 805 instructions. AlpacaEval 2.0 introduced length-controlled win rates to mitigate the bias toward verbose responses.
• MT-BenchPreference · 2023A multi-turn conversation benchmark with 80 carefully crafted two-turn questions across 8 categories. An LLM judge scores responses on a 1-10 scale, evaluating both first-turn quality and the ability to handle follow-up instructions.
• WildBenchPreference · 2024A benchmark of 1,024 challenging, real-world user queries mined from actual chatbot interaction logs. It uses a detailed LLM judge rubric with task-specific checklists to evaluate response quality more rigorously than prior LLM-as-judge approaches.