Preference · Evaluator card
WildBench
2024Bill Yuchen Lin et al. (Allen AI)Paper (opens in a new tab)
A benchmark of 1,024 challenging, real-world user queries mined from actual chatbot interaction logs. It uses a detailed LLM judge rubric with task-specific checklists to evaluate response quality more rigorously than prior LLM-as-judge approaches.
Evidence notes pending review. The verdicts on this card are drafts derived from the benchmark's documentation and published sources; no reviewer has checked them against those sources yet.
Headline claim
WildBench scores support comparing chat models on 1,024 challenging real user queries, judged by GPT-4 Turbo against task-specific checklists, with rankings that track Chatbot Arena's hard-prompt rankings; they remain one model's judgment of another.
At a glance
- Mechanism
- Learned judgePrior: learned judges, rung 3 of 4
- Item format
- Open-ended
- Size
- 1,024 challenging real-world queries
- Typical uses
- Measure
- Saturation
- Active
- Contamination
- Medium risk
- Year
- 2024
1Claim
What property are we trying to infer, over what world, and for what decision?
- Construct
- Response quality on challenging, real-world user queries
- Population
- 1,024 hard, diverse queries mined from real chatbot conversation logs (WildChat and LMSYS-Chat-1M)
- Measures
- Response quality on challenging, authentic user queries that reflect real-world usage patterns
- Performance evaluated against detailed, task-specific scoring rubrics rather than generic quality
- Ability to handle the diversity and difficulty of queries that real users bring to chatbots
- Fine-grained quality assessment across multiple evaluation dimensions per task
- Does not measure
- Performance on easy or common tasks — the benchmark deliberately selects for challenging queries
- Safety, ethical behavior, or refusal quality on harmful requests
- Consistency of quality across many interactions, as each query is evaluated independently
- Multi-turn conversational ability, as evaluation focuses on single-turn response quality
2Instrument
How is the judgment actually produced?
- Mechanism
Learned judge
GPT-4 Turbo scores or compares responses using a structured, checklist-guided prompt.
Hierarchy prior: learned judges, rung 3 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.
- Item format
- Open-ended
- Specification
- A response is judged against a task-specific checklist of what a high-quality answer should include, either scored on its own (WB-Score) or compared with baseline models' responses (WB-Reward).
- Anchor
- Automatically generated task-specific checklists; baseline models' responses for pairwise comparison
- Evidence view
- The query, the response or responses, and the checklist
- Signal
- Score per response (WB-Score) or graded pairwise preference against a baseline (WB-Reward)
- Resolution
- Single response per task
- Aggregation
- Mean WB-Score; WB-Reward averaged over pairwise comparisons with baseline models
- Metrics
- WB-Score, WB-Reward
- Size
- 1,024 challenging real-world queries
- Methodology
- Mines 1,024 challenging user queries from real chatbot conversation logs (WildChat and LMSYS-Chat-1M), filtering for difficulty and diversity. For each query, a detailed evaluation checklist is generated specifying what a high-quality response should include. An LLM judge (GPT-4 Turbo) evaluates model responses against these task-specific checklists, providing structured scoring rather than holistic preference. The WB-Score metric correlates well with Chatbot Arena hard-prompt rankings.
3Regime
What causal role does the signal play, and what happens when a system adapts to it?
- Typical uses
- Measure · supports inference and decision
- Exposure
- Tasks and checklists have been public since 2024, and the source conversation logs are public datasets. Agreement with Chatbot Arena is a convergent check against another style-sensitive preference signal, not an independent anchor.
4ValidationDraft
Why should we believe that the signal means what we claim it means — before and after the system adapts to it?
WildBench's evidence supports more meaningful LLM-judge comparisons than earlier approaches such as AlpacaEval or MT-Bench, by combining real user queries, task-specific checklists, and correlation checks against Chatbot Arena. Its strong agreement with hard-prompt Arena rankings suggests it captures something real about quality on challenging tasks, but the basic limit of every LLM-judge benchmark remains: it is one model's opinion about another.
Evidence notes1 supported · 2 partially supported · 1 untested
- Supported
Model rankings agree with Chatbot Arena's hard-prompt rankings
The authors report strong correlation between WildBench scores and Arena hard-prompt ratings for top models.
Source: Lin et al., 2024
- Untested
Checklists capture what makes a response good
The checklists are generated automatically and may not capture what makes a response truly good.
- Partially supported
Scores reflect human judgment rather than the judge's own preferences
Checklist guidance and Arena agreement support the scores, but dependence on GPT-4 Turbo as the judge carries its specific biases into every score.
- Partially supported
The query set represents real user needs
Queries come from real logs, but from specific chat platforms, and selection deliberately favors hard queries, so easy everyday use is under-represented.
5Lifecycle
Does the validity survive over time, and are we still evaluating the right thing?
- Saturation
- Active
Whether top systems still differ, or crowd the ceiling.
- Contamination
- Medium risk
How likely test items, or close variants, have leaked into training data.
- Successors
- None listed
- Last reviewed
- Not yet reviewed · card updated October 12, 2026
Limitations
- LLM judge evaluation, even with detailed rubrics, remains an approximation of human judgment
- Query selection from specific chat platforms may not represent all user populations or use cases
- Single-turn evaluation misses important aspects of extended conversational interactions
- The automatically generated checklists may not always capture what makes a response truly good
- Dependence on GPT-4 Turbo as judge introduces specific biases in quality assessment
When to use
- Evaluating model quality on hard, authentic user queries that stress-test real capabilities
- Comparing models with a more rigorous LLM-judge methodology than simple pairwise preference
- Assessing readiness for deployment on the types of challenging queries users actually submit
When not to use
- When you need human evaluation rather than LLM-based judgment for high-stakes decisions
- Evaluating safety or ethical behavior, which requires dedicated safety benchmarks
Related benchmarks
- Chatbot ArenaPreference · 2023A crowdsourced platform where users have blind conversations with two anonymous models and vote for the one they prefer. Ratings on an Elo-like scale are fitted to a large and growing pool of pairwise human preference votes.
- AlpacaEvalPreference · 2023An automated evaluation framework that uses a strong LLM judge (GPT-4 Turbo) to compare model outputs against a reference model on 805 instructions. AlpacaEval 2.0 introduced length-controlled win rates to mitigate the bias toward verbose responses.
- MT-BenchPreference · 2023A multi-turn conversation benchmark with 80 carefully crafted two-turn questions across 8 categories. An LLM judge scores responses on a 1-10 scale, evaluating both first-turn quality and the ability to handle follow-up instructions.