# WildBench

> A benchmark of 1,024 challenging, real-world user queries mined from actual chatbot interaction logs. It uses a detailed LLM judge rubric with task-specific checklists to evaluate response quality more rigorously than prior LLM-as-judge approaches.

## Claim

- **headline**: WildBench scores support comparing chat models on 1,024 challenging real user queries, judged by GPT-4 Turbo against task-specific checklists, with rankings that track Chatbot Arena's hard-prompt rankings; they remain one model's judgment of another.
- **construct**: Response quality on challenging, real-world user queries
- **population**: 1,024 hard, diverse queries mined from real chatbot conversation logs (WildChat and LMSYS-Chat-1M)
- **measures**: Response quality on challenging, authentic user queries that reflect real-world usage patterns; Performance evaluated against detailed, task-specific scoring rubrics rather than generic quality; Ability to handle the diversity and difficulty of queries that real users bring to chatbots; Fine-grained quality assessment across multiple evaluation dimensions per task
- **doesNotMeasure**: Performance on easy or common tasks — the benchmark deliberately selects for challenging queries; Safety, ethical behavior, or refusal quality on harmful requests; Consistency of quality across many interactions, as each query is evaluated independently; Multi-turn conversational ability, as evaluation focuses on single-turn response quality

## Instrument

- **itemFormat**: open-ended
- **specification**: A response is judged against a task-specific checklist of what a high-quality answer should include, either scored on its own (WB-Score) or compared with baseline models' responses (WB-Reward).
- **anchor**: Automatically generated task-specific checklists; baseline models' responses for pairwise comparison
- **evidenceView**: The query, the response or responses, and the checklist
- **mechanism**: learned_judge
- **mechanismDetail**: GPT-4 Turbo scores or compares responses using a structured, checklist-guided prompt.
- **signal**: Score per response (WB-Score) or graded pairwise preference against a baseline (WB-Reward)
- **resolution**: Single response per task
- **aggregation**: Mean WB-Score; WB-Reward averaged over pairwise comparisons with baseline models
- **metrics**: WB-Score; WB-Reward
- **size**: 1,024 challenging real-world queries
- **methodology**: Mines 1,024 challenging user queries from real chatbot conversation logs (WildChat and LMSYS-Chat-1M), filtering for difficulty and diversity. For each query, a detailed evaluation checklist is generated specifying what a high-quality response should include. An LLM judge (GPT-4 Turbo) evaluates model responses against these task-specific checklists, providing structured scoring rather than holistic preference. The WB-Score metric correlates well with Chatbot Arena hard-prompt rankings.

## Regime

- **typicalUses**: measure
- **exposure**: Tasks and checklists have been public since 2024, and the source conversation logs are public datasets. Agreement with Chatbot Arena is a convergent check against another style-sensitive preference signal, not an independent anchor.

## Validation

- **summary**: WildBench's evidence supports more meaningful LLM-judge comparisons than earlier approaches such as AlpacaEval or MT-Bench, by combining real user queries, task-specific checklists, and correlation checks against Chatbot Arena. Its strong agreement with hard-prompt Arena rankings suggests it captures something real about quality on challenging tasks, but the basic limit of every LLM-judge benchmark remains: it is one model's opinion about another.
- **evidence**: Model rankings agree with Chatbot Arena's hard-prompt rankings; Checklists capture what makes a response good; Scores reflect human judgment rather than the judge's own preferences; The query set represents real user needs
- **status**: draft

## Lifecycle

- **saturation**: active
- **contamination**: medium
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/wildbench