Preference · Evaluator card
MT-Bench
2023Lianmin Zheng et al. (LMSYS)Paper (opens in a new tab)
A multi-turn conversation benchmark with 80 carefully crafted two-turn questions across 8 categories. An LLM judge scores responses on a 1-10 scale, evaluating both first-turn quality and the ability to handle follow-up instructions.
Evidence notes pending review. The verdicts on this card are drafts derived from the benchmark's documentation and published sources; no reviewer has checked them against those sources yet.
Headline claim
MT-Bench scores support a quick, coarse read of two-turn conversational quality across eight categories as graded by GPT-4; with 80 questions and compressed scores, they do not separate frontier models.
At a glance
- Mechanism
- Learned judgePrior: learned judges, rung 3 of 4
- Item format
- Open-ended
- Size
- 80 two-turn conversations across 8 categories
- Typical uses
- Measure
- Saturation
- Near saturation
- Contamination
- High risk
- Year
- 2023
1Claim
What property are we trying to infer, over what world, and for what decision?
- Construct
- Multi-turn instruction following and conversational quality
- Population
- 80 two-turn questions, 10 in each of 8 categories: writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities
- Measures
- Multi-turn conversational ability including response to follow-up questions and instructions
- Quality across 8 categories: writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities
- Ability to maintain context and coherence across conversation turns
- Overall instruction-following quality as rated by an LLM judge on a 10-point scale
- Does not measure
- Performance on conversations longer than 2 turns — real conversations are much longer
- Objective correctness for math and coding categories where the judge may not catch errors
- Real user satisfaction, which depends on many factors beyond what 80 curated questions can capture
- Edge cases, adversarial inputs, or the long tail of real-world conversational challenges
- Meaningful differences between frontier models, as scores are highly compressed near the top
2Instrument
How is the judgment actually produced?
- Mechanism
Learned judge
GPT-4 as a single-answer grader that returns a score with an explanation.
Hierarchy prior: learned judges, rung 3 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.
- Item format
- Open-ended
- Specification
- Each turn's response receives a score from 1 to 10 from the judge under a fixed grading prompt.
- Anchor
- The judge's grading prompt; reference answers guide grading for math and reasoning questions
- Evidence view
- The conversation (question, follow-up, and responses), plus the reference answer where one is provided
- Signal
- Score from 1 to 10 per turn
- Resolution
- Each turn's response
- Aggregation
- Mean score per category and overall
- Metrics
- average score (1-10)
- Size
- 80 two-turn conversations across 8 categories
- Methodology
- Contains 80 two-turn conversations spanning 8 categories (10 per category). The first turn poses a question or task, and the second turn adds a follow-up or modification. GPT-4 acts as judge, scoring each turn's response from 1-10 with an explanation. Scores are averaged per category and overall. The two-turn format specifically tests whether models can handle follow-up instructions that build on prior context.
3Regime
What causal role does the signal play, and what happens when a system adapts to it?
- Typical uses
- Measure · supports inference and decision
- Exposure
- 80 public questions, with reference answers and model judgments published since 2023, widely used as a quick development signal for fine-tuned chat models; contamination risk is high and the judge prompt is public.
4ValidationDraft
Why should we believe that the signal means what we claim it means — before and after the system adapts to it?
MT-Bench's evidence supported its original contributions — multi-turn evaluation and the LLM-as-judge method — and its eight categories give some per-domain signal. Eighty questions are far too few for reliable model comparison, two turns barely test conversational ability, and score compression at the top means it no longer distinguishes frontier models; WildBench carries its ideas forward.
Evidence notes1 partially supported · 2 not supported
- Partially supported
GPT-4 judgments agree with human preferences
The original study found GPT-4 agreed with human experts about as often as experts agreed with each other, but also documented position, verbosity, and self-enhancement biases and weaker grading of math and reasoning.
Source: Zheng, L. et al., 2023
- Not supported
Separates frontier models
Frontier models cluster between about 8.5 and 9.5, and 80 questions cannot resolve small gaps.
- Not supported
Resists deliberate gaming
A null model returning a fixed crafted response reached a near-maximal score.
Source: Zheng, X. et al., 2024 (Cheating Automatic LLM Benchmarks)
5Lifecycle
Does the validity survive over time, and are we still evaluating the right thing?
- Saturation
- Near saturation
Whether top systems still differ, or crowd the ceiling.
- Contamination
- High risk
How likely test items, or close variants, have leaked into training data.
- Successors
- Last reviewed
- Not yet reviewed · card updated October 12, 2026
Limitations
- Only 80 questions — extremely small for reliable statistical comparisons between strong models
- Two-turn maximum is a shallow proxy for real multi-turn conversation ability
- LLM judge scores are compressed at the top, with frontier models clustered between 8.5 and 9.5
- The 1-10 scoring scale is subjective and judge-model-dependent
- Category balance (10 per category) means some categories have too few questions for reliable scoring
When to use
- Quick assessment of multi-turn conversational quality across diverse task categories
- Evaluating whether a model handles follow-up instructions and context-dependent requests
When not to use
- Differentiating between frontier models — the score compression makes small differences meaningless
- Making claims about conversational ability based on only 80 two-turn exchanges
- Evaluating real conversational depth, which requires far more than 2 turns
Related benchmarks
- Chatbot ArenaPreference · 2023A crowdsourced platform where users have blind conversations with two anonymous models and vote for the one they prefer. Ratings on an Elo-like scale are fitted to a large and growing pool of pairwise human preference votes.
- AlpacaEvalPreference · 2023An automated evaluation framework that uses a strong LLM judge (GPT-4 Turbo) to compare model outputs against a reference model on 805 instructions. AlpacaEval 2.0 introduced length-controlled win rates to mitigate the bias toward verbose responses.
- WildBenchPreference · 2024A benchmark of 1,024 challenging, real-world user queries mined from actual chatbot interaction logs. It uses a detailed LLM judge rubric with task-specific checklists to evaluate response quality more rigorously than prior LLM-as-judge approaches.