# MT-Bench

> A multi-turn conversation benchmark with 80 carefully crafted two-turn questions across 8 categories. An LLM judge scores responses on a 1-10 scale, evaluating both first-turn quality and the ability to handle follow-up instructions.

## Claim

- **headline**: MT-Bench scores support a quick, coarse read of two-turn conversational quality across eight categories as graded by GPT-4; with 80 questions and compressed scores, they do not separate frontier models.
- **construct**: Multi-turn instruction following and conversational quality
- **population**: 80 two-turn questions, 10 in each of 8 categories: writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities
- **measures**: Multi-turn conversational ability including response to follow-up questions and instructions; Quality across 8 categories: writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities; Ability to maintain context and coherence across conversation turns; Overall instruction-following quality as rated by an LLM judge on a 10-point scale
- **doesNotMeasure**: Performance on conversations longer than 2 turns — real conversations are much longer; Objective correctness for math and coding categories where the judge may not catch errors; Real user satisfaction, which depends on many factors beyond what 80 curated questions can capture; Edge cases, adversarial inputs, or the long tail of real-world conversational challenges; Meaningful differences between frontier models, as scores are highly compressed near the top

## Instrument

- **itemFormat**: open-ended
- **specification**: Each turn's response receives a score from 1 to 10 from the judge under a fixed grading prompt.
- **anchor**: The judge's grading prompt; reference answers guide grading for math and reasoning questions
- **evidenceView**: The conversation (question, follow-up, and responses), plus the reference answer where one is provided
- **mechanism**: learned_judge
- **mechanismDetail**: GPT-4 as a single-answer grader that returns a score with an explanation.
- **signal**: Score from 1 to 10 per turn
- **resolution**: Each turn's response
- **aggregation**: Mean score per category and overall
- **metrics**: average score (1-10)
- **size**: 80 two-turn conversations across 8 categories
- **methodology**: Contains 80 two-turn conversations spanning 8 categories (10 per category). The first turn poses a question or task, and the second turn adds a follow-up or modification. GPT-4 acts as judge, scoring each turn's response from 1-10 with an explanation. Scores are averaged per category and overall. The two-turn format specifically tests whether models can handle follow-up instructions that build on prior context.

## Regime

- **typicalUses**: measure
- **exposure**: 80 public questions, with reference answers and model judgments published since 2023, widely used as a quick development signal for fine-tuned chat models; contamination risk is high and the judge prompt is public.

## Validation

- **summary**: MT-Bench's evidence supported its original contributions — multi-turn evaluation and the LLM-as-judge method — and its eight categories give some per-domain signal. Eighty questions are far too few for reliable model comparison, two turns barely test conversational ability, and score compression at the top means it no longer distinguishes frontier models; WildBench carries its ideas forward.
- **evidence**: GPT-4 judgments agree with human preferences; Separates frontier models; Resists deliberate gaming
- **status**: draft

## Lifecycle

- **saturation**: near-saturated
- **contamination**: high
- **successors**: wildbench
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/mt-bench