Preference · Evaluator card
AlpacaEval
2023Xuechen Li et al. (Stanford)Paper (opens in a new tab)
An automated evaluation framework that uses a strong LLM judge (GPT-4 Turbo) to compare model outputs against a reference model on 805 instructions. AlpacaEval 2.0 introduced length-controlled win rates to mitigate the bias toward verbose responses.
Evidence notes pending review. The verdicts on this card are drafts derived from the benchmark's documentation and published sources; no reviewer has checked them against those sources yet.
Headline claim
AlpacaEval 2.0 length-controlled win rates support fast, cheap screening of instruction-following quality as judged by GPT-4 Turbo against a reference model; they do not support final claims about quality, correctness, or models stronger than the judge.
At a glance
- Mechanism
- Learned judgePrior: learned judges, rung 3 of 4
- Item format
- Pairwise
- Size
- 805 instructions
- Typical uses
- Measure, Optimize
- Saturation
- Active
- Contamination
- Medium risk
- Year
- 2023
1Claim
What property are we trying to infer, over what world, and for what decision?
- Construct
- Instruction-following quality as perceived by a strong LLM judge
- Population
- 805 general, mostly easy single-turn instructions from the AlpacaFarm evaluation set
- Measures
- Instruction-following quality as judged by an LLM evaluator against a reference baseline
- Relative improvement over a reference model (GPT-4 Turbo) on diverse instruction types
- Length-controlled win rate (v2) that attempts to disentangle quality from verbosity
- Broad instruction-following competence across creative, analytical, and informational tasks
- Does not measure
- Objective correctness — the LLM judge evaluates perceived quality, not factual accuracy
- Performance on hard, specialized tasks — the 805 instructions are general and mostly easy
- Capabilities that the judge model itself cannot evaluate well (e.g., novel reasoning, domain expertise)
- Real human preferences, which may diverge from GPT-4's judgments in important ways
- Safety, robustness, or consistency across different interaction contexts
2Instrument
How is the judgment actually produced?
- Mechanism
Learned judge
GPT-4 Turbo with a fixed pairwise comparison prompt.
Hierarchy prior: learned judges, rung 3 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.
- Item format
- Pairwise
- Specification
- For each instruction, the judge decides whether the evaluated model's output is better than the reference model's output.
- Anchor
- Reference outputs from a baseline model (GPT-4 Turbo in version 2.0); the judge's preferences define "better"
- Evidence view
- The instruction and the two outputs; the judge has no ground truth or sources
- Signal
- Pairwise preference against the reference output
- Resolution
- Single response per instruction
- Aggregation
- Win rate against the reference; the length-controlled win rate fits a regression (GLM) that removes the predicted effect of output length
- Metrics
- length-controlled win rate, raw win rate
- Size
- 805 instructions
- Methodology
- Presents 805 instructions from the AlpacaFarm evaluation set to the model being evaluated. Responses are compared pairwise against a reference model's outputs using GPT-4 Turbo as a judge. AlpacaEval 1.0 reports raw win rate; AlpacaEval 2.0 applies a length-controlled adjustment using a GLM to deweight length as a confound. Results correlate with Chatbot Arena rankings, which is cited as validation.
3Regime
What causal role does the signal play, and what happens when a system adapts to it?
- Typical uses
- Measure · supports inference and decision
- Optimize · shapes the candidate or policy distribution
- Exposure
- Instructions, reference outputs, and the judge prompt are public, and the leaderboard accepts submissions, so models can be tuned directly against the judge. Verbosity gaming motivated length control, and null models that ignore the instruction have been shown to reach high win rates.
4ValidationDraft
Why should we believe that the signal means what we claim it means — before and after the system adapts to it?
AlpacaEval's evidence supports it as a fast development signal: the length-controlled win rate addresses the most glaring bias in LLM-as-judge scoring, and rankings correlate with Chatbot Arena. It remains one LLM's opinion of another's outputs, with the circularity that implies, and its 805 mostly easy instructions do not support serious frontier comparisons.
Evidence notes2 partially supported · 1 not supported
- Partially supported
Rankings agree with human preference rankings
Length-controlled win rates correlate closely with Chatbot Arena rankings across models, but the agreement breaks down for models with unusual output characteristics.
Source: Dubois et al., 2024
- Partially supported
Length control removes verbosity bias
The regression adjustment reduces sensitivity to output length but is a statistical correction, not a fix, and other judge biases such as position and self-preference remain.
- Not supported
Resists deliberate gaming
A null model that returns the same crafted response to every instruction reached high length-controlled win rates.
Source: Zheng, X. et al., 2024 (Cheating Automatic LLM Benchmarks)
5Lifecycle
Does the validity survive over time, and are we still evaluating the right thing?
- Saturation
- Active
Whether top systems still differ, or crowd the ceiling.
- Contamination
- Medium risk
How likely test items, or close variants, have leaked into training data.
- Successors
- None listed
- Last reviewed
- Not yet reviewed · card updated October 12, 2026
Limitations
- LLM judge biases (verbosity preference, position bias, self-preference) are mitigated but not eliminated
- 805 instructions are relatively easy and do not stress-test model capabilities
- Correlation with Chatbot Arena, while decent, breaks down for models with unusual output characteristics
- The length control in v2 is a statistical correction, not a fundamental fix for the verbosity problem
- Judge model choice (GPT-4 Turbo) introduces a specific and potentially biased perspective on quality
When to use
- Fast, cheap automated evaluation of instruction-following quality during development iterations
- Screening models before more expensive human evaluation or targeted benchmarking
- Tracking instruction-following improvements across training runs or fine-tuning experiments
When not to use
- Making final claims about model quality — LLM judge evaluations are not ground truth
- Evaluating models that are significantly stronger than the judge model itself
- When objective correctness matters more than perceived quality
Related benchmarks
- Chatbot ArenaPreference · 2023A crowdsourced platform where users have blind conversations with two anonymous models and vote for the one they prefer. Ratings on an Elo-like scale are fitted to a large and growing pool of pairwise human preference votes.
- MT-BenchPreference · 2023A multi-turn conversation benchmark with 80 carefully crafted two-turn questions across 8 categories. An LLM judge scores responses on a 1-10 scale, evaluating both first-turn quality and the ability to handle follow-up instructions.
- WildBenchPreference · 2024A benchmark of 1,024 challenging, real-world user queries mined from actual chatbot interaction logs. It uses a detailed LLM judge rubric with task-specific checklists to evaluate response quality more rigorously than prior LLM-as-judge approaches.