# AlpacaEval

> An automated evaluation framework that uses a strong LLM judge (GPT-4 Turbo) to compare model outputs against a reference model on 805 instructions. AlpacaEval 2.0 introduced length-controlled win rates to mitigate the bias toward verbose responses.

## Claim

- **headline**: AlpacaEval 2.0 length-controlled win rates support fast, cheap screening of instruction-following quality as judged by GPT-4 Turbo against a reference model; they do not support final claims about quality, correctness, or models stronger than the judge.
- **construct**: Instruction-following quality as perceived by a strong LLM judge
- **population**: 805 general, mostly easy single-turn instructions from the AlpacaFarm evaluation set
- **measures**: Instruction-following quality as judged by an LLM evaluator against a reference baseline; Relative improvement over a reference model (GPT-4 Turbo) on diverse instruction types; Length-controlled win rate (v2) that attempts to disentangle quality from verbosity; Broad instruction-following competence across creative, analytical, and informational tasks
- **doesNotMeasure**: Objective correctness — the LLM judge evaluates perceived quality, not factual accuracy; Performance on hard, specialized tasks — the 805 instructions are general and mostly easy; Capabilities that the judge model itself cannot evaluate well (e.g., novel reasoning, domain expertise); Real human preferences, which may diverge from GPT-4's judgments in important ways; Safety, robustness, or consistency across different interaction contexts

## Instrument

- **itemFormat**: pairwise
- **specification**: For each instruction, the judge decides whether the evaluated model's output is better than the reference model's output.
- **anchor**: Reference outputs from a baseline model (GPT-4 Turbo in version 2.0); the judge's preferences define "better"
- **evidenceView**: The instruction and the two outputs; the judge has no ground truth or sources
- **mechanism**: learned_judge
- **mechanismDetail**: GPT-4 Turbo with a fixed pairwise comparison prompt.
- **signal**: Pairwise preference against the reference output
- **resolution**: Single response per instruction
- **aggregation**: Win rate against the reference; the length-controlled win rate fits a regression (GLM) that removes the predicted effect of output length
- **metrics**: length-controlled win rate; raw win rate
- **size**: 805 instructions
- **methodology**: Presents 805 instructions from the AlpacaFarm evaluation set to the model being evaluated. Responses are compared pairwise against a reference model's outputs using GPT-4 Turbo as a judge. AlpacaEval 1.0 reports raw win rate; AlpacaEval 2.0 applies a length-controlled adjustment using a GLM to deweight length as a confound. Results correlate with Chatbot Arena rankings, which is cited as validation.

## Regime

- **typicalUses**: measure; optimize
- **exposure**: Instructions, reference outputs, and the judge prompt are public, and the leaderboard accepts submissions, so models can be tuned directly against the judge. Verbosity gaming motivated length control, and null models that ignore the instruction have been shown to reach high win rates.

## Validation

- **summary**: AlpacaEval's evidence supports it as a fast development signal: the length-controlled win rate addresses the most glaring bias in LLM-as-judge scoring, and rankings correlate with Chatbot Arena. It remains one LLM's opinion of another's outputs, with the circularity that implies, and its 805 mostly easy instructions do not support serious frontier comparisons.
- **evidence**: Rankings agree with human preference rankings; Length control removes verbosity bias; Resists deliberate gaming
- **status**: draft

## Lifecycle

- **saturation**: active
- **contamination**: medium
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/alpacaeval