Benchmarks, read as evaluators

AlpacaEval 2.0 length-controlled win rates support fast, cheap screening of instruction-following quality as judged by GPT-4 Turbo against a reference model; they do not support final claims about quality, correctness, or models stronger than the judge.

ARC-Challenge accuracy supports only a basic check that very small or constrained models can answer grade-school science multiple-choice questions; it carries no information for ranking current LLMs.

AssistantBench scores support comparing web agents on realistic, time-consuming open-web tasks with verifiable answers, as a dated snapshot; the small task set and a changing web limit precision and reproducibility.

BBH accuracy supports per-task comparisons of few-shot chain-of-thought performance on 23 short, formulaic reasoning tasks for mid-range models; its macro-average no longer separates frontier models.

Chatbot Arena ratings support claims about which models the platform's self-selected users prefer, in blind pairwise comparison, on the prompts they choose to bring; they do not support claims about accuracy, safety, or any specific capability.

GAIA accuracy supports comparing complete assistant systems (model, tools, and scaffold) on short-answer questions that require web browsing, file handling, and multi-step reasoning; it does not isolate model capability or grade the process.

GPQA accuracy, usually on the 198-question Diamond subset, supports coarse comparisons of frontier models on expert-written biology, physics, and chemistry questions that resist web lookup; gaps of a few points are within sampling noise.

GSM8K accuracy supports a smoke test that small or fine-tuned models can solve two- to eight-step grade-school arithmetic word problems; it supports no claim about the mathematical reasoning of current frontier models.

HarmBench attack success rates support relative comparisons of models and defenses against a fixed suite of known red-teaming attacks on 510 harmful behaviors, under a standardized pipeline; they do not certify that a model is safe against adaptive attackers.

HellaSwag accuracy supports only a check that small models prefer plausible continuations of everyday activity descriptions; it carries no information for ranking current LLMs.

HumanEval pass@1 supports a basic check that a model can turn a docstring into a short, standalone Python function that passes a few unit tests; it does not support comparing frontier models or claims about software engineering.

LiveCodeBench pass@1 supports contamination-resistant comparisons of algorithmic code generation on competitive-programming problems released after a model's training cutoff; it does not support claims about real-world software engineering or code quality.

MATH accuracy supports comparing models' final-answer performance on competition-style problems across five difficulty levels, with most remaining signal at Level 5; it does not support claims about mathematical ability beyond competition math.

MathVista accuracy supports comparing vision-language models on mathematical reasoning grounded in charts, plots, geometry figures, and scientific diagrams; its aggregate mixes 31 source datasets of uneven quality, so per-type results support more specific claims.

MBPP pass@1 supports a basic check that small or specialized models can write short, entry-level Python functions that pass three example tests; it does not support ranking current models or broader claims about programming skill.

MMLU accuracy supports a coarse screen of whether a model can recognize correct answers to English-language academic exam questions across 57 subjects; it no longer supports ranking frontier models or claims about understanding.

MMLU-Pro accuracy supports comparing strong models on STEM-weighted, reasoning-heavy ten-option knowledge questions under a fixed chain-of-thought protocol; it does not support claims about balanced knowledge breadth or open-ended expertise.

MMMU accuracy supports comparing vision-language models on college-level, image-based exam questions across 30 subjects; some items are answerable from text alone, so it overstates visual reasoning for text-strong models.

MT-Bench scores support a quick, coarse read of two-turn conversational quality across eight categories as graded by GPT-4; with 80 questions and compressed scores, they do not separate frontier models.

ScienceAgentBench success rates support comparing agents on writing Python programs that reproduce data-driven analyses from peer-reviewed papers in four disciplines; the small task set limits precision, and they say nothing about hypothesis generation or the wider research process.

SWE-bench Verified resolve rates support comparing complete agent systems (model plus scaffold) on fixing real issues in 12 popular open-source Python repositories, as judged by the fix's own tests; they do not isolate model capability or extend to other languages or closed codebases.

Closed-book TriviaQA accuracy supports only a basic check that small or older models can recall common trivia facts; it carries no information for ranking current LLMs.

TruthfulQA supports a targeted check of whether a model repeats common misconceptions on 817 adversarially chosen English questions; it does not support broad claims about honesty or hallucination rates.

VQAv2 accuracy supports only a basic check of visual grounding on everyday COCO photographs and comparison with pre-LLM vision-language systems; it carries no information for ranking current multimodal models.

WebArena success rates support comparing autonomous web agents on end-to-end task completion in self-hosted replicas of real web applications; they do not support claims about the live web or about partial progress.

WildBench scores support comparing chat models on 1,024 challenging real user queries, judged by GPT-4 Turbo against task-specific checklists, with rankings that track Chatbot Arena's hard-prompt rankings; they remain one model's judgment of another.

WMDP accuracy supports measuring whether a model's recognition of hazardous-adjacent biosecurity, cybersecurity, and chemical-security knowledge drops after an unlearning or safety intervention; it does not support claims about operational weapons capability or deployment risk.

τ-bench pass rates support comparing tool-using agents on policy-compliant, multi-turn customer-service tasks in simulated retail and airline domains, including consistency across repeated trials; they do not support claims about other domains or real customers.