# Benchmaker

> A handbook, framework and studio for evaluator engineering: turning a construct into an instrument, validating the inference that instrument supports, and keeping that validity true under optimization and drift.

Every page below is plain Markdown. Pages marked (coming soon) are not written yet.

## Introduction

- [Introduction](https://benchmaker.ai/handbook.md): Evaluation is an engineered chain from intent to evidence to action. What this handbook covers, who it's for, and which path fits what you're trying to do.

## Foundations

- [Evaluation is measurement](https://benchmaker.ai/handbook/foundations/evaluation-is-measurement.md): A score is an observation, not a capability. Follow the instrument chain from construct to decision and the four places error gets in.
- [The evaluator system](https://benchmaker.ai/handbook/foundations/evaluator-systems.md): Meet the master object E_sys = (C, I, R, V, L), and why a proof checker is a mechanism but a release gate built on it is a system.
- [Vocabulary without category errors](https://benchmaker.ai/handbook/foundations/vocabulary.md): Reward model, rubric, PRM, RLHF, verifier: put each term on the axis it actually describes and most confusions disappear.
- [The verification hierarchy, demoted](https://benchmaker.ai/handbook/foundations/verification-hierarchy.md): Formal > execution > learned judges > intrinsic signals. A useful prior on how grounded a signal is, never a verdict on validity.
- [Marr's levels for evaluation](https://benchmaker.ai/handbook/foundations/marr-levels.md): Before arguing about judges, find the level: what claim, what procedure, what machinery.

## Claim

- [Constructs, facets, and nuisances](https://benchmaker.ai/handbook/claim/constructs.md): Define what the score should respond to, and just as carefully what it must ignore. Your nuisance list is your discriminant test plan.
- [Population, environment, harness, and validity domain](https://benchmaker.ai/handbook/claim/validity-domain.md): Every score was earned in some world. Declare the population, environment and harness, and say where the claim stops.
- [Decision claims](https://benchmaker.ai/handbook/claim/decision-claims.md): A score means something only when it changes a decision. Write the decision, the threshold and the cost of each error first.
- [Task distributions and dataset engineering](https://benchmaker.ai/handbook/claim/task-distributions.md): A dataset is a model of a population. Grow it from real failures, sample where the information is, reweight to the world, and keep leaks out.

## Instrument

- [Specification: what counts as success](https://benchmaker.ai/handbook/instrument/specification.md): Turn a latent construct into a checkable demand, outcome or trajectory, boolean or margin, without mistaking formality for validity.
- [Rubrics and criterion cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards.md): A rubric is an intermediate representation, not a weak verifier. Compile tacit expertise into cards that send each criterion to its strongest mechanism.
- [Anchors and evidence views](https://benchmaker.ai/handbook/instrument/anchors-and-evidence-views.md): What the judgment is rooted in, and what the evaluator may see: two coordinates most taxonomies forget, and the cheapest lever for better evaluators.
- [Mechanism families](https://benchmaker.ai/handbook/instrument/mechanisms.md): Proof checkers, executors, rule engines, learned judges, intrinsic signals and people: what each can guarantee, and what each can only estimate.
- [LLM judges in practice](https://benchmaker.ai/handbook/instrument/llm-judges.md): An LLM judge is a learned measurement instrument. Design its inputs, characterize its biases, calibrate it against adjudicated labels, and version it.
- [Signals and resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution.md): Verdict, score, preference or critique; about a step, action, artifact, trajectory or population. Why PRMs, value models and margins are different objects.
- [Aggregation geometry](https://benchmaker.ai/handbook/instrument/aggregation-geometry.md): Weighted means assume strengths can offset weaknesses; gates assert some failures are fatal. How criteria combine is a normative decision, so make it explicitly.

## Regime

- [Use regimes: same judge, four jobs](https://benchmaker.ai/handbook/regime/use-regimes.md): Measure, optimize or monitor: an evaluator changes meaning when its signal enters a different loop, and signals migrate between loops.
- [Exposure, modification, and access](https://benchmaker.ai/handbook/regime/exposure-and-surfaces.md): Who consumes the signal, who optimizes against it, how hard, what they can change, and what they can see.
- [Evaluators inside training loops](https://benchmaker.ai/handbook/regime/training-loops.md): RLHF is a pipeline, not an evaluator. Where the learned judge sits in RLHF, RLAIF, RLVR and DPO, and what it is exposed to.
- [Monitoring and control in production](https://benchmaker.ai/handbook/regime/production-monitoring.md): Monitoring is not continuous benchmarking: base rates, calibration, conformal caveats, and the moment a monitor becomes a training signal.

## Validation

- [The validity argument](https://benchmaker.ai/handbook/validation/validity-argument.md): Validity is a claim, not a sticker: a six-link argument from construct to decision, where any link can break.
- [Task validity and outcome validity](https://benchmaker.ai/handbook/validation/task-and-outcome-validity.md): Does success require the capability, and does a pass mean success? The two agent-era links, with checks by output type.
- [Sources of validity evidence](https://benchmaker.ai/handbook/validation/validity-evidence.md): Face, content, response-process, internal-structure, convergent, discriminant and criterion evidence, plus the multitrait–multimethod design.
- [Reliability and generalizability](https://benchmaker.ai/handbook/validation/reliability.md): Specify what may legitimately vary, then measure how much tasks, seeds, judges and environments move the score.
- [The judge as a noisy sensor](https://benchmaker.ai/handbook/validation/judge-as-noisy-sensor.md): Model judge error like a diagnostic test: correct pass rates, carry calibration noise into the result, and audit error per system.
- [Estimands, pass rates, and intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals.md): Write the estimand first, then report n, an honest interval and the right functional, from Wilson to pass^k to reweighted estimates.
- [Comparing systems](https://benchmaker.ai/handbook/validation/comparing-systems.md): Pair tasks, count discordant pairs, size for the smallest difference that matters, and never let significance stand in for validity.
- [Rare-event assurance](https://benchmaker.ai/handbook/validation/rare-events.md): Zero failures in n trials is not zero risk. Bound it, then prevent, search, oversample and monitor.

## Validity under pressure

- [Falsification and failure regions](https://benchmaker.ai/handbook/pressure/falsification.md): Estimation counts failures; falsification finds them. Search for the most likely breaks and map the region, not the anecdote.
- [Adaptive validity](https://benchmaker.ai/handbook/pressure/adaptive-validity.md): Accuracy before any optimization is not the question. How evaluator error grows under best-of-N and training, and how to measure an exploitability curve.
- [Goodhart pressure and self-confirming loops](https://benchmaker.ai/handbook/pressure/goodhart-and-self-confirming-loops.md): Reward hacking needs an exploit; self-confirmation needs only a shared blind spot. Tell them apart and measure the coupling behind both.
- [The validation profile and validity statement](https://benchmaker.ai/handbook/pressure/validation-profile.md): Replace hierarchy labels with measurements, Π = (N, E, T, G, C, D), and ship every evaluator with a falsifiable statement of scope.

## Lifecycle

- [Channels, independence, and anchor burn](https://benchmaker.ai/handbook/lifecycle/channels-and-independence.md): Separate the signal you optimize, the one that certifies, and the one that reports reality. Then budget every bit of anchor information you spend.
- [Living eval suites](https://benchmaker.ai/handbook/lifecycle/living-eval-suites.md): The asset is the machinery that notices when a score stops meaning what it used to: tripwires, drift and saturation.
- [Harness architecture and reproducibility](https://benchmaker.ai/handbook/lifecycle/harness-and-reproducibility.md): The harness is part of the instrument. Isolate trials, pin environments, version judges, and log enough to turn a surprise into a result.
- [Frame review and governance](https://benchmaker.ai/handbook/lifecycle/frame-review.md): Better judges can't tell you whether you're measuring the right thing. Frame review is the governance process that can.

## Agents and self-improvement

- [Evaluating agents](https://benchmaker.ai/handbook/agents/evaluating-agents.md): An agent is a trajectory through a changing world. Evaluate outcome, path, permissions, side effects and honesty, not a final answer.
- [Reading agent benchmarks](https://benchmaker.ai/handbook/agents/agent-benchmarks.md): Capability, application or generalist? Read CyberGym, τ-bench, GDPval and others in unified coordinates before trusting a leaderboard.
- [Self-improvement and evaluator co-evolution](https://benchmaker.ai/handbook/agents/self-improvement.md): When a system can change what defines "better", the evaluator becomes the load-bearing wall. What changes, who validates, how loops fail.

## Practice

- [The One Spine](https://benchmaker.ai/handbook/practice/one-spine.md): Ten steps from claim to frame review that organize every procedure in this handbook, in the order that keeps you out of trouble.
- [Verifiability engineering](https://benchmaker.ai/handbook/practice/verifiability-engineering.md): Don't ask whether a domain is verifiable. Decompose it, make artifacts emit evidence, compile what you can, and keep the residue honest.
- [Guarantee before you estimate](https://benchmaker.ai/handbook/practice/structural-guarantees.md): Make bad states unreachable, verify the scaffold you control, and spend samples only on what construction can't reach.
- [Inverse evaluation design](https://benchmaker.ai/handbook/practice/inverse-evaluation-design.md): Treat expert accept/reject decisions as evidence about a latent judgment, and use the posterior to find criteria, thresholds and the questions worth asking.
- [Auditing an eval](https://benchmaker.ai/handbook/practice/auditing-an-eval.md): Seven questions if you have an hour, twenty-five if you have a week: audit someone else's eval, or your own.
- [Anti-patterns and failure vocabulary](https://benchmaker.ai/handbook/practice/anti-patterns.md): Seventeen named ways evaluator systems fail, each with its tell, its layer and its fix.

## Case studies

- [A: Mathematical reasoning](https://benchmaker.ai/handbook/cases/math-reasoning.md): Answer checkers, proof assistants, PRMs and mathematicians: one construct, four mechanisms, three regimes, and why a PRM is not "above" the checker.
- [B: Coding agent](https://benchmaker.ai/handbook/cases/coding-agent.md): Tests, static analysis, sandbox invariants, trajectory checks and a calibrated readability judge: an agent eval as a portfolio, not a score.
- [C: Enterprise SQL/data agent](https://benchmaker.ai/handbook/cases/sql-data-agent.md): Formalize access policy, execute semantic consequences, judge only the residual. The SQL agent design end to end.
- [D: Autonomous research agent](https://benchmaker.ai/handbook/cases/research-agent.md): "Research quality" is not one unverifiable blob. Route provenance, reproducibility, entailment, novelty and direction-setting to different channels.
- [E: Reasoning-model training](https://benchmaker.ai/handbook/cases/reasoning-model-training.md): Dense RLVR and process feedback for development, an untouched certification channel, and judge error re-estimated at every checkpoint.
- [F: Production safety monitor](https://benchmaker.ai/handbook/cases/production-safety-monitor.md): Permissions first, trajectory checks second, a learned monitor for the residue, validated at production base rates, until training starts to see it.
- [G: Documentation agent](https://benchmaker.ai/handbook/cases/docs-agent.md): A judgment-heavy RAG agent: grounding and procedure criterion cards, SME-calibrated judges, and stale docs as an environment problem rather than a judge problem.

## Reference

- [Glossary](https://benchmaker.ai/handbook/reference/glossary.md): Every term grouped by role, with the chapter that defines it and the collisions it resolves.
- [Notation and formula card](https://benchmaker.ai/handbook/reference/notation-and-formulas.md): Every symbol and formula, plus the rules that stop R, V, κ and A from meaning two things on one page.
- [Templates](https://benchmaker.ai/handbook/reference/templates.md): Copy-ready validity statement, 16-part evaluator dossier, and 15-item minimum reporting checklist.
- [Crosswalk](https://benchmaker.ai/handbook/reference/crosswalk.md): Translate golden datasets, LLM judges, partial credit and release gates into unified coordinates, plus the six-axis type and Raschka's four approaches.
- [Exercises and drills](https://benchmaker.ai/handbook/reference/exercises.md): Analyze, Evaluate, Create: capstone prompts and study drills that test whether the ontology has stuck.
- [Sources and further reading](https://benchmaker.ai/handbook/reference/sources.md): Every work the handbook cites, from the classic measurement literature to the RSI survey and the Berkeley lecture, annotated.

## Articles

- [Evaluating AI Agents: Beyond Task Completion](https://benchmaker.ai/articles/evaluating-agents.md): Agent evaluation requires fundamentally different approaches than model evaluation. Task completion rates alone miss critical dimensions of agent behavior.
- [Binary Beats Likert: Why Simpler Scales Win](https://benchmaker.ai/articles/binary-beats-likert.md): Research consistently shows that binary pass/fail judgments produce more reliable and reproducible evaluation results than Likert scales. Here's why.
- [The Construct Validity Problem in AI Evaluation](https://benchmaker.ai/articles/construct-validity.md): Most AI benchmarks measure proxy constructs, not the capabilities we actually care about. Understanding this gap is essential for designing meaningful evaluations.

## Benchmarks

- [AlpacaEval](https://benchmaker.ai/benchmarks/alpacaeval.md): An automated evaluation framework that uses a strong LLM judge (GPT-4 Turbo) to compare model outputs against a reference model on 805 instructions. AlpacaEval 2.0 introduced length-controlled win rates to mitigate the bias toward verbose responses.
- [ARC](https://benchmaker.ai/benchmarks/arc.md): The AI2 Reasoning Challenge consists of grade-school science questions split into Easy and Challenge sets. The Challenge set filters for questions that simple retrieval and co-occurrence methods fail on.
- [AssistantBench](https://benchmaker.ai/benchmarks/assistantbench.md): A benchmark of challenging, realistic web tasks that require an assistant to navigate the open web to find precise, verifiable answers. Tasks span planning, fact-checking, data collection, and complex information retrieval.
- [BBH](https://benchmaker.ai/benchmarks/bbh.md): BIG-Bench Hard is a curated subset of 23 tasks from the BIG-Bench suite where language models initially performed below average human raters. It tests diverse reasoning skills including logical deduction, causal reasoning, and algorithmic thinking.
- [Chatbot Arena](https://benchmaker.ai/benchmarks/chatbot-arena.md): A crowdsourced platform where users have blind conversations with two anonymous models and vote for the one they prefer. Ratings on an Elo-like scale are fitted to a large and growing pool of pairwise human preference votes.
- [GAIA](https://benchmaker.ai/benchmarks/gaia.md): General AI Assistants benchmark evaluates AI systems on real-world tasks requiring multi-step reasoning, web browsing, tool use, and file manipulation. Questions have unambiguous, verifiable answers but require complex workflows to solve.
- [GPQA](https://benchmaker.ai/benchmarks/gpqa.md): Graduate-Level Google-Proof Questions & Answers is a dataset of extremely difficult questions written and validated by domain experts. Questions are designed so that even skilled non-experts with internet access struggle to answer correctly.
- [GSM8K](https://benchmaker.ai/benchmarks/gsm8k.md): Grade School Math 8K is a dataset of 8,792 grade-school-level math word problems requiring multi-step arithmetic reasoning. Once a key reasoning benchmark, it is now fully saturated and largely serves as a historical reference point.
- [HarmBench](https://benchmaker.ai/benchmarks/harmbench.md): A standardized framework for evaluating both automated red-teaming attacks and model defenses against harmful content generation. It provides a consistent evaluation pipeline for measuring attack success rates across diverse harm categories.
- [HellaSwag](https://benchmaker.ai/benchmarks/hellaswag.md): A commonsense natural language inference benchmark where models must choose the most plausible continuation of an everyday scenario. It was designed using adversarial filtering to be hard for models at the time of creation.
- [HumanEval](https://benchmaker.ai/benchmarks/humaneval.md): A set of 164 hand-written Python programming problems with function signatures, docstrings, and unit tests. It was the first widely-adopted code generation benchmark but is now far too easy and small to be informative.
- [LiveCodeBench](https://benchmaker.ai/benchmarks/livecodebench.md): A code generation benchmark built to resist contamination: it continuously collects new problems from competitive programming platforms (LeetCode, Codeforces, AtCoder) and scores models on problems released after their training cutoffs. It evaluates code generation, self-repair, code execution, and test output prediction.
- [MATH](https://benchmaker.ai/benchmarks/math.md): A dataset of 12,500 competition-level mathematics problems spanning seven subjects from prealgebra to number theory. It remains more discriminating than GSM8K but is approaching saturation for the strongest frontier models.
- [MathVista](https://benchmaker.ai/benchmarks/mathvista.md): A consolidated benchmark for mathematical reasoning in visual contexts, combining problems that require understanding charts, plots, geometry figures, and scientific diagrams to solve mathematical questions.
- [MBPP](https://benchmaker.ai/benchmarks/mbpp.md): Mostly Basic Python Programming contains 974 crowd-sourced Python programming tasks designed to be solvable by entry-level programmers. It complements HumanEval with more problems but at a similar or lower difficulty level.
- [MMLU-Pro](https://benchmaker.ai/benchmarks/mmlu-pro.md): A harder, more discriminating successor to MMLU that expands answer choices to 10 options, increases reasoning demands, and reduces noise from trivially easy questions. It addresses several known MMLU weaknesses while retaining broad subject coverage.
- [MMLU](https://benchmaker.ai/benchmarks/mmlu.md): Massive Multitask Language Understanding evaluates broad academic knowledge across 57 subjects ranging from STEM to humanities. It became the de facto standard for measuring general knowledge in LLMs but is now largely superseded by harder variants.
- [MMMU](https://benchmaker.ai/benchmarks/mmmu.md): Massive Multi-discipline Multimodal Understanding evaluates multimodal models on college-level subject knowledge requiring both image understanding and domain expertise. It spans 30 subjects across art, business, science, health, humanities, and engineering.
- [MT-Bench](https://benchmaker.ai/benchmarks/mt-bench.md): A multi-turn conversation benchmark with 80 carefully crafted two-turn questions across 8 categories. An LLM judge scores responses on a 1-10 scale, evaluating both first-turn quality and the ability to handle follow-up instructions.
- [ScienceAgentBench](https://benchmaker.ai/benchmarks/scienceagentbench.md): A benchmark for evaluating AI agents on data-driven scientific discovery tasks drawn from real published research papers. Agents must write code to process datasets, run experiments, and produce results that match the original paper's findings.
- [SWE-bench](https://benchmaker.ai/benchmarks/swe-bench.md): A benchmark of real-world software engineering tasks drawn from actual GitHub issues and pull requests in popular Python repositories. Models must generate patches that resolve genuine issues and pass the corresponding test suites.
- [τ-bench](https://benchmaker.ai/benchmarks/tau-bench.md): A benchmark for evaluating language model agents as customer service representatives in simulated retail and airline domains. Agents must handle complex customer requests by navigating policies, databases, and multi-turn conversations.
- [TriviaQA](https://benchmaker.ai/benchmarks/triviaqa.md): A large-scale reading comprehension and trivia knowledge dataset with questions authored by trivia enthusiasts. Originally designed for QA systems with evidence documents, it is now mostly used as a closed-book factual recall benchmark.
- [TruthfulQA](https://benchmaker.ai/benchmarks/truthfulqa.md): A benchmark testing whether language models generate truthful answers to questions that are designed to elicit common misconceptions, superstitions, and falsehoods. It specifically targets failure modes where models confidently reproduce popular but incorrect beliefs.
- [VQAv2](https://benchmaker.ai/benchmarks/vqav2.md): Visual Question Answering v2 is a large-scale benchmark where models answer open-ended questions about natural images. It was the foundational VQA benchmark but is now saturated and limited by its simplistic question types.
- [WebArena](https://benchmaker.ai/benchmarks/webarena.md): A realistic web environment benchmark where agents must complete tasks on self-hosted replicas of real websites (Reddit, GitLab, shopping sites, maps, Wikipedia). It evaluates autonomous web navigation and task completion.
- [WildBench](https://benchmaker.ai/benchmarks/wildbench.md): A benchmark of 1,024 challenging, real-world user queries mined from actual chatbot interaction logs. It uses a detailed LLM judge rubric with task-specific checklists to evaluate response quality more rigorously than prior LLM-as-judge approaches.
- [WMDP](https://benchmaker.ai/benchmarks/wmdp.md): The Weapons of Mass Destruction Proxy benchmark measures hazardous knowledge in LLMs related to biosecurity, cybersecurity, and chemical security. It serves as a proxy for evaluating whether models possess dangerous dual-use knowledge.

## Optional

- [Everything in one file](https://benchmaker.ai/llms-full.txt)