Evaluators deserve engineering

Benchmaker exists because AI systems are increasingly trusted through their evaluations rather than by inspection. If the evaluation is wrong, everything downstream of it — releases, rankings, rewards, monitors — inherits the error. We think evaluators deserve the same engineering discipline as the systems they judge.

What Benchmaker is#

Benchmaker is a handbook, a framework, a catalog and a studio for evaluator engineering: turning a construct into an instrument, validating the inference that instrument supports, and keeping that validity true in the causal regime where its signal is used.

• The Handbook is the reference: definition-first chapters, worked case studies, interactive calculators, and exercises.
• The Framework is the map: one page that lays out the evaluator system from claim to lifecycle.
• Benchmarks reads popular benchmarks through that map: what each one supports, what it doesn't, and why.
• The Studio helps you design an evaluator for your own system and leaves you with a dossier and a validity statement.

What we believe#

Validity is a claim, not a sticker. No benchmark is "valid" in the abstract. Evidence supports an interpretation, for a population, for a decision, in a regime — and stops there.

The evaluator is load-bearing. The same judge can be a fine reporter and a dangerous reward. What an evaluator is used for changes what we need to know about it.

Guarantee before you estimate. Where a property can be made true by construction, do that; spend measurement on what construction cannot reach.

Practice what we preach. The Studio is itself an evaluator system: its deterministic checks, independent critic, and human checkpoints are designed with the same principles the handbook teaches, and it publishes its own validity statement.

Sources and credits#

The handbook builds on the measurement and evaluation literature, and especially on:

• Chen, Wang & Qu, Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (2026).
• The Berkeley CS 194/294 lecture on LLM agent evaluations (2025), and the agentic benchmark checklist of Zhu et al. (2025).
• The Studio's orchestration design follows the workflow-first principles in Victor Dibia, Designing Multi-Agent Systems (2025).

The full annotated list is in Sources and further reading.