Evaluator engineering

Engineer evaluators you can defend.

Evaluator engineering is the discipline of turning a construct into an instrument, validating the inference that instrument supports, and ensuring that the validity survives the causal regime in which its signal is used.

Read the handbook See the framework Open the Studio

Validation feeds back into redesign; lifecycle controls keep validity true over time.

Three questions organize the field

1. Claim (current)
2. Instrument (current)
3. Regime
4. Validation
5. Lifecycle

Creation

How do we turn human intent into an operational signal?

Start with Constructs

1. Claim
2. Instrument
3. Regime
4. Validation (current)
5. Lifecycle

Validation

Why should we believe that the signal means what we claim it means?

Start with The validity argument

1. Claim
2. Instrument
3. Regime (current)
4. Validation
5. Lifecycle (current)

Use

What causal role does the signal play, and what happens when a system adapts to it?

Start with Use regimes

The idea in one widget

Same judge, four jobs

An LLM judge that agrees with experts 90% of the time can be a fine reporter, a weak selector, a dangerous training reward, and a noisy production gate — with identical weights. The mechanism hasn't changed; the causal regime has.

That is why evaluator engineering starts with the claim and the regime, not the metric.

Read about use regimes

InteractiveSame judge, four jobs3Regime

Four ways in

Handbook

59 pages · 11 parts

Definition-first chapters with worked cases, interactive calculators and check-yourself prompts.

Framework

5 layers · 31 coordinates

One page that maps the evaluator system from claim to lifecycle, with every coordinate linked.

Benchmarks

28 evaluator cards

Popular benchmarks read through the framework: what each supports, what it doesn't, and why.

Studio

Multi-agent workflow

Design an evaluator for your own system and leave with a dossier and a validity statement.