Part 2 · The five layers
Instrument
How the judgment is actually produced: specification, anchor, evidence view, mechanism, signal, resolution and aggregation.
- 2.1Specification: what counts as successTurn a latent construct into a checkable demand, outcome or trajectory, boolean or margin, without mistaking formality for validity.10 min read
- 2.2Rubrics and criterion cardsInteractiveA rubric is an intermediate representation, not a weak verifier. Compile tacit expertise into cards that send each criterion to its strongest mechanism.10 min read
- 2.3Anchors and evidence viewsWhat the judgment is rooted in, and what the evaluator may see: two coordinates most taxonomies forget, and the cheapest lever for better evaluators.6 min read
- 2.4Mechanism familiesProof checkers, executors, rule engines, learned judges, intrinsic signals and people: what each can guarantee, and what each can only estimate.10 min read
- 2.5LLM judges in practiceInteractiveAn LLM judge is a learned measurement instrument. Design its inputs, characterize its biases, calibrate it against adjudicated labels, and version it.10 min read
- 2.6Signals and resolutionVerdict, score, preference or critique; about a step, action, artifact, trajectory or population. Why PRMs, value models and margins are different objects.10 min read
- 2.7Aggregation geometryInteractiveWeighted means assume strengths can offset weaknesses; gates assert some failures are fatal. How criteria combine is a normative decision, so make it explicitly.9 min read