Rubrics and criterion cards

A rubric is a specification in structured, human-readable form: a set of criteria, each with the evidence it may use, exemplar anchors or a scale, and gate or aggregation semantics. A criterion card specifies one criterion completely enough to route it to a mechanism, calibrate it, and version it.

A rubric is a specification#

Write a rubric as a set of criteria with their attachments:

$$\Psi = \{(c_i, E_i, A_i, G_i)\}_{i=1}^{k}$$

where:

• $c_i$ is the $i$-th criterion and $k$ the number of criteria;
• $E_i$ is the evidence the evaluator may use for $c_i$;
• $A_i$ is its exemplar anchors or scale;
• $G_i$ is its gate or aggregation semantics.

In plain words: for every criterion, a rubric says what to look for, what to look at, what passing and failing look like, and what a failure costs.

A rubric says what the evaluator should attend to; it does not guarantee that the evaluator attends correctly. The same rubric can be applied by a person, an LLM, a rules engine, or a hybrid. A rubric is therefore not a rung of the verification hierarchy and not a weaker verifier: it is a specification that can send different criteria to different levels of verification at once. The rung belongs to whichever mechanism applies each criterion.

A research-report rubric shows the dispatch: citation existence can be checked deterministically, experiment reproduction by execution, and claim–evidence entailment by a calibrated learned judge, while novelty and importance may remain expert territory. One rubric, four evidence channels: a rubric can therefore be a bridge up the verification hierarchy even though it is not itself a hierarchy rung.

Three layers inside every metric#

Any metric, whatever it is called, hides three design layers:

1. Criterion representation: what counts as good, such as a formal property, rubric item, acceptable-state set, reference answer, or process invariant.
2. Evidence acquisition: which observable facts bear on the criterion, such as cited passages, test results, environment changes, tool traces, or expert rationale.
3. Decision rule: how evidence becomes a verdict or score, such as a boolean check, threshold, pairwise preference, calibrated probability, or hard gate.

Formally, the path from expert judgment to decision runs:

$$J(x, y) \;\to\; (c_i) \;\to\; (e_i) \;\xrightarrow{\;g_i\;}\; z \;\xrightarrow{\;\Gamma\;}\; \delta$$

where:

• $J(x, y)$ is the expert's latent judgment of an output or trajectory $y$ on task $x$;
• $(c_i)$ are the explicit criteria $c_1, \dots, c_k$, and $(e_i)$ the evidence linked to each;
• $g_i$ is the evaluator for criterion $c_i$, each possibly a different mechanism;
• $z$ is the vector of criterion outcomes, $\Gamma$ the aggregation or gating rule, and $\delta$ the decision.

In words: tacit judgment becomes explicit criteria, each criterion gets its own evidence and evaluator, and only then do the results combine into a decision.

The intermediate layers are where expert judgment gets encoded: which criterion, which evidence, which boundary cases, which errors are fatal. Automating only the final classification does not mean the judgment has been codified well. A prompt asking a judge to rate the helpfulness of an answer from DocsAgent, the documentation agent of Case G, from 1 to 10 jumps from output to decision and leaves all three layers implicit: which claims must be grounded, whether the judge sees the documentation, and which score blocks a release.

The field is moving the same way. BinEval breaks one opaque holistic judgment into atomic yes-or-no questions and combines the answers into scores that can be inspected and debugged (Cho et al., 2026, via Chen, Wang & Qu, 2026, §3.1). Signals and resolution covers verdicts, abstention, and margins as signal forms.

The judgment compiler#

The judgment compiler is this handbook's synthesis, not terminology from the evaluator literature: an architecture for turning tacit expertise into auditable measurement. In its own metaphor, expert demonstrations and rationales are the source language; a structured rubric or specification is the intermediate representation; executable graders are the target code; and calibration and validity studies are the compiler tests. Its eleven steps:

1. Name the construct and the decision. Replace an adjective such as "high-quality answer" with a construct whose clauses can each be checked against evidence, as in Constructs, facets, and nuisances, and state which decision the score will control.
2. Elicit expert judgment before writing the rubric. On real or contrasted cases, ask experts for their verdicts and reasons, which failures are unacceptable, and what would change their minds; boundary disagreements teach more than obvious cases.
3. Decompose into atomic criteria. Break broad judgments into facets that can be evidenced independently, and hold off on an overall-quality scalar until the facets' meanings are stable.
4. Define evidence contracts. Say what evidence each criterion may use: citations for grounding, the tool trace or state transition, not the prose, for a forbidden action.
5. Push each criterion up the hierarchy. Turn whatever has checkable structure into assertions, execution, state checks, or reference comparisons, and keep learned judgment for meaning that resists them (mechanism families).
6. Encode boundary exemplars and an abstention region. Give positive and negative exemplars, adversarial near-misses, and an abstention region in which "cannot determine" is the correct output, because a forced verdict converts epistemic uncertainty into false confidence.
7. Choose decision semantics. Decide whether criteria compensate, conjoin, take priority, or gate; no amount of style should buy back a forbidden customer action (aggregation geometry).
8. Build an adjudicated calibration set. Have experts resolve edge cases and document disagreement, sampling the decision boundary rather than easy examples.
9. Validate the automated evaluator as an instrument. Measure its error by criterion and slice, its reliability, its response to perturbations, and convergent and discriminant evidence (LLM judges in practice).
10. Stress it under optimization pressure. Ask how a system could maximize the score without satisfying the construct, and red-team the evaluator before training or gating does (adaptive validity).
11. Version the judgment. Rubric text, examples, judge model, checkers, thresholds, aggregation logic, and expert policy form one measurement version; changing a load-bearing part requires revalidation or a bridging study.

In the One Spine, the compiler is the sub-procedure for steps 2–6; its first step belongs to spine step 1, and its last two reach forward into step 7, adaptive validity, and step 10, revalidation. Step 2 can itself be treated as inference over expert decisions, the subject of inverse evaluation design.

The criterion card#

A criterion card is the atomic specification for one criterion. This handbook uses one canonical card:

The DocsAgent grounding card#

DocsAgent answers users' questions from a product's documentation, and its hardest criteria are judgment-based. The card for a grounded documentation answer serves as this handbook's reference example:

Three features make it worth copying. Its evidence is claims and spans, not prose, so the judge rules on entailment rather than plausibility. Its mechanism is a portfolio: citation existence is checked deterministically, entailment is judged, and SMEs periodically audit the judge. And its blind spot sends stale documentation to the environment and the validity domain, the move Case G builds on.

The builder opens on this card, with illustrative exemplars, boundary and abstention rules, a regime, an owner, and a version added. One field does not fit: any unsupported instruction fails the criterion, but only a safety-relevant one fails the whole answer, and a single severity cannot say both. The preset marks the criterion tradeable and writes the gate into its boundary rule, a hint that the card wants splitting in two. Try switching the mechanism from Hybrid portfolio to Learned judge, and ask what the deterministic citation check was buying.

Soup, atoms, and generated rubrics#

Two design errors bracket the right grain. Rubric soup packs several logically distinct criteria into a single paragraph that one judge scores from 1 to 10, so no failure can be localized and trade-offs between criteria stay hidden. Rubric atomization overcorrects into dozens of micro-criteria that, combined, no longer reproduce how experts decide. The rule: the right decomposition is the smallest set of independently meaningful criteria that still reconstructs the expert decision with acceptable fidelity. For DocsAgent, a single vague "helpfulness" score is soup; Case G instead decomposes grading into factual correctness, citation grounding, procedural faithfulness, and abstention when evidence is absent.

Rubrics are also becoming cheap to produce: the RSI survey reports that rubrics written by models are approaching the quality of expert-written ones, and that some systems now evolve their own (Chen, Wang & Qu, 2026, §5.1). Cheap drafts call for the discipline of synthetic data: check a sample against expert judgment before any of them gates a decision. A generated rubric is a hypothesis about $\psi$; only calibration promotes it to an instrument.

From metric operator to judgment engineer#

Evaluation skill comes in four levels. A metric operator runs benchmarks; a grader engineer builds and calibrates graders; a measurement designer defines constructs, task distributions, validity arguments, and estimands; and a judgment engineer works out how experts actually decide, turns that structure into criterion, evidence, and decision artifacts, and shows where automation can and cannot stand in for them. The fourth level is the one that sets practitioners apart, and its observable product is a measurement dossier, not a clever prompt. The scarce skill is not writing a judge prompt. It is criterion engineering: discovering the hidden variables in expert judgment, operationalizing them, attaching each to observable evidence, choosing the strongest feasible verifier, and validating the remaining judgment channel.

In practice#

Run this mastery drill on a judgment-heavy task in your domain.

1. Collect real outputs. Gather 30 outputs from the system.
2. Find the disagreements. Have two experts rank or classify them independently and explain their five hardest disagreements.
3. Build criteria from the disagreements. Derive a criterion taxonomy, and write a criterion card for each criterion.
4. Climb. Write an evidence contract for each criterion, then move the criterion to the strongest mechanism able to check it.
5. Judge only the residue. Implement a learned judge only for what no stronger mechanism can check.
6. Compare three designs. Score the outputs with a holistic judge, a decomposed rubric judge, and the hybrid stack, and report nuisance invariance, per-criterion error, decision flips, and what each could be gamed on.

The artifact is a set of versioned criterion cards with a comparison report. If you can say why the hybrid stack earns more trust without pointing at its headline score, the cards are doing their job.

Common failure modes#

• Rubric soup — one 1–10 score covers several criteria, and no failure can be localized → split it into criterion cards with their own evidence and gates. See near-misses.
• Forced verdicts — every item must pass or fail, so uncertainty becomes false confidence → give each card an abstention region and an escalation route.
• An uncalibrated generated rubric — a model-drafted rubric gates a release → validate a sample against expert judgment first.
• Unversioned judgment — rubric text or examples change without a new measurement version → version the whole judgment together and revalidate. See judge rot.

Check yourself#

Analyze. Take a judge prompt your team uses and mark its three layers: criterion representation, evidence acquisition, and decision rule. Which design commitments does it hide?

Hint

Look for the evidence the judge is never shown and the threshold nobody wrote down; both are decisions someone made implicitly.

Evaluate. The DocsAgent card fails any unsupported material instruction, but fails the whole answer only for safety-relevant ones. Is that one criterion or two?

Hint

If one card needs two severities, split it, then check that the pair still reconstructs the experts' decision.

Create. Write a criterion card for DocsAgent's procedural-faithfulness criterion, and push as much of it as you can up the hierarchy.

Hint

Whether each step appears in the documented procedure may be checkable against the documentation, and whether the procedure works by running it in a sandbox. Leave to a judge only what resists both.

Further reading#

• Specification — the forms a specification can take, and why formality is not validity.
• Mechanism families — what each mechanism a card can name guarantees and estimates.
• Aggregation geometry — turning severities and gates into a decision rule.
• Inverse evaluation design — inferring criteria, weights, and thresholds from expert decisions.
• Chen, Wang & Qu (2026) — §3.1 on BinEval's binary questions and §5.1 on generated and evolving rubrics.