The One Spine

The One Spine is the framework's ten-step workflow for building an evaluator system, from stating the claim and regime to reviewing the frame. It does not replace other evaluation procedures; it puts them in the order that keeps you out of trouble.

The ten steps#

The spine runs through every layer of the evaluator system. Step 1 fixes the claim and the regime; steps 2 to 5 build the instrument, criterion by criterion; steps 6 and 7 validate it, first on ordinary outputs and then under the optimizer that will push against it; and steps 8 to 10 keep it valid in use.

The map lists the steps with the layers each one touches. Highlight Regime and only step 1 lights up; highlight Validation, then Lifecycle, to see that steps 6 to 10 belong to the spine rather than to an appendix.

Each step leaves an artifact, and together the artifacts fill most of the evaluator dossier. The DocsAgent column traces them for the documentation assistant of Case G. Its two uses follow the Studio's Case G preset.

Why claim and regime come first#

The tempting first question is which grader to use. The spine asks it fifth, because the answer depends on step 1: training may need millions of cheap evaluator calls, while certification may need hundreds of expensive independent audits. A mechanism that fits one regime can be unaffordable or unsafe in the other, which is why the selection rule reads "per criterion per regime". Verifiability engineering applies that rule.

The regime also decides what steps 6 and 7 have to show. Evidence from ordinary outputs does not carry over to an optimize use, because a judge calibrated on ordinary model outputs may become badly miscalibrated on outputs adversarially selected to maximize the judge. Take DocsAgent's grounding judge. As the release gate, it needs calibrated error rates against SME labels. In daily prompt tuning, developers read its rationales on the dev set and keep whatever raises its score, so the same judge is now an optimization target, and its evidence must include an exploitability curve run with that tuning loop. Writing the uses down at step 1 also exposes the channel risks that step 8 must control: if the release set is ever reused for tuning, some of its independence is spent.

The discipline fits in four words: locate first, instrument second. Steps 1 and 2 locate, and steps 3 to 5 instrument. The order guards against a failure that no later step can repair: a perfect grader of task success cannot repair a task that measures the wrong capability.

One spine, five workflows#

Four other workflows overlap with the spine, and none contradicts it once you see which steps each one covers.

• The expert workflow has ten design steps, from writing the claim to monitoring drift.
• The judgment compiler, a synthesis this handbook proposes, has eleven steps that compile tacit expert judgment into executable evaluation. It is the sub-procedure for spine steps 2 to 6, and Rubrics and criterion cards covers it in full.
• The living-suite loop has nine operational steps, from collecting real traces to feeding new failures back into the suite.
• The green-agent checklist of the Berkeley Agentic AI course's evaluation lecture (Song, 2025, slides 81–85) has four steps for building a green agent, the course's name for the hosting agent that serves as the evaluator (slides 65–68): choose the task, design the environment, design the metrics, and design test cases with ground-truth results.

In the table, numbers are each workflow's own step numbers, and a dash means the workflow has no step for that part of the spine.

Four patterns stand out. Only the spine fixes the regime before the instrument; the expert workflow decides the use at its seventh step, and the living-suite loop's seventh step, gating releases, is the measure regime in operation rather than a design decision. Steps 4 and 8 each appear in only one other workflow, and step 7 in two that stop at red-teaming and falsification search, while the spine asks for a stress test with the actual optimizer. Frame review appears in no other column: the spine adds it to step 10 for the case when metrics improve but real outcomes do not. And the shortest workflow holds step 6 in miniature: test cases of agents that succeed and fail in different ways, each with its ground-truth result and "as many edge cases as possible", used to check whether the evaluator itself gives reliable results (Song, 2025, slide 85).

How the Studio walks the spine#

The Studio implements the spine as a fixed graph of agents and code, with you at two checkpoints. Each stage below names the spine steps it covers.

1. Frame interview (step 1). An interviewer agent elicits the claim, the trust root, and every use of the signal with its consumer and optimizer.
2. Checkpoint A (step 1). You approve the claim and regime before anything is designed.
3. Criteria (steps 2–3). A specifier decomposes the construct into facets, nuisances, and atomic criteria with gates and thresholds.
4. Checkpoint B (step 3). You edit and approve the criteria.
5. Instruments (steps 3–5). One criterion engineer per criterion, all running in parallel, designs the evidence view, the reference anchor, and the strongest affordable mechanism for each channel the regime uses.
6. Validation and lifecycle plans (steps 6–10). Two planners read the finished criteria: one plans reliability, calibration, and adaptive validity; the other plans channels, tripwires, revalidation, and frame review. Sample sizes and bounds are then computed in code, never generated by a model.
7. Review. A deterministic audit, whose rules read only the dossier's structured fields, runs alongside an independent critic that reads only the composed dossier and never sees the audit's findings. Findings the agents can fix get one round of targeted revisions, followed by a fresh review.
8. Dossier. Code composes the 15-item minimum reporting checklist, a validity statement for each use of the signal, and the known limitations. Each statement lists the profile components that are planned rather than measured.

Review is not a spine step: it audits what the spine produced, as Auditing an eval does for someone else's evaluator.

Three things stay human by design. The two checkpoints turn drafts into decisions: the agents propose the claim, the regime, and the criteria, and you approve them. Findings triage is yours: you accept a finding as a known risk or dismiss it with a note; findings only you can resolve, such as a wrong construct, wait for you; and accepted risks and unresolved serious findings go into the dossier's known limitations. And the trust root is yours to name, because every validity statement is relative to it: it is where the evaluate-the-evaluator recursion stops.

In practice#

Run the spine on paper before you build anything.

1. Write the claim and list every use. For each use, name the consumer, the optimizer, the exposure, and what the optimizer can see and modify. A use with no decision attached is eval theater: name the decision or drop the use.
2. Decompose and specify. List facets and nuisances, choose the aggregation geometry, and write a criterion card per criterion, using the judgment compiler wherever expert judgment has to be codified.
3. Instrument each criterion. Write its evidence contract, choose the strongest affordable mechanism for each regime that consumes it, and record the hierarchy family as a prior.
4. Plan validation before you collect data. Size the natural-error and reliability studies, and plan an exploitability curve with the actual optimizer for every optimize use.
5. Separate the channels and write the tripwires. Decide what each channel may see, start an anchor-burn ledger, and state which divergences trigger action.
6. Write the revalidation triggers and the frame-review policy, naming who may revise the construct.
7. Write one validity statement per use, marking every profile component that is planned rather than measured.
8. Re-enter at step 1 when a use changes. A judge promoted from reporting to selection is in a new regime, and steps 5 to 8 must be revisited for it.

The artifact is an evaluator dossier with one validity statement per use, the same object the Studio exports; Templates has its 16 fields.

Common failure modes#

• Mechanism first — the design starts with "should we use an LLM judge?" and the claim is reverse-engineered from the grader → write the claim and every use first, then choose a mechanism per criterion. See eval theater.
• Regime decided last — a judge calibrated for reporting is promoted to selection or tuning on its old agreement numbers → state every use at step 1, and revisit steps 5 to 8 when a use changes. See Goodharting.
• A spine that stops at step 6 — the evaluator was validated once and never maintained: no channels, no ledger, no revalidation after the judge changed → treat steps 7 to 10 as part of the build. See judge rot.
• Frame never reviewed — scores keep rising while user outcomes stall, and the team tunes harder → trigger frame review when metrics improve but real outcomes do not. See frame lock-in.

Check yourself#

Analyze. Take a public benchmark's paper and mark, for each of the ten steps, whether the paper documents its artifact. Where is the weakest link?

Hint

Start at step 1: is a decision named, and are the uses of the score listed? Then check step 7 for every optimize use and step 8 for a held-out set kept out of tuning. The earliest missing artifact is usually the weakest link, because every later step inherits it.

Evaluate. A team skips step 7 for its grounding judge because "it agrees with our SMEs and nobody trains against it." Developers tune prompts against its scores every day. Is skipping step 7 defensible?

Hint

Hill-climbing is optimization, so daily prompt tuning is an optimize use. Agreement on ordinary outputs says nothing about outputs selected to raise the judge's score; run the tuning loop against the judge and score the results with an independent channel.

Create. Write your team's version of the spine as a release checklist: for each step, the artifact, its owner, and the event that sends you back to it.

Hint

Start from the artifact table. Owners come from step 1 (the consumer and the trust root) and step 10 (the frame-review authority); the return events are the revalidation triggers plus any change in how the signal is used.

Further reading#

• Rubrics and criterion cards — the judgment compiler, the sub-procedure for steps 2 to 6.
• Verifiability engineering — step 5 in depth: the strongest affordable mechanism per criterion per regime.
• The validation profile and validity statement — what steps 6 and 7 measure, and how a statement is scoped.
• Case studies — the spine applied end to end, with a Studio preset for each case.
• Song (2025), "Agent evaluation & project overview," UC Berkeley lecture slides — slides 81–85 for the green-agent checklist.