# Exercises and drills

> Analyze, Evaluate, Create: capstone prompts and study drills that test whether the ontology has stuck.

These exercises test whether the framework has stuck. They are grouped by the part of the handbook that teaches each idea. Each links to its chapter and hides a hint with the key idea.

**Analyze** asks you to take an evaluation apart, **Evaluate** to judge a design or a claim, and **Create** to design one; a **Drill** is a longer exercise you run on real outputs. Prompts that already appear in a chapter's Check yourself section are not repeated here: the [table at the end](#check-yourself-by-chapter) links to every chapter's set.

## Foundations

**Analyze.** Take the phrase "process reward model" and name the mechanism, the signal, the resolution, and the use it implies. See [Vocabulary without category errors](https://benchmaker.ai/handbook/foundations/vocabulary).

> [!DETAILS] Hint
> Apply the compact rule word by word. "Reward model" names a learned-judge mechanism, its signal is a score per step, and "process" names the resolution. The use is whatever consumes the score: diagnosis, search, or a training reward. None of these makes it a rung of the hierarchy.

**Analyze.** Explain this handbook's framework using only the two mental pictures, the verification hierarchy and the evaluator construction stack. See [Vocabulary without category errors](https://benchmaker.ai/handbook/foundations/vocabulary).

> [!DETAILS] Hint
> The hierarchy asks how strongly a signal is grounded; the construction stack asks how the evaluation was built and what causal role its signal plays, with resolution cutting across. Place each layer of the evaluator system on one picture or the other, and note which layers neither picture covers.

## Claim

**Evaluate.** When is human preference the construct you intend to measure, and when is it only a proxy for something else? See [Constructs, facets, and nuisances](https://benchmaker.ai/handbook/claim/constructs).

> [!DETAILS] Hint
> Preference is the construct when the claim is about what a defined population of people prefers. When the claim is about factual correctness, safety, or expert utility, preference is a proxy, and the sampled people, prompts, interface, and aggregation model all need evidence that they track the real construct.

**Drill.** Take a rubric for an LLM judge. Design its calibration set and specify the replication universe. Decide which costs more in your application, a false pass or a false fail, and how the answer should move the judge's threshold and change how you report uncertainty. See [Decision claims](https://benchmaker.ai/handbook/claim/decision-claims).

> [!DETAILS] Hint
> The replication universe lists the conditions over which the score must stay stable, such as task forms, rollouts, judge runs, and environment snapshots. Moving the threshold trades one error for the other rather than reducing both, so the costlier error sets the operating point; report error rates on each side of it, with intervals.

## Instrument

**Create.** Design a hybrid evaluator stack for one rubric in which one criterion is checked formally, one by execution, and one by a learned judge. See [Rubrics and criterion cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards).

> [!DETAILS] Hint
> One rubric can feed several evidence channels: in a research report, citations that resolve are checked deterministically, claims entailed by their evidence need a calibrated judge or an expert, and the importance of the research direction stays with people. For each criterion, record what the check rests on: a trusted base, an anchor, or a calibration set.

**Evaluate.** When is a test suite strong enough for development feedback but too weak for certification? See [Mechanism families](https://benchmaker.ai/handbook/instrument/mechanisms).

> [!DETAILS] Hint
> A test suite samples behavior: passing establishes behavior on the tested inputs, not the whole domain, and an optimizer can exploit untested regions. Development needs dense, fast feedback and tolerates that gap; certification needs independence from what developers iterated against and coverage of the claim's population.

**Create.** Turn one outcome-only coding evaluation into a mixed outcome and process evaluation. See [Signals and resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution).

> [!DETAILS] Hint
> Keep tests on the final repository state as the outcome criterion. Add process criteria only where the claim includes the path, such as approvals observed before merge or forbidden files left untouched, and check them as ordering constraints over the event trace, not as one canonical sequence of commands.

**Analyze.** Explain why "learned judge", like "neural network", is a family label rather than a complete description of a mechanism. See [Mechanism families](https://benchmaker.ai/handbook/instrument/mechanisms).

> [!DETAILS] Hint
> Members share an empirical validity domain: their mapping from evidence to judgment is learned, so their error must be measured. They differ in what they were trained to do and what they emit: a prompted LLM judge, a reward model trained on preferences, a narrow classifier, a PRM working on steps.

**Evaluate.** Under what conditions could a specialized reward model, or a small specialized judge, deserve more trust than a much larger general-purpose LLM judge? See [LLM judges in practice](https://benchmaker.ai/handbook/instrument/llm-judges).

> [!DETAILS] Hint
> Trust follows measured error on the outputs the judge will grade, not model size. Compare the judges' error rates against adjudicated labels on the target distribution, by slice, their transport across generators and time, and their coupling to the generator's model family.

**Analyze.** Give one formal, one execution-based, one learned, and one human process evaluator. See [Signals and resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution).

> [!DETAILS] Hint
> Resolution cuts across mechanism families. Start from process properties such as every database write being authorized or every tool call being schema-valid, and ask which mechanism could check each one. Of the four you name, only a learned step scorer is a PRM.

**Evaluate.** When does process supervision end up measuring conformity to a preferred method rather than correctness? See [Signals and resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution).

> [!DETAILS] Hint
> When the method is not part of the construct and several legitimate strategies exist. Grading against one canonical chain then penalizes valid novel solutions, the anti-pattern called rigid trajectory grading; judge the observable constraints the claim names instead.

**Drill.** Collect 30 real outputs of a judgment-heavy task. Have two experts rank or classify them independently and explain their five hardest disagreements, then build criteria and evidence contracts from the disagreements, move each criterion to the strongest mechanism that can check it, and implement a learned judge only for the residue. Compare a holistic judge, a decomposed rubric judge, and the hybrid stack. See [Rubrics and criterion cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards).

> [!DETAILS] Hint
> The chapter's In practice procedure walks through the drill. Report nuisance invariance, per-criterion error, decision flips, and what each design could be gamed on. You have done it well if you can say why the hybrid stack deserves more trust without pointing at its headline score.

## Regime

**Evaluate.** Which confusions arise when a training pipeline and an evaluator are treated as the same object? See [Evaluators inside training loops](https://benchmaker.ai/handbook/regime/training-loops).

> [!DETAILS] Hint
> RLHF, RLAIF, RLVR, and DPO are pipelines that consume evaluator signals. Naming the pipeline as the evaluator hides the evaluator inside it, which needs its own validity argument, and the pipeline's name says nothing about whether the final policy is good.

**Evaluate.** When does a single label hide a dangerous change in use? See [Use regimes](https://benchmaker.ai/handbook/regime/use-regimes).

> [!DETAILS] Hint
> When "the judge" or "the benchmark" keeps its name while its signal moves from reporting to selection, training, or gating. The mechanism is unchanged, but the use and the regime are not, and neither is the evidence the signal now needs.

## Validation

**Drill.** Pick a benchmark you trust and write down the exact score interpretation you believe. List one piece each of content, response-process, convergent, discriminant, criterion, and reliability evidence that would be required to defend it. See [Sources of validity evidence](https://benchmaker.ai/handbook/validation/validity-evidence).

> [!DETAILS] Hint
> The chapter's evidence plan starts from this interpretation. For each source, name an observation that could come out against you, such as a facet with too few tasks, a shortcut route to success, a score that tracks answer length, or a weak relation to the outcome users care about.

## Validity under pressure

**Evaluate.** Why does independent holdout evaluation matter more once optimization pressure increases? See [Adaptive validity](https://benchmaker.ai/handbook/pressure/adaptive-validity).

> [!DETAILS] Hint
> Optimization moves outputs toward wherever the evaluator is wrong, so error rates measured on ordinary outputs no longer describe the outputs being scored. Only an anchor the optimizer can neither see nor modify can measure the divergence, $\varepsilon(b)$, as exposure grows.

## Lifecycle

**Create.** Design an evaluator ensemble that uses two learned-judge mechanisms with intentionally different failure modes. See [Channels, independence, and anchor burn](https://benchmaker.ai/handbook/lifecycle/channels-and-independence).

> [!DETAILS] Hint
> Diversify what the judges share: model family, training data, prompts, and the evidence they see. Majority vote reduces independent noise only, so measure the joint miss rate on adjudicated failures to find out whether the holes line up.

## Agents and self-improvement

**Analyze.** Why can an agent reach the correct final state through an unacceptable trajectory? See [Evaluating agents](https://benchmaker.ai/handbook/agents/evaluating-agents).

> [!DETAILS] Hint
> The end state can be right after a forbidden intermediate action, such as a destructive call followed by a repair, an unauthorized read, or a message sent before confirmation. Grade the trajectory constraints the claim names alongside the end state.

**Create.** Design a four-channel evaluator stack for an enterprise SQL agent. See [Evaluating agents](https://benchmaker.ai/handbook/agents/evaluating-agents).

> [!DETAILS] Hint
> Formalize the access policy, execute the semantic consequences on pinned snapshots, reserve learned judgment for the residual, and keep people for adjudication and policy changes. Then compare your design with [Case C](https://benchmaker.ai/handbook/cases/sql-data-agent).

## Practice

**Create.** Redesign one agent permission rule so that the unsafe action becomes unreachable by construction. See [Guarantee before you estimate](https://benchmaker.ai/handbook/practice/structural-guarantees).

> [!DETAILS] Hint
> Remove the capability instead of detecting its misuse: scope the credentials, allowlist the destinations, or make the tool refuse to run without a confirmation token. State the guarantee relative to the configured trusted base, and take the property out of the sampling budget.

**Analyze.** Decompose "good legal research" into criteria and classify each by the strongest evidence channel available for it. See [Verifiability engineering](https://benchmaker.ai/handbook/practice/verifiability-engineering).

> [!DETAILS] Hint
> Split the construct into atomic criteria a domain expert could adjudicate, then ask the strongest-oracle questions in order for each: can it be verified formally, checked by execution or a deterministic rule, judged by a calibrated learned judge, or must an expert decide? Provenance, entailment, and importance usually land on different rungs.

**Analyze.** Why is pairwise preference learning already shaped like inverse reinforcement learning, even when no decision process is written down? See [Inverse evaluation design](https://benchmaker.ai/handbook/practice/inverse-evaluation-design).

> [!DETAILS] Hint
> A preference between two answers is a choice. With two options, the Boltzmann choice model reduces to the Bradley–Terry form used to train reward models, so fitting a reward model to pairs infers a latent judgment from behavior.

**Create.** Build an evaluator dossier template for your team, and require every future evaluation to fill it in. See [Templates](https://benchmaker.ai/handbook/reference/templates).

> [!DETAILS] Hint
> Start from the sixteen fields and the validity statement. For each field, decide what counts as filled, such as an artifact with an owner and a version, and n with an interval for every measured number, and make "planned" a legitimate value that must be marked.

## Capstones

**Analyze.** For each case study, locate every evaluator on both pictures: the verification hierarchy and the construction stack. See the [case studies](https://benchmaker.ai/handbook/cases).

> [!DETAILS] Hint
> Give each mechanism its hierarchy family, with people outside the ladder, then its place in the stack: specification, mechanism, signal, use, and resolution. A case built as a portfolio places its evaluators on several rungs at once.

**Evaluate.** Which case study has the largest gap between construct and specification, and which the largest mechanism error? See the [case studies](https://benchmaker.ai/handbook/cases).

> [!DETAILS] Hint
> Compare how much of each construct the specification captures, which shows up in what each validity statement leaves to human judgment, with how much of each specification a learned judge has to apply. The two gaps need not peak in the same case.

**Create.** Add a case from your own domain and design a criterion-by-criterion evaluator portfolio for it. See the [case studies](https://benchmaker.ai/handbook/cases).

> [!DETAILS] Hint
> Follow the cases' template: claim and decision, regimes, a table giving each criterion a mechanism, hierarchy family, evidence view, and anchor, a validation plan as entries of $\Pi$, channels and tripwires, and a validity statement written as a plan. The [Studio](https://benchmaker.ai/studio) walks through the same steps.

**Create.** Design an evaluator for an open-ended domain, such as the quality of research, decision support in an enterprise, or the generation of scientific hypotheses. Begin with expert judgment, decompose it into criteria, instrument the artifacts so they expose evidence, compile the verifiable criteria upward, retain the judgment that cannot be compiled, and separate the development, certification, and production channels. Then write its validity statement, including the optimizer regime, the access surface, the tested adaptive-validity budget, and the frame-review triggers. See [The One Spine](https://benchmaker.ai/handbook/practice/one-spine).

> [!DETAILS] Hint
> The spine's ten steps give the order, and the judgment compiler covers its middle steps. In the statement, mark $b^{*}$ and every component of $\Pi$ measured or planned, and close with what remains human judgment.

Other capstone prompts appear as Check yourself items in [Evaluation is measurement](https://benchmaker.ai/handbook/foundations/evaluation-is-measurement#check-yourself), [The evaluator system](https://benchmaker.ai/handbook/foundations/evaluator-systems#check-yourself), and [Use regimes](https://benchmaker.ai/handbook/regime/use-regimes#check-yourself).

## One-page study drill

Answer these ten questions without looking back. If you can, the ontology is probably stable.

**1.** What do reward models and LLM judges share that makes both learned judges, and what makes them different mechanisms? See [Mechanism families](https://benchmaker.ai/handbook/instrument/mechanisms).

> [!DETAILS] Hint
> Both learn their mapping from evidence to judgment, so both have empirical error that must be measured. A reward model is trained to map candidates to scalar or pairwise preference estimates and is often optimized against; an LLM judge is a general model prompted to apply a rubric to evidence.

**2.** Why does a PRM not get a rung of its own in the verification hierarchy? See [Signals and resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution).

> [!DETAILS] Hint
> A PRM is a learned judge at process resolution. Process versus outcome is a resolution, and resolution cuts across the hierarchy.

**3.** How can one rubric be applied at several levels of the hierarchy? See [Rubrics and criterion cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards).

> [!DETAILS] Hint
> A rubric is a specification. The rung belongs to the mechanism that applies each criterion, and one rubric's criteria can go to different mechanisms.

**4.** Why does training with RLHF not, by itself, show that the final policy is good? See [Evaluators inside training loops](https://benchmaker.ai/handbook/regime/training-loops).

> [!DETAILS] Hint
> RLHF is a pipeline. Its reward model learns an ordering, not a proof of quality, and the policy is optimized against that model's blind spots; evidence about the policy needs an independent holdout.

**5.** What changes when an evaluator becomes a training reward rather than a reporting metric? See [Use regimes](https://benchmaker.ai/handbook/regime/use-regimes).

> [!DETAILS] Hint
> The regime moves from measure to optimize. An optimizer now searches for the evaluator's errors, so accuracy measured before optimization no longer covers the outputs it scores.

**6.** Why is the choice between process and outcome independent of the choice among formal, execution, learned, and intrinsic mechanisms? See [Signals and resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution).

> [!DETAILS] Hint
> One axis is the unit judged; the other is how the judgment is grounded. A process check can be a deterministic trajectory checker or a learned PRM, and an outcome check can be a test suite or a judge.

**7.** What does Marr's computational level ask that no better judge can answer? See [Marr's levels for evaluation](https://benchmaker.ai/handbook/foundations/marr-levels).

> [!DETAILS] Hint
> Which property, over which population, for which decision. A more capable grader works at the algorithmic and implementation levels; it cannot repair a badly posed question.

**8.** State the strongest-oracle-per-criterion rule. See [Verifiability engineering](https://benchmaker.ai/handbook/practice/verifiability-engineering).

> [!DETAILS] Hint
> For each criterion, use the strongest evidence channel that can check it, and do not use a learned judge where an executor can answer. The framework amends it to the strongest affordable mechanism per criterion per regime, deliberately different across development and certification.

**9.** How can inverse reinforcement learning help discover a specification without providing formal verification? See [Inverse evaluation design](https://benchmaker.ai/handbook/practice/inverse-evaluation-design).

> [!DETAILS] Hint
> It infers a distribution over criteria, weights, and thresholds from expert behavior, which shows where to ask the experts next. It cannot turn behavior into a proof obligation, or validate the construct from inside the preference data.

**10.** Draw the two mental pictures from memory and say when to use each. See [Vocabulary without category errors](https://benchmaker.ai/handbook/foundations/vocabulary).

> [!DETAILS] Hint
> The hierarchy orders evidence channels by grounding (formal verification, execution feedback, learned judges, intrinsic signals, with human judgment as the external frontier) and asks how strong the evidence is. The construction stack runs construct, specification, mechanism, signal, use, with resolution cutting across, and asks how the evaluation was engineered and what its signal does.

## Check yourself by chapter

Every chapter ends with three prompts of its own, tagged Analyze, Evaluate, and Create:

| Part | Chapters |
|---|---|
| Foundations | [Evaluation is measurement](https://benchmaker.ai/handbook/foundations/evaluation-is-measurement#check-yourself) · [The evaluator system](https://benchmaker.ai/handbook/foundations/evaluator-systems#check-yourself) · [Vocabulary](https://benchmaker.ai/handbook/foundations/vocabulary#check-yourself) · [Verification hierarchy](https://benchmaker.ai/handbook/foundations/verification-hierarchy#check-yourself) · [Marr's levels](https://benchmaker.ai/handbook/foundations/marr-levels#check-yourself) |
| Claim | [Constructs](https://benchmaker.ai/handbook/claim/constructs#check-yourself) · [Validity domain](https://benchmaker.ai/handbook/claim/validity-domain#check-yourself) · [Decision claims](https://benchmaker.ai/handbook/claim/decision-claims#check-yourself) · [Task distributions](https://benchmaker.ai/handbook/claim/task-distributions#check-yourself) |
| Instrument | [Specification](https://benchmaker.ai/handbook/instrument/specification#check-yourself) · [Rubrics and criterion cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards#check-yourself) · [Anchors and evidence views](https://benchmaker.ai/handbook/instrument/anchors-and-evidence-views#check-yourself) · [Mechanism families](https://benchmaker.ai/handbook/instrument/mechanisms#check-yourself) · [LLM judges](https://benchmaker.ai/handbook/instrument/llm-judges#check-yourself) · [Signals and resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution#check-yourself) · [Aggregation geometry](https://benchmaker.ai/handbook/instrument/aggregation-geometry#check-yourself) |
| Regime | [Use regimes](https://benchmaker.ai/handbook/regime/use-regimes#check-yourself) · [Exposure and surfaces](https://benchmaker.ai/handbook/regime/exposure-and-surfaces#check-yourself) · [Training loops](https://benchmaker.ai/handbook/regime/training-loops#check-yourself) · [Production monitoring](https://benchmaker.ai/handbook/regime/production-monitoring#check-yourself) |
| Validation | [Validity argument](https://benchmaker.ai/handbook/validation/validity-argument#check-yourself) · [Task and outcome validity](https://benchmaker.ai/handbook/validation/task-and-outcome-validity#check-yourself) · [Validity evidence](https://benchmaker.ai/handbook/validation/validity-evidence#check-yourself) · [Reliability](https://benchmaker.ai/handbook/validation/reliability#check-yourself) · [Judge as a noisy sensor](https://benchmaker.ai/handbook/validation/judge-as-noisy-sensor#check-yourself) · [Estimands and intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals#check-yourself) · [Comparing systems](https://benchmaker.ai/handbook/validation/comparing-systems#check-yourself) · [Rare events](https://benchmaker.ai/handbook/validation/rare-events#check-yourself) |
| Validity under pressure | [Falsification](https://benchmaker.ai/handbook/pressure/falsification#check-yourself) · [Adaptive validity](https://benchmaker.ai/handbook/pressure/adaptive-validity#check-yourself) · [Goodhart and self-confirming loops](https://benchmaker.ai/handbook/pressure/goodhart-and-self-confirming-loops#check-yourself) · [Validation profile](https://benchmaker.ai/handbook/pressure/validation-profile#check-yourself) |
| Lifecycle | [Channels and independence](https://benchmaker.ai/handbook/lifecycle/channels-and-independence#check-yourself) · [Living eval suites](https://benchmaker.ai/handbook/lifecycle/living-eval-suites#check-yourself) · [Harness and reproducibility](https://benchmaker.ai/handbook/lifecycle/harness-and-reproducibility#check-yourself) · [Frame review](https://benchmaker.ai/handbook/lifecycle/frame-review#check-yourself) |
| Agents and self-improvement | [Evaluating agents](https://benchmaker.ai/handbook/agents/evaluating-agents#check-yourself) · [Agent benchmarks](https://benchmaker.ai/handbook/agents/agent-benchmarks#check-yourself) · [Self-improvement](https://benchmaker.ai/handbook/agents/self-improvement#check-yourself) |
| Practice | [One Spine](https://benchmaker.ai/handbook/practice/one-spine#check-yourself) · [Verifiability engineering](https://benchmaker.ai/handbook/practice/verifiability-engineering#check-yourself) · [Structural guarantees](https://benchmaker.ai/handbook/practice/structural-guarantees#check-yourself) · [Inverse evaluation design](https://benchmaker.ai/handbook/practice/inverse-evaluation-design#check-yourself) · [Auditing an eval](https://benchmaker.ai/handbook/practice/auditing-an-eval#check-yourself) · [Anti-patterns](https://benchmaker.ai/handbook/practice/anti-patterns#check-yourself) |
| Case studies | [A: Math](https://benchmaker.ai/handbook/cases/math-reasoning#check-yourself) · [B: Coding](https://benchmaker.ai/handbook/cases/coding-agent#check-yourself) · [C: SQL](https://benchmaker.ai/handbook/cases/sql-data-agent#check-yourself) · [D: Research](https://benchmaker.ai/handbook/cases/research-agent#check-yourself) · [E: Training](https://benchmaker.ai/handbook/cases/reasoning-model-training#check-yourself) · [F: Safety monitor](https://benchmaker.ai/handbook/cases/production-safety-monitor#check-yourself) · [G: Docs](https://benchmaker.ai/handbook/cases/docs-agent#check-yourself) |

---
Source: https://benchmaker.ai/handbook/reference/exercises