Vocabulary without category errors

An evaluator is any mechanism or artifact that produces a quality signal. On its own the word is underspecified. Reward model, rubric, PRM, RLHF, and verifier are not five kinds of evaluator: they describe different axes of one, and most confusions disappear once each term is put on the axis it describes.

Evaluator, verifier, judge#

The RSI survey fixes the three most common words: an evaluator maps a candidate artifact to a quality signal, whether it is a proof checker, test suite, reward model, LLM judge, rubric, or human rater; a verifier is an evaluator with soundness guarantees; a judge is a learned or prompted evaluator without them (Chen, Wang & Qu, 2026, §2.1).

This handbook enforces those senses. In loose LLM usage "verifier" often means a deterministic or execution checker, which blurs the difference between accepted, so true and no tested counterexample found. Take SQLAgent, the text-to-SQL agent of Case C. A team may call its result-set comparator its verifier, but the comparator executes a query against one pinned snapshot and compares rows with a reference anchor; a pass means the rows matched there, which is why SQLAgent's evaluator adds metamorphic tests and multiple database instances. It is a strong execution checker, not a verifier.

Whether a rubric is an evaluator, a process reward model a verifier, or RLHF a kind of eval can each be answered yes or no, depending on which role is meant. These are one-dimensional questions about a multi-dimensional design space. Treating objects that live on different axes, such as a specification, a mechanism, and a training pipeline, as peers on one list is a category error. The evaluator system walks through six such objects in a single evaluation.

Two maps, never merged#

This handbook organizes the vocabulary around two pictures that must be kept apart. The verification hierarchy tells you how strong the evidence channel is; the evaluator design stack tells you how that channel was constructed.

• The hierarchy orders evidence channels — formal verifiers, execution feedback, learned judges, intrinsic signals — with human judgment outside the ladder. It is not a catalog of evaluator artifacts. The verification hierarchy, demoted covers it in full.
• The construction stack asks how an evaluation was built and what its signal does: construct, specification, evaluator mechanism, signal, use. Resolution and deployment context cut across every stage.

The confusion starts in the survey itself, which does two jobs at once. It inventories an evaluator design space — process supervision, LLM judges, reward models, rubrics, self-verification, meta-evaluation — and then collapses it into the coarser hierarchy (Chen, Wang & Qu, 2026, §5.1–5.2). The inventory asks what researchers build; the hierarchy asks how much trust the resulting signal deserves under optimization. A reward model and an LLM judge are separate designs in the inventory and members of one family in the hierarchy.

Separating the pictures resolves two standing confusions. A rubric belongs at the specification stage, and something else must apply it: one research-report rubric can send citation resolution to a deterministic check, claim–evidence entailment to a calibrated judge or an expert, and the importance of the research direction to human judgment. RLHF belongs at the use stage: the reward model is the mechanism, its scalar is the signal, policy optimization is the use, and "RLHF" names the whole pipeline. Reading the rubric as a specification is this handbook's convention; the survey's broad "evaluator" includes rubrics (Chen, Wang & Qu, 2026, §2.1).

A type, not a noun#

Describe an evaluator with a tuple rather than a noun:

$$\mathcal{E} = \langle C, \Psi, M, Z, R, U \rangle$$

where:

• $C$ is the construct, the latent property that matters;
• $\Psi$ is the specification, the observable demand that operationalizes it;
• $M$ is the mechanism that computes the judgment;
• $Z$ is the signal that leaves the evaluator;
• $R$ is the resolution, the unit that is judged;
• $U$ is the use, the causal role the signal plays.

In words: an evaluator's six-axis type says what it is about, what it demands, what computes the verdict, what comes out, at what grain, and what the output is for. The framework distributes the type over the layers of $\mathcal{E}_{\text{sys}}$:

The instrument also gains three coordinates the six axes leave implicit — the reference anchor $A$, the evidence view $V$, and the aggregation geometry $\Gamma$ — giving $\mathcal{I} = \langle \Psi, A, V, M, Z, R, \Gamma \rangle$. They are not details: an LLM judge shown only the final answer is a different instrument from the same judge shown tool traces or hidden state.

Seven worked classifications#

Seven familiar evaluators, classified on the axes:

Read down the columns, not across the names. The math "verifier" is an execution or deterministic checker unless it carries a soundness argument. The PRM differs from the preference reward model in resolution, not in rung. And the framework splits the first row further: the answer key is the reference anchor, and the specification is that the selected option matches the key, as the MMLU evaluator card records.

The compact rule#

Most collisions disappear once each term is assigned to the axis it primarily describes:

• Reward model, LLM judge, learned PRM mostly name a mechanism.
• Rubric mostly names a specification.
• Process versus outcome names a resolution.
• RLHF, best-of-N, release gating name a use.
• The verification hierarchy rates the epistemic strength of the evidence channel.

Apply it to a methods sentence such as "we used a PRM as a verifier for best-of-16": the PRM is a learned mechanism at step resolution, best-of-16 is a selection use, and "verifier" claims a soundness guarantee no learned model has. The full set of collisions:

Signals and resolution explains why a PRM score, a value estimate, and a robustness margin are different objects.

Raschka's four approaches, re-expressed#

Raschka (2025) maps practical LLM evaluation into four approaches — multiple-choice benchmarks, verifiers, leaderboards, and LLM judges — which an earlier version of this site taught as "the four evaluation methods". The map answers a different question from the hierarchy, and its categories mix response format, evidence mechanism, aggregation protocol, and grader implementation:

Because the approaches name different axes, one benchmark can belong to two of them. AlpacaEval is a leaderboard by aggregation and an LLM judge by mechanism, since its pairwise preferences come from a GPT-4 Turbo judge; the earlier page listed it under both human preference and LLM-as-a-judge. Chatbot Arena aggregates crowdsourced human votes, so it measures what its voters prefer, not automatically factual correctness, safety, or expert utility.

In practice#

Write a type block for every evaluator in your reports.

1. List the nouns your team uses: "the verifier", "the RM", "the rubric", "the PRM", "the judge".
2. Apply the compact rule. Write the axis each noun names; a noun that names a use or a resolution does not name an evaluator.
3. Write the type. Fill in $C$, $\Psi$, $M$, $Z$, $R$, and $U$, then add the reference anchor, evidence view, and aggregation.
4. Record the hierarchy family next to the mechanism, as a prior that measurement will replace.
5. Fix the words. Keep "verifier" where you can state the soundness argument; call people raters or panels and rubrics specifications.
6. Version the use. When a signal moves from reporting to selection, training, or gating, record a change of $U$ and of regime, not a new tool.

The artifact is a one-line type block per evaluator — for DocsAgent, the documentation agent of Case G, ⟨grounded answers, criterion cards, citation checks plus an entailment judge, binary verdict per criterion, final answer, release gate⟩ (illustrative) — which fills the instrument and regime rows of the coordinate sheet in The evaluator system.

Common failure modes#

• Ranking nouns from different axes — "we moved up from a reward model to a PRM" → a PRM differs in resolution, not rung; compare measured error, not names. See The verification hierarchy, demoted.
• A rubric credited as a verifier — a specification is given a mechanism's guarantees → name the mechanism that applies each criterion and measure its error. See Rubrics and criterion cards.
• A pipeline cited as evidence — "it was trained with RLHF, so it is aligned" → RLHF is an optimization pipeline, not evidence that the final policy is good; evaluate the policy with an instrument the pipeline did not optimize. See Evaluators inside training loops.
• One label, a new use — the same "judge" moves from reporting to best-of-N to RL reward under one name → record $U$ in every report. See Goodharting.

Check yourself#

Analyze. Find a paper that calls something a "verifier". Which axis is the term describing, and does the paper state a soundness argument?

Hint

Ask what a pass guarantees. If it means "no tested counterexample was found" or "a model rated it acceptable", the word names an execution, deterministic, or learned mechanism.

Evaluate. A team writes: "We replaced our LLM judge with a reward model trained on our raters' preferences, so the evaluator is now more rigorous." Evaluate the claim axis by axis.

Hint

Both are learned judges, so the mechanism changed within one family and the rung did not. Rigor comes from measured error against adjudicated labels, by slice, on the outputs it will see, re-measured if it becomes a reward.

Create. Invent a compact notation for every eval report that makes the six axes, the reference anchor, the evidence view, and the aggregation explicit, and write it out for one evaluator you use.

Hint

One line per evaluator works if every slot is always filled, even with "none" or "not yet decided": an empty slot marks a design decision someone skipped.

Further reading#

• The verification hierarchy, demoted — the first map in full, and why gold answers and rubrics are not rungs.
• Mechanism families — what each mechanism can guarantee and what it can only estimate.
• Crosswalk — older vocabulary translated into coordinates.
• Raschka (2025), "Understanding the 4 Main Approaches to LLM Evaluation (From Scratch)" — the practical four-way map, read here on the axes.