Reference

Crosswalk

7 min readDraftUpdated 2026-10-13

This crosswalk translates evaluation vocabulary you already know into the framework's coordinates: practice terms such as golden datasets, LLM judges, partial credit, and release gates; the evidence that agent-benchmark validity concepts call for; the six-axis evaluator type; Raschka's four approaches; and the three-level framework this site used to teach. Each target links to its glossary entry or home chapter.

Practice terms in coordinates#

The tables below map everyday evaluation vocabulary onto measurement, layer by layer: each practice term gets its coordinate in the framework and a reading in measurement terms.

Claim#

Practice term	Coordinate	Read it as
Golden dataset, task suite	Population $P$ and how tasks are sampled from it	A sampling frame and content blueprint for the construct: in measurement terms, the test content
Task taxonomy, slices	Construct facets and population strata	How the content and its heterogeneity are represented
Operational design domain	Validity domain $D$	The conditions under which the evidence or guarantees apply: the population of justified inference
Contamination	A validity threat	An illicit information path that gives a construct-irrelevant advantage; see contamination
System under test	The system under test	The agent or policy, plus the observation pipeline that is being measured

Instrument#

Practice term	Coordinate	Read it as
Task success	Specification $\Psi$ plus the observed state	A desired state or trajectory property, observed as a performance indicator
Rubric, assertions	Specification $\Psi$	The operationalization of human judgment, written as a rubric
Trajectory checks ("never X", "Y before Z")	Specification at process resolution	Temporal constraints and invariants over a trajectory, written as temporal patterns
Partial credit, margin tracking	Signal $Z$ or robustness margin $\rho(\psi, \tau)$	A graded response, or a signed distance to satisfying the property
Step-level grading	Resolution $R$	Process evaluation, whose mechanism may be formal, executable, learned, or human
Execution grader	Mechanism $M$	A property or effect checker, with high criterion fidelity for executable properties
LLM judge	Learned judge	A noisy semantic sensor: in measurement terms, an imperfect rater
Dashboard number	Signal plus estimator	A chosen statistical functional of measured behavior

Regime#

Practice term	Coordinate	Read it as
Release gate	Release gate, a use in the measure regime	A score mapped to a ship-or-hold decision through a threshold: a cut-score decision. This handbook treats it as measure until developers iterate against it (Use regimes)
Guardrail threshold	Runtime control, in the monitor-control regime	A runtime intervention rule; with a guarantee attached, a conformal threshold
Read-only credentials, sandbox walls	Structural guarantee	Reachability restricted by construction: $P_f = 0$ as a property of the configured system, not an estimate
Reward hacking	A failure of the optimize regime	Optimization against an imperfect specification or evaluator; see reward hacking

Validation#

Practice term	Coordinate	Read it as
Judge calibration set	Validation anchor sample	The labeled data that identifies a judge's error rates: criterion evidence for the scorer
Corrected pass rate	Statistical estimator	The inverted sensor model, a Rogan–Gladen prevalence estimate
Inter-judge agreement	Reliability	One consistency component, inter-rater reliability, not an error rate
Repeated rollouts	Reliability design	Test-retest measurement of stochastic trajectories
Alternate task forms	Reliability and generalization	Alternate-form reliability, by resampling the environment
Pass rate	An estimate of $p$	An estimated probability of satisfying the specification: a pass rate, reported with its interval
Confidence interval	Statistics	Estimation precision, not validity or reliability in the broad sense
Hard-slice oversampling with reweighting	Stratified estimate or importance sampling	Biased sampling with unbiased estimation, weighting by $p/q$
Severity ladders, staged red-team funnels	Multilevel splitting	A product of conditional probabilities
Red teaming	Validation stress test	A falsification search for failures and boundary evidence
Plausible versus exotic red-team finds	Most-likely-failure objective	A likelihood-weighted worst case; see adversary regimes
Dataset drift audit	Environment-model validation	A population-validity or measurement-invariance check

Lifecycle#

Practice term	Coordinate	Read it as
Eval saturation	Saturation	An instrument range limit, the ceiling effect: a maintenance event, not a success (Living eval suites)

What agent-benchmark concepts call for#

Agent-benchmark work names its own validity concepts. Each maps to the measurement evidence it calls for, and the mapping is many-to-many:

Concept	Evidence it calls for	The question it asks	Home
Task validity	Content and response-process evidence; discriminant evidence	Do the tasks elicit the intended capability and cover its domain?	Task and outcome validity
Outcome validity	Scoring and criterion evidence; scorer reliability	Does a positive grade correspond to real success?	Task and outcome validity
Task distribution	Content evidence; generalization	Does the sample represent the deployment population, slice by slice?	Task distributions
Harness	Standardized administration; control of the response process	Does the setup afford or obstruct construct-relevant behavior?	Harness and reproducibility
Judge calibration	Criterion evidence, plus reliability and precision	How does the scorer behave against an external reference, and how stable is it?	The judge as a noisy sensor
Slice structure	Internal-structure evidence	Do the subscales and the aggregate behave as the construct theory predicts?	Sources of validity evidence
Cross-benchmark comparison	Convergent and discriminant evidence	Does the score track same-construct measures and ignore nuisance constructs?	Sources of validity evidence
Deployment transfer	Predictive and ecological criterion evidence	Does the score predict the outcome users care about?	Sources of validity evidence
Static or dynamic maintenance	Validity-domain and reliability evidence	What must stay fixed for comparisons, and what must refresh to stay representative?	Living eval suites

The typical failures run down the same rows: shortcuts and impossible tasks, incomplete tests and brittle matching, easy-case bias and wrong weights, broken simulators and hidden rate limits, judges that agree on wrong labels, misleading means, artifact-driven rankings, offline gains that do not transfer, and the tension between refreshing a suite against contamination and drift and keeping it reproducible. The validity argument places all of them in one argument.

The six-axis type in layers#

An evaluator can be typed by six axes rather than named by a noun, $\mathcal{E} = \langle C, \Psi, M, Z, R, U \rangle$ . The framework distributes the six-axis type over three layers of the evaluator system:

Axis	Layer	Coordinates in the framework
Construct $C$	Claim $\mathcal{C}$	The construct $C$ , joined by population $P$ , environment $E$ , harness $H$ , validity domain $D$ , and decision $\delta$
Specification $\Psi$	Instrument $\mathcal{I}$	$\Psi$
Mechanism $M$	Instrument $\mathcal{I}$	$M$ , with its hierarchy family recorded as a prior
Signal $Z$	Instrument $\mathcal{I}$	$Z$
Resolution $R$	Instrument $\mathcal{I}$	$R$
Use $U$	Regime $\mathcal{R}$	The consumer $\kappa$ and the regime it puts the signal in, joined by optimizer $O$ , exposure budget $b$ , modification surface $S$ , information access $A_{\text{access}}$ , and cost envelope $K$

The instrument adds three coordinates the type leaves implicit: the anchor $A$ , the evidence view $V$ , and the aggregation geometry $\Gamma$ . The remaining two layers have no axis in the type: validation, where the validation profile $\Pi$ replaces the hierarchy label with measurements, and the lifecycle. Vocabulary without category errors applies the type to seven familiar evaluators, and the notation card defines every symbol.

Raschka's four approaches, and this site's four methods#

Raschka (2025) maps practical LLM evaluation into four approaches. This site's earlier framework page taught the same four, with leaderboards renamed human preference evaluation, as its "Four Evaluation Methods". The approaches name different axes, mixing response format, evidence mechanism, aggregation protocol, and grader implementation, so each translates into several coordinates:

Earlier method (Raschka's approach)	Axis it mostly names	Coordinates	Hierarchy family
Multiple-choice benchmarks (multiple choice)	Task and response design	A fixed answer key as the anchor $A$ , compared by a deterministic checker, at outcome resolution	Not a rung; the scorer may be deterministic
Verifiers, or symbolic checking (verifiers)	Mechanism	An external check $M$ of a free-form answer: an execution checker or deterministic checker, or a formal verifier when it carries a soundness argument	Execution or deterministic; formal only with soundness
Human preference evaluation (leaderboards)	Aggregation protocol over raters	Human raters as $M$ , pairwise preference as $Z$ , and a ranking model as $\Gamma$ ; the construct is what the sampled people prefer	A human judgment protocol, not an automated rung
LLM-as-a-judge (LLM judges)	Mechanism	An LLM judge as $M$ , with its rubric $\Psi$ , evidence view $V$ , and configuration	The learned-judge family

Because the approaches name different axes, one benchmark can belong to two of them: a leaderboard whose pairwise preferences come from an LLM judge is both. The earlier page also ranked the methods, calling human preference the most valid for subjective quality and verifiers the strongest choice for deterministic tasks. In coordinates, validity belongs to a claim, and trust in a mechanism comes from the validation profile measured for a configured system; the hierarchy family is only a prior (The verification hierarchy, demoted).

The retired three-level framework#

This site's earlier framework had three levels: a hypothesis about a capability, a behavioral specification of the observable behaviors that demonstrate it, and an evaluation design that chose a method for each behavior. Each level maps onto coordinates, and the framework adds what none of them covered:

Earlier level	Its question	Coordinates now	What the level left out
1. Hypothesis	What capability do you need to measure?	The claim $\mathcal{C}$ : construct, population, and decision. Its thresholds belong to the decision and the gate semantics	The environment, harness, and validity domain; who consumes the score
2. Behavioral spec	What observable behaviors demonstrate it?	The facets $F^{+}$ and the specification $\Psi$ , as criteria judged at a stated resolution	The nuisances $F^{-}$ the score must ignore; whether the process is part of the construct at all
3. Eval design	How will you reliably measure those behaviors?	The rest of the instrument: anchor, evidence view, mechanism, signal, and aggregation	The regime, the validation profile, and the lifecycle

The earlier method chooser, which matched a method to each behavior, survives as the rule of the strongest affordable mechanism per criterion per regime (Verifiability engineering). The earlier page's four pitfalls map onto the handbook's vocabulary too. Single-number fixation is one-number reporting; overfitting to benchmarks is contamination or Goodharting; cost blindness becomes the regime's cost and latency envelope $K$ ; and benchmark shopping is answered as the earlier page answered it, by starting from the claim, which now includes its decision (Decision claims).