# Crosswalk

> Translate golden datasets, LLM judges, partial credit and release gates into unified coordinates, plus the six-axis type and Raschka's four approaches.

This crosswalk translates evaluation vocabulary you already know into the framework's coordinates: practice terms such as golden datasets, LLM judges, partial credit, and release gates; the evidence that agent-benchmark validity concepts call for; the six-axis evaluator type; Raschka's four approaches; and the three-level framework this site used to teach. Each target links to its glossary entry or home chapter.

## Practice terms in coordinates

The tables below map everyday evaluation vocabulary onto measurement, layer by layer: each practice term gets its coordinate in the framework and a reading in measurement terms.

### Claim

| Practice term | Coordinate | Read it as |
|---|---|---|
| Golden dataset, task suite | [Population](https://benchmaker.ai/glossary/population) $P$ and how tasks are sampled from it | A sampling frame and content blueprint for the construct: in measurement terms, the test content |
| Task taxonomy, slices | [Construct facets](https://benchmaker.ai/glossary/construct-facet) and population strata | How the content and its heterogeneity are represented |
| Operational design domain | [Validity domain](https://benchmaker.ai/glossary/validity-domain) $D$ | The conditions under which the evidence or guarantees apply: the population of justified inference |
| Contamination | A validity threat | An illicit information path that gives a construct-irrelevant advantage; see [contamination](https://benchmaker.ai/handbook/practice/anti-patterns#contamination) |
| System under test | The [system under test](https://benchmaker.ai/glossary/system-under-test) | The agent or policy, plus the observation pipeline that is being measured |

### Instrument

| Practice term | Coordinate | Read it as |
|---|---|---|
| Task success | [Specification](https://benchmaker.ai/glossary/specification) $\Psi$ plus the observed state | A desired state or trajectory property, observed as a performance indicator |
| Rubric, assertions | Specification $\Psi$ | The operationalization of human judgment, written as a [rubric](https://benchmaker.ai/glossary/rubric) |
| Trajectory checks ("never X", "Y before Z") | Specification at process [resolution](https://benchmaker.ai/glossary/resolution) | Temporal constraints and invariants over a trajectory, written as [temporal patterns](https://benchmaker.ai/glossary/temporal-pattern) |
| Partial credit, margin tracking | [Signal](https://benchmaker.ai/glossary/signal) $Z$ or [robustness margin](https://benchmaker.ai/glossary/robustness-margin) $\rho(\psi, \tau)$ | A graded response, or a signed distance to satisfying the property |
| Step-level grading | Resolution $R$ | Process evaluation, whose mechanism may be formal, executable, learned, or human |
| Execution grader | [Mechanism](https://benchmaker.ai/glossary/mechanism) $M$ | A property or effect checker, with high criterion fidelity for executable properties |
| LLM judge | [Learned judge](https://benchmaker.ai/glossary/learned-judge) | A noisy semantic sensor: in measurement terms, an imperfect rater |
| Dashboard number | Signal plus estimator | A chosen statistical functional of measured behavior |

> [!CONVENTION]
> This handbook does not file step-level grading, or a best-of-n critic, under "process reward model": process is a resolution, and a PRM is one learned mechanism that works at it ([Signals and resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution)).

### Regime

| Practice term | Coordinate | Read it as |
|---|---|---|
| Release gate | [Release gate](https://benchmaker.ai/glossary/release-gate), a use in the measure regime | A score mapped to a ship-or-hold decision through a threshold: a cut-score decision. This handbook treats it as measure until developers iterate against it ([Use regimes](https://benchmaker.ai/handbook/regime/use-regimes)) |
| Guardrail threshold | [Runtime control](https://benchmaker.ai/glossary/runtime-control), in the monitor-control regime | A runtime intervention rule; with a guarantee attached, a [conformal](https://benchmaker.ai/glossary/conformal-prediction) threshold |
| Read-only credentials, sandbox walls | [Structural guarantee](https://benchmaker.ai/glossary/structural-guarantee) | Reachability restricted by construction: $P_f = 0$ as a property of the configured system, not an estimate |
| Reward hacking | A failure of the optimize regime | Optimization against an imperfect specification or evaluator; see [reward hacking](https://benchmaker.ai/glossary/reward-hacking) |

### Validation

| Practice term | Coordinate | Read it as |
|---|---|---|
| Judge calibration set | Validation anchor sample | The labeled data that identifies a judge's error rates: criterion evidence for the scorer |
| Corrected pass rate | Statistical estimator | The inverted sensor model, a [Rogan–Gladen](https://benchmaker.ai/glossary/rogan-gladen-correction) prevalence estimate |
| Inter-judge agreement | [Reliability](https://benchmaker.ai/glossary/reliability) | One consistency component, inter-rater reliability, not an error rate |
| Repeated rollouts | Reliability design | Test-retest measurement of stochastic trajectories |
| Alternate task forms | Reliability and generalization | Alternate-form reliability, by resampling the environment |
| Pass rate | An estimate of $p$ | An estimated probability of satisfying the specification: a [pass rate](https://benchmaker.ai/glossary/pass-rate), reported with its interval |
| Confidence interval | Statistics | Estimation precision, not validity or reliability in the broad sense |
| Hard-slice oversampling with reweighting | [Stratified estimate](https://benchmaker.ai/glossary/stratified-estimate) or [importance sampling](https://benchmaker.ai/glossary/importance-sampling) | Biased sampling with unbiased estimation, weighting by $p/q$ |
| Severity ladders, staged red-team funnels | [Multilevel splitting](https://benchmaker.ai/glossary/multilevel-splitting) | A product of conditional probabilities |
| Red teaming | Validation stress test | A [falsification](https://benchmaker.ai/glossary/falsification) search for failures and boundary evidence |
| Plausible versus exotic red-team finds | Most-likely-failure objective | A likelihood-weighted worst case; see [adversary regimes](https://benchmaker.ai/glossary/adversary-regime) |
| Dataset drift audit | Environment-model validation | A population-validity or [measurement-invariance](https://benchmaker.ai/glossary/measurement-invariance) check |

### Lifecycle

| Practice term | Coordinate | Read it as |
|---|---|---|
| Eval saturation | [Saturation](https://benchmaker.ai/glossary/saturation) | An instrument range limit, the ceiling effect: a maintenance event, not a success ([Living eval suites](https://benchmaker.ai/handbook/lifecycle/living-eval-suites)) |

## What agent-benchmark concepts call for

Agent-benchmark work names its own validity concepts. Each maps to the measurement evidence it calls for, and the mapping is many-to-many:

| Concept | Evidence it calls for | The question it asks | Home |
|---|---|---|---|
| [Task validity](https://benchmaker.ai/glossary/task-validity) | Content and response-process evidence; discriminant evidence | Do the tasks elicit the intended capability and cover its domain? | [Task and outcome validity](https://benchmaker.ai/handbook/validation/task-and-outcome-validity) |
| [Outcome validity](https://benchmaker.ai/glossary/outcome-validity) | Scoring and criterion evidence; scorer reliability | Does a positive grade correspond to real success? | [Task and outcome validity](https://benchmaker.ai/handbook/validation/task-and-outcome-validity) |
| Task distribution | Content evidence; generalization | Does the sample represent the deployment population, slice by slice? | [Task distributions](https://benchmaker.ai/handbook/claim/task-distributions) |
| Harness | Standardized administration; control of the response process | Does the setup afford or obstruct construct-relevant behavior? | [Harness and reproducibility](https://benchmaker.ai/handbook/lifecycle/harness-and-reproducibility) |
| Judge calibration | Criterion evidence, plus reliability and precision | How does the scorer behave against an external reference, and how stable is it? | [The judge as a noisy sensor](https://benchmaker.ai/handbook/validation/judge-as-noisy-sensor) |
| Slice structure | [Internal-structure evidence](https://benchmaker.ai/glossary/internal-structure-evidence) | Do the subscales and the aggregate behave as the construct theory predicts? | [Sources of validity evidence](https://benchmaker.ai/handbook/validation/validity-evidence) |
| Cross-benchmark comparison | [Convergent](https://benchmaker.ai/glossary/convergent-evidence) and [discriminant](https://benchmaker.ai/glossary/discriminant-evidence) evidence | Does the score track same-construct measures and ignore nuisance constructs? | [Sources of validity evidence](https://benchmaker.ai/handbook/validation/validity-evidence) |
| Deployment transfer | Predictive and ecological [criterion evidence](https://benchmaker.ai/glossary/criterion-evidence) | Does the score predict the outcome users care about? | [Sources of validity evidence](https://benchmaker.ai/handbook/validation/validity-evidence) |
| Static or dynamic maintenance | Validity-domain and reliability evidence | What must stay fixed for comparisons, and what must refresh to stay representative? | [Living eval suites](https://benchmaker.ai/handbook/lifecycle/living-eval-suites) |

The typical failures run down the same rows: shortcuts and impossible tasks, incomplete tests and brittle matching, easy-case bias and wrong weights, broken simulators and hidden rate limits, judges that agree on wrong labels, misleading means, artifact-driven rankings, offline gains that do not transfer, and the tension between refreshing a suite against contamination and drift and keeping it reproducible. [The validity argument](https://benchmaker.ai/handbook/validation/validity-argument) places all of them in one argument.

## The six-axis type in layers

An evaluator can be typed by six axes rather than named by a noun, $\mathcal{E} = \langle C, \Psi, M, Z, R, U \rangle$. The framework distributes the [six-axis type](https://benchmaker.ai/glossary/six-axis-type) over three layers of the evaluator system:

| Axis | Layer | Coordinates in the framework |
|---|---|---|
| Construct $C$ | Claim $\mathcal{C}$ | The [construct](https://benchmaker.ai/glossary/construct) $C$, joined by population $P$, environment $E$, harness $H$, validity domain $D$, and [decision](https://benchmaker.ai/glossary/decision-claim) $\delta$ |
| Specification $\Psi$ | Instrument $\mathcal{I}$ | $\Psi$ |
| Mechanism $M$ | Instrument $\mathcal{I}$ | $M$, with its hierarchy family recorded as a prior |
| Signal $Z$ | Instrument $\mathcal{I}$ | $Z$ |
| Resolution $R$ | Instrument $\mathcal{I}$ | $R$ |
| Use $U$ | Regime $\mathcal{R}$ | The [consumer](https://benchmaker.ai/glossary/consumer) $\kappa$ and the regime it puts the signal in, joined by optimizer $O$, exposure budget $b$, modification surface $S$, information access $A_{\text{access}}$, and cost envelope $K$ |

The instrument adds three coordinates the type leaves implicit: the [anchor](https://benchmaker.ai/glossary/anchor) $A$, the [evidence view](https://benchmaker.ai/glossary/evidence-view) $V$, and the [aggregation geometry](https://benchmaker.ai/glossary/aggregation-geometry) $\Gamma$. The remaining two layers have no axis in the type: validation, where the [validation profile](https://benchmaker.ai/glossary/validation-profile) $\Pi$ replaces the hierarchy label with measurements, and the lifecycle. [Vocabulary without category errors](https://benchmaker.ai/handbook/foundations/vocabulary) applies the type to seven familiar evaluators, and the [notation card](https://benchmaker.ai/handbook/reference/notation-and-formulas) defines every symbol.

## Raschka's four approaches, and this site's four methods

[Raschka (2025)](https://benchmaker.ai/handbook/reference/sources#raschka-2025) maps practical LLM evaluation into four approaches. This site's earlier framework page taught the same four, with leaderboards renamed human preference evaluation, as its "Four Evaluation Methods". The approaches name different axes, mixing response format, evidence mechanism, aggregation protocol, and grader implementation, so each translates into several coordinates:

| Earlier method (Raschka's approach) | Axis it mostly names | Coordinates | Hierarchy family |
|---|---|---|---|
| Multiple-choice benchmarks (multiple choice) | Task and response design | A fixed answer key as the anchor $A$, compared by a [deterministic checker](https://benchmaker.ai/glossary/deterministic-checker), at outcome resolution | Not a rung; the scorer may be deterministic |
| Verifiers, or symbolic checking (verifiers) | Mechanism | An external check $M$ of a free-form answer: an [execution checker](https://benchmaker.ai/glossary/execution-checker) or deterministic checker, or a [formal verifier](https://benchmaker.ai/glossary/formal-verifier) when it carries a soundness argument | Execution or deterministic; formal only with soundness |
| Human preference evaluation (leaderboards) | Aggregation protocol over raters | [Human raters](https://benchmaker.ai/glossary/human-rater) as $M$, pairwise preference as $Z$, and a ranking model as $\Gamma$; the construct is what the sampled people prefer | A human judgment protocol, not an automated rung |
| LLM-as-a-judge (LLM judges) | Mechanism | An [LLM judge](https://benchmaker.ai/glossary/llm-as-a-judge) as $M$, with its rubric $\Psi$, evidence view $V$, and configuration | The learned-judge family |

Because the approaches name different axes, one benchmark can belong to two of them: a leaderboard whose pairwise preferences come from an LLM judge is both. The earlier page also ranked the methods, calling human preference the most valid for subjective quality and verifiers the strongest choice for deterministic tasks. In coordinates, validity belongs to a claim, and trust in a mechanism comes from the validation profile measured for a configured system; the hierarchy family is only a prior ([The verification hierarchy, demoted](https://benchmaker.ai/handbook/foundations/verification-hierarchy)).

## The retired three-level framework

This site's earlier framework had three levels: a hypothesis about a capability, a behavioral specification of the observable behaviors that demonstrate it, and an evaluation design that chose a method for each behavior. Each level maps onto coordinates, and the framework adds what none of them covered:

| Earlier level | Its question | Coordinates now | What the level left out |
|---|---|---|---|
| 1. Hypothesis | What capability do you need to measure? | The claim $\mathcal{C}$: construct, population, and decision. Its thresholds belong to the decision and the gate semantics | The environment, harness, and validity domain; who consumes the score |
| 2. Behavioral spec | What observable behaviors demonstrate it? | The facets $F^{+}$ and the specification $\Psi$, as criteria judged at a stated resolution | The [nuisances](https://benchmaker.ai/glossary/nuisance-variable) $F^{-}$ the score must ignore; whether the process is part of the construct at all |
| 3. Eval design | How will you reliably measure those behaviors? | The rest of the instrument: anchor, evidence view, mechanism, signal, and aggregation | The regime, the validation profile, and the lifecycle |

The earlier method chooser, which matched a method to each behavior, survives as the rule of the [strongest affordable mechanism](https://benchmaker.ai/glossary/strongest-affordable-mechanism) per criterion per regime ([Verifiability engineering](https://benchmaker.ai/handbook/practice/verifiability-engineering)). The earlier page's four pitfalls map onto the handbook's vocabulary too. Single-number fixation is [one-number reporting](https://benchmaker.ai/handbook/practice/anti-patterns#one-number-reporting); overfitting to benchmarks is contamination or [Goodharting](https://benchmaker.ai/handbook/practice/anti-patterns#goodharting); cost blindness becomes the regime's [cost and latency envelope](https://benchmaker.ai/glossary/cost-latency-envelope) $K$; and benchmark shopping is answered as the earlier page answered it, by starting from the claim, which now includes its decision ([Decision claims](https://benchmaker.ai/handbook/claim/decision-claims)).

---
Source: https://benchmaker.ai/handbook/reference/crosswalk