# Glossary

> Every term grouped by role, with the chapter that defines it and the collisions it resolves.

This **glossary** defines the handbook's vocabulary in one place. Terms are grouped by role rather than alphabetically, so that related terms sit together in nine groups. Each definition is short and links to the chapter that develops it.

> [!KEY] In one sentence
> Every term names one object on one axis — mechanism, specification, signal, use, or evidence — so the same word never has to stand for two different things.

## How to use it

Hover a dotted term in any chapter to see its definition without leaving the page; the link brings you here. Filter the list below by group or by any word in a term, symbol, or definition.

Where the field's words collide — reward model, rubric, PRM, RLHF, verifier — the glossary gives the house meaning. [Vocabulary without category errors](https://benchmaker.ai/handbook/foundations/vocabulary) explains why each term sits on the axis it does, and the [crosswalk](https://benchmaker.ai/handbook/reference/crosswalk) maps common evaluation vocabulary onto the framework's coordinates.

A few conventions run through every entry:

- An **evaluator** is the umbrella term. A **verifier** has a soundness argument relative to a specification; a **judge** is a learned or prompted evaluator without one. People are raters, panels, or adjudicators, not judges.
- A **rubric** is a specification, and gold or reference data is an **anchor**; neither is a rung of the verification hierarchy.
- **pass@k** means at least one of k attempts succeeds; **pass^k** means all k succeed.
- Validity is a property of a claim, never of a benchmark on its own.

Symbols follow the [notation and formula card](https://benchmaker.ai/handbook/reference/notation-and-formulas).

## All terms

### Objects and system context

- **Agent**: A goal-directed system operating through an iterative perceive–act loop, usually with tools, state, and stopping conditions. Agency is a property of the loop, not of the model. See [/handbook/agents/evaluating-agents](https://benchmaker.ai/handbook/agents/evaluating-agents).
- **Artifact**: The output presented for evaluation: an answer, proof, program, report, plan, edited file, or database state. See [/handbook/foundations/evaluation-is-measurement](https://benchmaker.ai/handbook/foundations/evaluation-is-measurement).
- **Category error**: Treating objects that describe different axes — a specification, a mechanism, a resolution, a use — as peers on one list, as in ranking a rubric against a reward model or calling RLHF an evaluator. See [/handbook/foundations/vocabulary](https://benchmaker.ai/handbook/foundations/vocabulary).
- **Dual-control environment**: An interactive environment in which both the agent and a simulated user act through tools on a shared world, each with its own state and toolset, so success depends on coordination and guidance, as in τ²-bench. See [/handbook/agents/evaluating-agents](https://benchmaker.ai/handbook/agents/evaluating-agents).
- **Environment**: External state and transition process the system interacts with; may be simulated, sandboxed, pinned, or live. See [/handbook/claim/validity-domain](https://benchmaker.ai/handbook/claim/validity-domain).
- **Eval harness**: The machinery that administers an evaluation and records trials. It is part of the instrument, distinct from the agent harness, which is part of the system under test. See [/handbook/lifecycle/harness-and-reproducibility](https://benchmaker.ai/handbook/lifecycle/harness-and-reproducibility).
- **Evaluator agent**: An agent that administers an assessment: it prepares the environment, sends tasks to participant agents, collects results, and reports the metrics it defines. It bundles harness, specification, and mechanism; Berkeley's lecture calls it a green agent. See [/handbook/lifecycle/harness-and-reproducibility](https://benchmaker.ai/handbook/lifecycle/harness-and-reproducibility).
- **Evaluator construction stack**: How an evaluation is built: construct, specification, mechanism, signal, and use, with resolution and deployment context cutting across. It records how the evaluator was engineered, a separate question from the verification hierarchy's evidential strength. See [/handbook/foundations/vocabulary](https://benchmaker.ai/handbook/foundations/vocabulary).
- **Evaluator system** (E_sys): The complete arrangement by which a claim is operationalized, measured, consumed, and maintained: E_sys = (C, I, R, V, L). A mechanism is only one part of it. See [/handbook/foundations/evaluator-systems](https://benchmaker.ai/handbook/foundations/evaluator-systems).
- **Harness (scaffolding)**: Everything around the model that turns it into an agent: prompts, tools, memory, retrieval, skills, orchestration, permissions, and stopping logic. See [/handbook/claim/validity-domain](https://benchmaker.ai/handbook/claim/validity-domain).
- **Instrument chain**: The links through which a score supports a decision: target construct, sampled tasks and environments, trials and trajectories, grader readings, an estimate with uncertainty, and the decision. Error and invalid inference can enter at every link. See [/handbook/foundations/evaluation-is-measurement](https://benchmaker.ai/handbook/foundations/evaluation-is-measurement).
- **Measurement version**: The unit of evaluator versioning: rubric text, examples, judge model and prompt, checkers, thresholds, aggregation logic, and expert policy, versioned together. Changing a load-bearing part creates a new measurement form: revalidate it or run a bridging study. See [/handbook/lifecycle/harness-and-reproducibility](https://benchmaker.ai/handbook/lifecycle/harness-and-reproducibility).
- **One Spine**: The framework's ten-step workflow for building an evaluator system: claim and regime first, then decomposition, criteria, evidence, mechanisms, natural and adaptive validity, separate channels, tripwires, revalidation, and frame review. See [/handbook/practice/one-spine](https://benchmaker.ai/handbook/practice/one-spine).
- **Population** (P): The target task, user, or input distribution a claim is about. An eval set is a finite model of it; its headline estimates the population's pass rate only when the set's mix matches. See [/handbook/claim/validity-domain](https://benchmaker.ai/handbook/claim/validity-domain).
- **Simulated user**: A language model, prompted with personas mined from real conversation logs, that plays the user in multi-turn evals. It is environment, not system: its quirks can create fake failures and fake passes, and it drifts like a judge. See [/handbook/agents/evaluating-agents](https://benchmaker.ai/handbook/agents/evaluating-agents).
- **Six-axis evaluator type** (⟨C, Ψ, M, Z, R, U⟩): The tuple ⟨C, Ψ, M, Z, R, U⟩ — construct, specification, mechanism, signal, resolution, use — that describes an evaluator by its value on every axis instead of a single noun. The framework distributes it across E_sys. See [/handbook/foundations/vocabulary](https://benchmaker.ai/handbook/foundations/vocabulary).
- **System under test (SUT)**: The model, policy, or agent, plus the relevant observation and action pipeline, whose behavior is being measured. See [/handbook/foundations/evaluation-is-measurement](https://benchmaker.ai/handbook/foundations/evaluation-is-measurement).
- **Trajectory** (τ): The sequence of states and actions (s₀, a₀, …, s_T) generated by an agent interaction. See [/handbook/agents/evaluating-agents](https://benchmaker.ai/handbook/agents/evaluating-agents).
- **Validation triad**: The system under test, the environment it acts in, and the specification its behavior is checked against. It exposes two structural failures: faithfully measuring the wrong world, and faithfully checking the wrong property. See [/handbook/foundations/evaluation-is-measurement](https://benchmaker.ai/handbook/foundations/evaluation-is-measurement).

### Meaning and specification

- **Abstention region**: Where a criterion's evaluator should answer "cannot determine" or route the item to a person instead of forcing a verdict, for example when sources conflict or the case is out of domain. Forced verdicts turn uncertainty into false confidence. See [/handbook/instrument/rubrics-and-criterion-cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards).
- **Aggregation geometry** (Γ): The rule that combines criterion-level signals into a decision: weighted mean, conjunctive gate, lexicographic order, Pareto vector, or tail-risk rule. See [/handbook/instrument/aggregation-geometry](https://benchmaker.ai/handbook/instrument/aggregation-geometry).
- **Claim scope**: Whether an evaluation supports claims about a specific capability, a specific application, or a general set of applications. Scope fixes the population; the broadest scope makes the strongest and most fragile claim. See [/handbook/claim/validity-domain](https://benchmaker.ai/handbook/claim/validity-domain).
- **Claim tuple** (𝒞): The claim layer of an evaluator system, 𝒞 = (C, P, E, H, D, δ): the construct to infer, over which population, environment, and harness, within which validity domain, for which decision. See [/handbook/claim/constructs](https://benchmaker.ai/handbook/claim/constructs).
- **Compensability**: The assumption that a high score on one criterion can offset a low score on another. A weighted mean asserts it; a gate or a lexicographic order denies it. It is a normative claim, never a default. See [/handbook/instrument/aggregation-geometry](https://benchmaker.ai/handbook/instrument/aggregation-geometry).
- **Conjunctive gate**: An aggregation rule that accepts only if every required criterion clears its threshold, Γ(z) = ∏ᵢ 𝟙[zᵢ ≥ θᵢ]. Nothing compensates a missed gate, so it encodes fatal failures such as forbidden actions. See [/handbook/instrument/aggregation-geometry](https://benchmaker.ai/handbook/instrument/aggregation-geometry).
- **Construct** (C): The latent property an evaluation intends to measure: a capability, risk, quality, preference, or behavior. Scores and artifacts are observations of it, never the construct itself. See [/handbook/claim/constructs](https://benchmaker.ai/handbook/claim/constructs).
- **Construct facet** (F⁺): A distinct dimension within a construct, used to build a content blueprint for the task population. See [/handbook/claim/constructs](https://benchmaker.ai/handbook/claim/constructs).
- **Construct underrepresentation**: A validity threat in which important facets of the construct are absent from the tasks or too rare, so the score is too narrow for its claim. The facet list guards against it. See [/handbook/claim/constructs](https://benchmaker.ai/handbook/claim/constructs).
- **Construct-irrelevant variance**: A validity threat in which factors outside the construct, such as output format, token budgets, tool latency, or answer length, drive the score, making it impure. The nuisance list guards against it. See [/handbook/claim/constructs](https://benchmaker.ai/handbook/claim/constructs).
- **Cooperative inverse reinforcement learning (CIRL)**: A cooperative partial-information game in which human and machine both pursue the human's reward but the machine is uncertain of it, so optimal play includes questions and teaching. Evaluation analogue: query where posterior uncertainty is high. See [/handbook/practice/inverse-evaluation-design](https://benchmaker.ai/handbook/practice/inverse-evaluation-design).
- **Criterion card**: The atomic specification for one criterion: facet, operational definition, allowed evidence, positive and negative exemplars, boundary and abstention rules, severity (fatal, tradeable, advisory, or escalate), mechanism, blind spots, owner, and version. See [/handbook/instrument/rubrics-and-criterion-cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards).
- **Decision claim** (δ): The decision the evidence is meant to change: deploy, gate, select, terminate, reward, or page a human. See [/handbook/claim/decision-claims](https://benchmaker.ai/handbook/claim/decision-claims).
- **Error analysis**: Discovering failure modes from real traces: a domain expert open-codes what went wrong, axial-codes the notes into a counted failure taxonomy, and iterates until about twenty consecutive traces add no new category. See [/handbook/claim/task-distributions](https://benchmaker.ai/handbook/claim/task-distributions).
- **Evidence contract**: The part of a criterion card that states which observable evidence is admissible for that criterion: for example cited spans for grounding, or the tool trace and state transition, never the final prose, for a forbidden action. See [/handbook/instrument/anchors-and-evidence-views](https://benchmaker.ai/handbook/instrument/anchors-and-evidence-views).
- **Evidence view** (V): What the evaluator may observe: the final answer only, or also sources, tool traces, hidden state, snapshots, provenance and logs. See [/handbook/instrument/anchors-and-evidence-views](https://benchmaker.ai/handbook/instrument/anchors-and-evidence-views).
- **Exemplar anchors**: Positive and negative examples, adversarial near-misses, and scale points attached to a criterion card to fix its boundary. Not the reference anchor A, the anchor channel, or the linking items kept across suite versions. See [/handbook/instrument/anchors-and-evidence-views](https://benchmaker.ai/handbook/instrument/anchors-and-evidence-views).
- **Inverse evaluation design**: A proposed research program that treats expert preferences, corrections, escalations, rubric edits, and accept/reject decisions as evidence of a latent judgment and infers its criteria, weights, gates, and thresholds. See [/handbook/practice/inverse-evaluation-design](https://benchmaker.ai/handbook/practice/inverse-evaluation-design).
- **Inverse reinforcement learning (IRL)**: Infers which reward functions could explain observed behavior, often as a Bayesian posterior under a Boltzmann (noisily rational) choice model. Many rewards can explain the same behavior, so the inferred reward is only partly identifiable. See [/handbook/practice/inverse-evaluation-design](https://benchmaker.ai/handbook/practice/inverse-evaluation-design).
- **Inverse reward design**: Treats a hand-written reward as evidence about its designer's true objective, given the situations it was designed for. Evaluation analogue: a rubric or benchmark score is evidence about what its designer meant to measure, not the construct itself. See [/handbook/practice/inverse-evaluation-design](https://benchmaker.ai/handbook/practice/inverse-evaluation-design).
- **Judgment compiler**: An eleven-step procedure that compiles expert judgment, the source language, through a rubric, the intermediate representation, into executable graders, with calibration and validity studies as its tests. See [/handbook/instrument/rubrics-and-criterion-cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards).
- **Lexicographic rule** (≻): An aggregation rule that compares options on the highest-priority criterion first, such as safety ≻ quality ≻ cost; a lower criterion counts only among options that the higher ones do not separate. See [/handbook/instrument/aggregation-geometry](https://benchmaker.ai/handbook/instrument/aggregation-geometry).
- **Marr's levels**: Three levels at which an evaluation question sits: computational (what claim and decision), algorithmic (what representation and procedure), implementation (what machinery). Locate a disagreement's level before arguing about tools. See [/handbook/foundations/marr-levels](https://benchmaker.ai/handbook/foundations/marr-levels).
- **Metric**: A function mapping a trajectory or artifact to a quantity. Over a task distribution it has a distribution of values, and any reported number is a chosen functional of it. A specification adds the demand, such as a threshold. See [/handbook/instrument/specification](https://benchmaker.ai/handbook/instrument/specification).
- **Nomological network**: The expected network of relationships between a construct's measure and other measures: which facets it should track, which nuisances it should ignore, and how it should relate to independent measures of the same and different constructs. See [/handbook/claim/constructs](https://benchmaker.ai/handbook/claim/constructs).
- **Nuisance variable** (F⁻): A property that should not materially move the score when the construct is held fixed, such as length, formatting, or assertive tone. The nuisance list is the discriminant test plan. See [/handbook/claim/constructs](https://benchmaker.ai/handbook/claim/constructs).
- **Outer design axes**: Four independent axes that locate an eval design: response space (closed to open), oracle strength (verifiable to judgment-based), maintenance (static to versioned dynamic), and interaction (single-turn to stateful trajectory). See [/handbook/agents/agent-benchmarks](https://benchmaker.ai/handbook/agents/agent-benchmarks).
- **Pareto frontier**: The options that no other option beats on every criterion at once, so improving one criterion means worsening another. Plot it before scalarizing; weights then choose a point on it. See [/handbook/instrument/aggregation-geometry](https://benchmaker.ai/handbook/instrument/aggregation-geometry).
- **Reference anchor** (A): The external object a judgment is rooted in: a canonical answer, gold state, reference distribution, adjudicated expert labels, a stronger evaluator, or production outcomes. Gold is a reference anchor, not a mechanism or a rung. See [/handbook/instrument/anchors-and-evidence-views](https://benchmaker.ai/handbook/instrument/anchors-and-evidence-views).
- **Rubric**: A structured, human-readable specification with criteria, allowed evidence, exemplar anchors or scales, and gate or aggregation semantics. A rubric is a specification, not a hierarchy rung. See [/handbook/instrument/rubrics-and-criterion-cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards).
- **Rubric atomization**: Decomposition taken too far: dozens of micro-criteria whose combination no longer reproduces how experts decide. The right grain is the smallest set of independently meaningful criteria that still reconstructs the expert decision. See [/handbook/instrument/rubrics-and-criterion-cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards).
- **Rubric soup**: Several logically distinct criteria packed into one paragraph and scored by one judge on a single scale, so failures cannot be localized and compensation between criteria stays hidden. Its opposite error is rubric atomization. See [/handbook/instrument/rubrics-and-criterion-cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards).
- **Specification** (Ψ): The operational statement of success or acceptable behavior: formal, executable, rubric-based, preference-based, or state-based. See [/handbook/instrument/specification](https://benchmaker.ai/handbook/instrument/specification).
- **Temporal pattern**: A trajectory requirement written with temporal operators: safety invariant 𝖦 ¬bad, precedence ¬a 𝖴 b, liveness 𝖥 goal, bounded response, or reachability. The same property grades offline trials and runs as a runtime monitor. See [/handbook/instrument/specification](https://benchmaker.ai/handbook/instrument/specification).

### Evaluator mechanisms

- **Capability removal**: Making an unsafe state unreachable by construction, with read-only credentials, egress allowlists, spend caps, or sandbox walls, so its failure probability is zero as a property of the configured system rather than an estimate. See [/handbook/practice/structural-guarantees](https://benchmaker.ai/handbook/practice/structural-guarantees).
- **Completeness**: Whether a verifier accepts every true instance of its property. Distinct from soundness, and often traded for tractability. Passing tests establishes neither: it means no tested counterexample was found. See [/handbook/foundations/verification-hierarchy](https://benchmaker.ai/handbook/foundations/verification-hierarchy).
- **Configured trusted base**: Everything a formal guarantee is relative to: permitted axioms, theorem statements, checker version, kernel configuration, environment assumptions, and whether the optimizer can modify any of them. The Goodhart boundary moves here rather than disappearing. See [/handbook/foundations/verification-hierarchy](https://benchmaker.ai/handbook/foundations/verification-hierarchy).
- **Constrained decoding**: Grammar-constrained generation that guarantees every output belongs to a formal language: every emission parses and every tool call is schema-valid. A genuine guarantee about syntax, never semantics. See [/handbook/practice/structural-guarantees](https://benchmaker.ai/handbook/practice/structural-guarantees).
- **Critic**: A mechanism that produces diagnostic critique or error information; it need not emit a scalar reward. See [/handbook/instrument/mechanisms](https://benchmaker.ai/handbook/instrument/mechanisms).
- **Dark current**: A judge's baseline preference when there is no real quality signal to detect: what it favors when the options do not differ in the construct. One of the properties a judge datasheet probes. See [/handbook/instrument/llm-judges](https://benchmaker.ai/handbook/instrument/llm-judges).
- **Deterministic checker**: An exact rule, parser, or comparator without learned semantic judgment. It can still be invalid or incomplete, and is only as good as its reference anchor and normalization. See [/handbook/instrument/mechanisms](https://benchmaker.ai/handbook/instrument/mechanisms).
- **Evaluator**: The umbrella term for any mechanism or artifact that maps evidence to a quality signal. On its own it is underspecified: ask which coordinate is meant. See [/handbook/foundations/vocabulary](https://benchmaker.ai/handbook/foundations/vocabulary).
- **Execution / effect checker**: An evaluator that runs code, compiles, simulates, queries state, or observes consequences. Grounded but incomplete: passing tests means no tested counterexample was found. See [/handbook/instrument/mechanisms](https://benchmaker.ai/handbook/instrument/mechanisms).
- **Formal verifier**: A proof checker, model checker, type system, or similar mechanism whose acceptance has a soundness argument relative to a configured trusted base. See [/handbook/instrument/mechanisms](https://benchmaker.ai/handbook/instrument/mechanisms).
- **Human rater or panel**: A human measurement mechanism whose reliability, expertise, and disagreement should be characterized rather than assumed infallible. See [/handbook/instrument/mechanisms](https://benchmaker.ai/handbook/instrument/mechanisms).
- **Hybrid portfolio**: An evaluator that routes each criterion to the strongest affordable mechanism for it, such as structural checks, execution, a learned judge, or people, instead of one universal evaluator. Its assurance depends on how coupled its channels are. See [/handbook/instrument/mechanisms](https://benchmaker.ai/handbook/instrument/mechanisms).
- **Intrinsic signal**: Confidence, likelihood, self-consistency, or ungrounded self-critique produced by the generator itself. Cheap, but not an independent observation of truth. See [/handbook/instrument/mechanisms](https://benchmaker.ai/handbook/instrument/mechanisms).
- **Judge**: A learned or prompted evaluator without formal soundness guarantees. People who score are raters, panels, or adjudicators, not judges, in this handbook. See [/handbook/foundations/vocabulary](https://benchmaker.ai/handbook/foundations/vocabulary).
- **Judge datasheet**: A maintained record that characterizes a learned judge as an instrument: target sensitivity, nuisance invariance, threshold behavior, reference agreement by criterion and slice, and repeatability and transport. Measure the measuring device before trusting it. See [/handbook/instrument/llm-judges](https://benchmaker.ai/handbook/instrument/llm-judges).
- **Learned judge**: A reward model, LLM judge, learned PRM, classifier, or other learned evaluator: a noisy semantic sensor whose error must be characterized. See [/handbook/instrument/mechanisms](https://benchmaker.ai/handbook/instrument/mechanisms).
- **LLM-as-a-judge**: A prompted general-purpose LLM evaluating another artifact, usually with a rubric, references, or pairwise comparison. See [/handbook/instrument/llm-judges](https://benchmaker.ai/handbook/instrument/llm-judges).
- **Mechanism** (M): What computes a judgment: a formal verifier, execution checker, deterministic checker, structural control, learned judge, human rater or panel, intrinsic signal, or a hybrid portfolio. A mechanism is not an evaluator system. See [/handbook/instrument/mechanisms](https://benchmaker.ai/handbook/instrument/mechanisms).
- **Meta-evaluation**: Evaluation of an evaluator. It adds assurance only when it contributes information not already coupled to the primary evaluator's failure modes. See [/handbook/instrument/mechanisms](https://benchmaker.ai/handbook/instrument/mechanisms).
- **Metamorphic test**: A test that checks a known relation between the outputs of related inputs rather than each output against a reference, so errors such as accidental result-set equivalence surface without a fresh anchor for every input. See [/handbook/cases/sql-data-agent](https://benchmaker.ai/handbook/cases/sql-data-agent).
- **Outcome reward model (ORM)**: A learned judge operating on terminal outputs or completed trajectories. See [/handbook/instrument/signal-and-resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution).
- **Process reward model (PRM)**: A learned judge operating at step or state resolution. "Process" names the resolution, not a rung above outcome checkers. See [/handbook/instrument/signal-and-resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution).
- **Reliability–coverage trade-off**: Signal reliability rises toward the top of the verification hierarchy while task coverage widens toward the bottom. The response is to decompose the judgment and push each criterion as high as it can legitimately go. See [/handbook/foundations/verification-hierarchy](https://benchmaker.ai/handbook/foundations/verification-hierarchy).
- **Reward model (RM)**: A learned model that predicts preference or quality, typically as a scalar or pairwise estimate, and is often optimized against. See [/handbook/regime/training-loops](https://benchmaker.ai/handbook/regime/training-loops).
- **Soundness**: A verifier V_ψ is sound when V_ψ(y) = 1 implies ψ(y) = 1: nothing it accepts violates the formal property, provided the verifier and its configured trusted base behave as claimed. See [/handbook/foundations/verification-hierarchy](https://benchmaker.ai/handbook/foundations/verification-hierarchy).
- **Strongest affordable mechanism**: The rule that amends strongest-oracle-per-criterion for cost and regime: choose the strongest affordable mechanism per criterion per regime, then use different mechanisms across development and certification when independence is valuable. See [/handbook/practice/verifiability-engineering](https://benchmaker.ai/handbook/practice/verifiability-engineering).
- **Structural control**: A mechanism family that makes a configured bad state unreachable through read-only credentials, sandboxes, permissions, or allowlists, instead of measuring it. It sits off the verification ladder; the guarantee covers only the configured roles and paths. See [/handbook/instrument/mechanisms](https://benchmaker.ai/handbook/instrument/mechanisms).
- **Structural guarantee**: A property enforced by the system's construction (permissions, sandboxes, verified scaffold rules, or an output grammar), so a failure is unreachable rather than rare. It holds only for its configured trusted base. See [/handbook/practice/structural-guarantees](https://benchmaker.ai/handbook/practice/structural-guarantees).
- **Substitution claim**: The implicit claim every automated evaluator makes: this observable signal is an adequate substitute for the relevant human judgment, for this decision, on this validity domain. Evaluator engineering makes it explicit, tests it, and bounds where it fails. See [/handbook/instrument/mechanisms](https://benchmaker.ai/handbook/instrument/mechanisms).
- **Subtraction principle**: Every property enforced by construction or formal verification is removed from the sampling burden, so scarce empirical evaluation budget is spent only on the properties that cannot be guaranteed. See [/handbook/practice/structural-guarantees](https://benchmaker.ai/handbook/practice/structural-guarantees).
- **Trust root (reference authority)**: Where the evaluate-the-evaluator chain stops: a configured proof kernel, or an expert panel with measured reliability and a governance mandate. See [/handbook/pressure/validation-profile](https://benchmaker.ai/handbook/pressure/validation-profile).
- **Value model**: Estimates expected future return from a state or prefix, conditional on a policy. Rollout-labeled "step quality" usually behaves like one. See [/handbook/instrument/signal-and-resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution).
- **Verifiability engineering**: Moving each property of a task toward the strongest evidence channel its semantics permit, through four moves (decompose, instrument, compile, retain the residue honestly), instead of asking whether a whole domain is verifiable. See [/handbook/practice/verifiability-engineering](https://benchmaker.ai/handbook/practice/verifiability-engineering).
- **Verification hierarchy**: Formal verification > execution feedback > learned judges > intrinsic signals: a prior over expected grounding strength, not a verdict on validity. Human judgment sits outside the ladder as the external frontier and trust root. See [/handbook/foundations/verification-hierarchy](https://benchmaker.ai/handbook/foundations/verification-hierarchy).
- **Verifier**: Strictly, an evaluator with a soundness guarantee relative to a formal specification. In loose usage it often means a deterministic or execution checker. See [/handbook/foundations/vocabulary](https://benchmaker.ai/handbook/foundations/vocabulary).

### Signal and resolution

- **Best-of-N**: Generate several candidates and select by evaluator score. It introduces extreme-selection distribution shift toward the evaluator's tail. See [/handbook/pressure/adaptive-validity](https://benchmaker.ai/handbook/pressure/adaptive-validity).
- **Outcome evaluation**: Judges the terminal artifact or terminal environment state. See [/handbook/instrument/signal-and-resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution).
- **Pass rate** (p): The estimated probability that evaluated behavior satisfies the operational specification under a declared distribution. See [/handbook/validation/estimands-and-intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals).
- **pass@k**: The probability that at least one of k sampled candidates succeeds: 1 − (1 − p)^k under independence. Not to be confused with pass^k. See [/handbook/validation/estimands-and-intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals).
- **pass^k**: The probability that all k independent attempts succeed: p^k. The right quantity for unattended or repeated work. Some sources label it pass@k; check which functional a leaderboard computed. See [/handbook/validation/estimands-and-intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals).
- **Process evaluation**: Judges intermediate steps, actions, states, or evidence acquisition. Justified when the path itself matters; harmful when it forces one canonical chain. See [/handbook/instrument/signal-and-resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution).
- **Process verifier**: Any process-resolution evaluator, whatever its mechanism family: formal, executable, learned, or human. Not a synonym for PRM. See [/handbook/instrument/signal-and-resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution).
- **Resolution** (R): The unit that is judged: token, step, action, final artifact, trajectory, or a system-level rate. See [/handbook/instrument/signal-and-resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution).
- **Robustness margin** (ρ): A signed or graded distance to satisfying a formal or quantitative specification; useful for partial credit, early warning, and search. See [/handbook/instrument/specification](https://benchmaker.ai/handbook/instrument/specification).
- **Signal** (Z): What leaves the evaluator: a boolean, scalar, probability, vector, preference, critique, confidence, or error trace. See [/handbook/instrument/signal-and-resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution).
- **Unbiased pass@k estimator**: For a task with c passes among n ≥ k samples, 1 − C(n − c, k)/C(n, k): the share of k-sample subsets containing a pass. Compute per task, then average; the plug-in 1 − (1 − c/n)^k reads low. See [/handbook/validation/estimands-and-intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals).

### Use and training pipelines

- **Bradley–Terry model**: A pairwise preference model, Pr[y_w ≻ y_l | x] = σ(r(x, y_w) − r(x, y_l)), used to train reward models from comparisons. Only score differences matter: it learns a preference ordering, not a proof of quality. See [/handbook/regime/training-loops](https://benchmaker.ai/handbook/regime/training-loops).
- **Consumer** (κ): Who or what acts on an evaluator's signal: a report reader, a selector, an optimizer, a release gate, or a runtime controller. The first variable of the regime tuple. See [/handbook/regime/exposure-and-surfaces](https://benchmaker.ai/handbook/regime/exposure-and-surfaces).
- **Cost and latency envelope** (K): What each evaluator call may cost and how fast it must answer in a given use. Training may need millions of cheap calls; certification, hundreds of expensive independent audits. See [/handbook/regime/exposure-and-surfaces](https://benchmaker.ai/handbook/regime/exposure-and-surfaces).
- **DPO**: Direct preference optimization: trains directly from preference pairs without a separately optimized reward-model-plus-RL loop. It changes the optimizer, not the measurement problem. See [/handbook/regime/training-loops](https://benchmaker.ai/handbook/regime/training-loops).
- **Exposure budget** (b): How much optimization pressure an evaluator absorbs, recorded in the optimizer's own unit (candidates, KL, developer iterations, queries, self-modification events), with the induced shift D(Q_b ‖ P_0) where measurable. There is no universal unit. See [/handbook/regime/exposure-and-surfaces](https://benchmaker.ai/handbook/regime/exposure-and-surfaces).
- **Information access** (A_access): What flows from the evaluator, reference anchor, or certification process back to the optimizer. Every revealed score, failure explanation, or hidden test spends independence; the anchor condition is A ∉ S with I(A → O) controlled. See [/handbook/regime/exposure-and-surfaces](https://benchmaker.ai/handbook/regime/exposure-and-surfaces).
- **KL-regularized objective**: The RLHF-style training objective: maximize expected reward minus β times the KL divergence from a reference policy. The KL term limits policy drift and is the natural exposure axis for this optimizer. See [/handbook/regime/training-loops](https://benchmaker.ai/handbook/regime/training-loops).
- **Modification surface** (S): Everything the optimizing loop can change: policy, prompts, tools, harness, tests, rubric, judge prompt, judge weights, reference anchors, environment. Trusted bases belong to it whenever the optimizer can touch them. See [/handbook/regime/exposure-and-surfaces](https://benchmaker.ai/handbook/regime/exposure-and-surfaces).
- **Optimizer** (O): Who or what adapts against an evaluator's signal: developers, best-of-N selection, search, reinforcement learning, or the system itself. A measuring signal has, by design, none. See [/handbook/regime/exposure-and-surfaces](https://benchmaker.ai/handbook/regime/exposure-and-surfaces).
- **Process supervision**: Supervision applied to intermediate procedure or steps rather than only the final outcome; outcome supervision derives its signal from terminal outcomes alone. A training use of a process-resolution signal, not a mechanism. See [/handbook/instrument/signal-and-resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution).
- **Regime migration**: When a signal's causal role changes, such as a benchmark becoming an optimization target or a monitor becoming a training signal. Its validation obligations move with it. See [/handbook/regime/use-regimes](https://benchmaker.ai/handbook/regime/use-regimes).
- **Release gate**: An evaluator that determines whether a system or version can ship: a score-to-decision mapping with a threshold. This handbook treats a pre-release gate as measure until developers iterate against it; a request-path threshold is runtime control. See [/handbook/regime/use-regimes](https://benchmaker.ai/handbook/regime/use-regimes).
- **Reporting (use)**: A use in which the evaluator supports descriptive comparison without directly selecting or training candidates. It belongs to the measure regime. See [/handbook/regime/use-regimes](https://benchmaker.ai/handbook/regime/use-regimes).
- **RLAIF**: RL-style alignment using AI-generated feedback instead of, or alongside, human feedback. See [/handbook/regime/training-loops](https://benchmaker.ai/handbook/regime/training-loops).
- **RLHF**: Reinforcement learning from human feedback: a training pipeline that consumes preference data, usually via a learned reward model and policy optimization. Not itself an evaluator family. See [/handbook/regime/training-loops](https://benchmaker.ai/handbook/regime/training-loops).
- **RLVR**: Reinforcement learning with automatically verifiable reward signals, usually exact, executable, or formal. Grounded but incomplete. See [/handbook/regime/training-loops](https://benchmaker.ai/handbook/regime/training-loops).
- **Runtime control**: An evaluator that can block, reroute, escalate, or roll back deployed behavior, rather than only observe it. See [/handbook/regime/production-monitoring](https://benchmaker.ai/handbook/regime/production-monitoring).
- **Search (use)**: A use in which the evaluator guides iterative generation or optimization. It belongs to the optimize regime. See [/handbook/regime/use-regimes](https://benchmaker.ai/handbook/regime/use-regimes).
- **Selection (use)**: A use in which the evaluator chooses among generated candidates, as in best-of-N. It belongs to the optimize regime. See [/handbook/regime/use-regimes](https://benchmaker.ai/handbook/regime/use-regimes).
- **Training reward (use)**: A use in which the evaluator's signal enters parameter updates, as in RLHF or RLVR. Of the four uses of a judge (reporting, selection, search, training), it applies the most optimization pressure. See [/handbook/regime/use-regimes](https://benchmaker.ai/handbook/regime/use-regimes).
- **Use regime**: The causal role a signal plays: measure (supports inference), optimize (shapes the candidate or policy distribution), or monitor-control (detects and intervenes in deployment). See [/handbook/regime/use-regimes](https://benchmaker.ai/handbook/regime/use-regimes).

### Validity and evidence

- **Adaptive validity**: Validity of an evaluator after the system it measures has been optimized against it by selection, search, training, or developer iteration. Measured with an exploitability curve; accuracy on ordinary outputs does not establish it. See [/handbook/pressure/adaptive-validity](https://benchmaker.ai/handbook/pressure/adaptive-validity).
- **Bias feature**: A legitimate, predictive cue in a benchmark, such as wording, answer length, or file naming, whose relation to the outcome does not hold across environments or challenge sets. One of Riezler and Hagmann's three validity pathologies. See [/handbook/claim/task-distributions](https://benchmaker.ai/handbook/claim/task-distributions).
- **Circular feature**: An input feature that deterministically defines the target, so the task reconstructs the annotation rule instead of evidencing the capability; for example, a task generator and its evaluator sharing a hidden template. See [/handbook/claim/task-distributions](https://benchmaker.ai/handbook/claim/task-distributions).
- **Consequential evidence**: Evidence about what happens because a score is used: once it becomes a gate, leaderboard target, or reward, does optimization against it degrade the relation between score and construct? See [/handbook/validation/validity-evidence](https://benchmaker.ai/handbook/validation/validity-evidence).
- **Construct validity**: The overall evidential argument that a measure behaves as expected for the target construct and not for competing explanations. See [/handbook/validation/validity-argument](https://benchmaker.ai/handbook/validation/validity-argument).
- **Contamination**: Leakage of evaluation items or close variants into training, prompts, tools, or fine-tuning, giving a construct-irrelevant advantage. Hygiene: task provenance, private or time-stamped holdouts, canaries, and post-cutoff splits. See [/handbook/claim/task-distributions](https://benchmaker.ai/handbook/claim/task-distributions).
- **Content evidence**: Evidence that the task blueprint covers the construct's domain and facets in justified proportions. It guards against construct underrepresentation: a text-to-SQL suite of simple filters cannot support a claim about enterprise SQL. See [/handbook/validation/validity-evidence](https://benchmaker.ai/handbook/validation/validity-evidence).
- **Convergent evidence**: Agreement among independent methods intended to measure the same construct, at the strength theory predicts. See [/handbook/validation/validity-evidence](https://benchmaker.ai/handbook/validation/validity-evidence).
- **Criterion evidence**: Relationship of scores to external outcomes such as production success, incident rate, or expert acceptance. Concurrent when measured at the same time, predictive when it forecasts later behavior; ecological evidence asks about the outcome users care about. See [/handbook/validation/validity-evidence](https://benchmaker.ai/handbook/validation/validity-evidence).
- **Discriminant evidence**: Evidence that scores do not mainly track nuisance variables or distinct constructs, such as length or rhetorical confidence. See [/handbook/validation/validity-evidence](https://benchmaker.ai/handbook/validation/validity-evidence).
- **Error families**: The four kinds of error kept apart along the instrument chain: representation error, behavioral or trial variation, scoring error, and estimation error. Wrong world and wrong property are structural failures, not error families. See [/handbook/foundations/evaluation-is-measurement](https://benchmaker.ai/handbook/foundations/evaluation-is-measurement).
- **Evaluator dossier**: The record an evaluator system ships with: sixteen fields from claim and regime through evidence, mechanisms, reliability, natural and adaptive validity, statistics, channels, validity domain, maintenance, reproducibility, and frame governance. See [/handbook/reference/templates](https://benchmaker.ai/handbook/reference/templates).
- **Face validity**: Informed plausibility that an evaluation appears to measure the intended construct. A smoke test, never a stopping rule. See [/handbook/validation/validity-evidence](https://benchmaker.ai/handbook/validation/validity-evidence).
- **Illegitimate feature**: A feature carrying target information the deployed system should not have, such as a gold file reachable through tools or task IDs correlated with labels. It alone predicts the label, even after arbitrary re-encoding. See [/handbook/claim/task-distributions](https://benchmaker.ai/handbook/claim/task-distributions).
- **Internal-structure evidence**: Evidence that score dimensionality, item relations, and aggregation behave as the construct theory predicts. It asks whether one headline number should exist at all, or whether a vector with explicit gates is more defensible. See [/handbook/validation/validity-evidence](https://benchmaker.ai/handbook/validation/validity-evidence).
- **Measurement invariance**: Whether the measurement relation remains stable across relevant groups, model families, languages, or conditions, so that the same score means the same thing in each. See [/handbook/validation/validity-evidence](https://benchmaker.ai/handbook/validation/validity-evidence).
- **Multitrait–multimethod design (MTMM)**: A design that measures several constructs through several methods, so that correlations that follow the trait can be separated from correlations that follow the method. A factor that appears under only one format is suspect. See [/handbook/validation/validity-evidence](https://benchmaker.ai/handbook/validation/validity-evidence).
- **Natural validity**: Validity of an evaluator's interpretation for ordinary measurement on the target distribution, before any system has been optimized against it. Contrast adaptive validity. See [/handbook/pressure/adaptive-validity](https://benchmaker.ai/handbook/pressure/adaptive-validity).
- **Outcome validity**: The degree to which grading recognizes actual task success and failure, rather than being fooled by weak tests, brittle matching, or incomplete state checks. See [/handbook/validation/task-and-outcome-validity](https://benchmaker.ai/handbook/validation/task-and-outcome-validity).
- **Response-process evidence**: Evidence that the system solves tasks through construct-relevant processes rather than artifacts or shortcuts: which causal route from capability to outcome produced the score? See [/handbook/validation/validity-evidence](https://benchmaker.ai/handbook/validation/validity-evidence).
- **Swiss cheese model**: Assurance from stacking validation methods, each a slice whose holes are its limitations. It works only if the holes do not line up, which joint miss rates across channels measure. See [/handbook/pressure/goodhart-and-self-confirming-loops](https://benchmaker.ai/handbook/pressure/goodhart-and-self-confirming-loops).
- **Task validity**: The degree to which tasks elicit the target capability rather than shortcuts, impossible setups, or irrelevant bottlenecks. See [/handbook/validation/task-and-outcome-validity](https://benchmaker.ai/handbook/validation/task-and-outcome-validity).
- **Transport**: Whether an evaluator's measured error rates carry over across generators, slices, time, and environments. A component of the validation profile; it is what licenses the validity domain. See [/handbook/pressure/validation-profile](https://benchmaker.ai/handbook/pressure/validation-profile).
- **Validation profile** (Π): The measured profile Π = (N, E, T, G, C, D): natural error, exploitability, transport, generator coupling, channel coupling, drift. It replaces hierarchy labels with measurements. See [/handbook/pressure/validation-profile](https://benchmaker.ai/handbook/pressure/validation-profile).
- **Validity**: The strength of evidence and theory supporting a particular interpretation and use of scores. Not a permanent property of a benchmark: validity is a claim, not a sticker. See [/handbook/validation/validity-argument](https://benchmaker.ai/handbook/validation/validity-argument).
- **Validity argument**: The chain of claims behind an interpretation and use of a score: construct, representation, elicitation, scoring, generalization (task, seed, judge, environment, generator, time), and decision. Validity attaches to the argument, not a benchmark. See [/handbook/validation/validity-argument](https://benchmaker.ai/handbook/validation/validity-argument).
- **Validity domain** (D): The declared conditions and population over which an interpretation is justified. Outside it, a benchmark has made no justified claim. See [/handbook/claim/validity-domain](https://benchmaker.ai/handbook/claim/validity-domain).
- **Validity statement**: A falsifiable statement of scope: evaluator system E supports claim C for consumer κ over domain D, under regime R, up to exposure b*, with profile Π, relative to trust root T. See [/handbook/pressure/validation-profile](https://benchmaker.ai/handbook/pressure/validation-profile).
- **Verification, validation, evaluation, testing**: Verification gives guarantees against a formal specification; validation asks whether a system behaves as desired in the real world; evaluation is the concrete act of measurement; testing evaluates a system on a discrete set of cases. See [/handbook/foundations/evaluation-is-measurement](https://benchmaker.ai/handbook/foundations/evaluation-is-measurement).

### Reliability, calibration, and statistics

- **Alert precision**: The share of a monitor's alerts that are real, Pr[danger | alert] = pt / (pt + (1 − p)f), for sensitivity t, false-positive rate f, and base rate p. When p is tiny, even a small f can dominate. See [/handbook/regime/production-monitoring](https://benchmaker.ai/handbook/regime/production-monitoring).
- **Beta-binomial posterior**: A Bayesian model for a pass rate: a Beta(α, β) prior updated on s passes in n tasks gives a Beta(α + s, β + n − s) posterior, which answers release questions such as Pr[pass rate > 95%]. See [/handbook/validation/estimands-and-intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals).
- **Calibration**: Agreement between predicted confidence or score and empirical frequencies of correctness. See [/handbook/regime/production-monitoring](https://benchmaker.ai/handbook/regime/production-monitoring).
- **Clustered trials**: Repeated trials of the same task, whose outcomes are correlated. Treating n tasks × k trials as nk independent samples understates uncertainty; resample whole tasks or fit a hierarchical model. See [/handbook/validation/estimands-and-intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals).
- **Cohen's κ**: Chance-corrected agreement, (p_o − p_e) / (1 − p_e). It depends on prevalence: identical sensitivity and specificity give very different values at different base rates, so report the confusion matrix and prevalence with it. See [/handbook/validation/judge-as-noisy-sensor](https://benchmaker.ai/handbook/validation/judge-as-noisy-sensor).
- **Conformal prediction**: A distribution-free method giving finite-sample coverage or miss-rate guarantees under exchangeability, which production drift can violate. See [/handbook/regime/production-monitoring](https://benchmaker.ai/handbook/regime/production-monitoring).
- **Covariate-shift weight** (w(x)): The ratio of a task's probability under the target population to its probability under the eval sample, used to reweight results to deployment. Weighting cannot create evidence where the sample has no support. See [/handbook/validation/estimands-and-intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals).
- **Differential misclassification**: Evaluator error rates that differ between compared systems, for example because one was optimized against the evaluator. One shared calibration then biases corrections and comparisons; audit error per system or at least per generator family. See [/handbook/validation/judge-as-noisy-sensor](https://benchmaker.ai/handbook/validation/judge-as-noisy-sensor).
- **Discordant pairs**: In a paired comparison of two systems on the same tasks, the tasks on which exactly one system passes. Only discordant pairs carry information about which system is better; McNemar's test uses only them. See [/handbook/validation/comparing-systems](https://benchmaker.ai/handbook/validation/comparing-systems).
- **Estimand**: The exact quantity an evaluation sets out to estimate: the outcome, the population it ranges over, and the functional (single-trial rate, pass@k, pass^k, a tail quantile, a paired difference). Many statistical mistakes are estimand mistakes. See [/handbook/validation/estimands-and-intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals).
- **Estimation set**: Tasks sampled, or deliberately over-sampled and then reweighted, to answer a population-level question such as the production pass rate. Kept apart from a regression set that over-samples known failures. See [/handbook/claim/task-distributions](https://benchmaker.ai/handbook/claim/task-distributions).
- **Family-wise false-alarm rate**: The probability that at least one of m gates fires on a change that does nothing: 1 − (1 − α)^m for independent gates at level α, about 26% for six gates at α = .05. See [/handbook/validation/comparing-systems](https://benchmaker.ai/handbook/validation/comparing-systems).
- **Generalizability coefficient** (Φ): Generalizability theory's absolute coefficient: the variance attributable to the object of measurement, divided by that variance plus the variance from sources that should not change the interpretation, Φ = σ²_object / (σ²_object + σ²_Δ). See [/handbook/validation/reliability](https://benchmaker.ai/handbook/validation/reliability).
- **Generalizability theory**: A variance-component framework that assigns measurement variance to facets such as task, judge, seed and environment, and their interactions. See [/handbook/validation/reliability](https://benchmaker.ai/handbook/validation/reliability).
- **Importance sampling**: Reweighting samples from a proposal distribution q to estimate expectations under the target p, by the likelihood ratio p(x)/q(x). For rare failures q over-produces failures; it must cover the failure region. See [/handbook/validation/rare-events](https://benchmaker.ai/handbook/validation/rare-events).
- **Kish effective sample size** (n_eff): n_eff = (Σ w_i)² / Σ w_i²: the number of equally weighted items that carry the same information as a weighted sample. A collapse of n_eff relative to n signals a suite that badly mismatches the target population. See [/handbook/validation/estimands-and-intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals).
- **Linking items**: Tasks carried unchanged across versions of a refreshed eval suite, with overlap samples, so that scores on different versions can be linked. Some sources call them anchor tasks; this handbook reserves 'anchor' for reference anchors and the anchor channel. See [/handbook/lifecycle/living-eval-suites](https://benchmaker.ai/handbook/lifecycle/living-eval-suites).
- **McNemar's test**: A paired test for binary outcomes: χ² = (b − c)²/(b + c), where b and c count the tasks that only one of the two systems passes. Use the exact binomial version when b + c is small. See [/handbook/validation/comparing-systems](https://benchmaker.ai/handbook/validation/comparing-systems).
- **Mixed-effects significance**: Testing a difference between systems with a model in which tasks, rollouts, judges, and environment versions are variance components, asking whether the system effect survives the measured experimental variation. See [/handbook/validation/comparing-systems](https://benchmaker.ai/handbook/validation/comparing-systems).
- **Multilevel splitting**: Rare-event estimation by factorizing a very small failure probability into a product of larger conditional probabilities across severity levels, each estimated among the survivors of the previous level. See [/handbook/validation/rare-events](https://benchmaker.ai/handbook/validation/rare-events).
- **Optional stopping**: Checking a comparison repeatedly and stopping once it looks significant. Each look is another chance for noise to cross the threshold, so false positives inflate; use a fixed sample size or an explicitly sequential design. See [/handbook/validation/comparing-systems](https://benchmaker.ai/handbook/validation/comparing-systems).
- **Regression set**: Tasks that deliberately over-sample known failures to support fast iteration. Its pass rate is not a population estimate; keep it apart from an estimation set sampled or weighted to the population. See [/handbook/claim/task-distributions](https://benchmaker.ai/handbook/claim/task-distributions).
- **Reliability**: Stability of measurement across legitimate replications. Necessary but not sufficient: a perfectly repeatable evaluator can reliably measure the wrong construct. See [/handbook/validation/reliability](https://benchmaker.ai/handbook/validation/reliability).
- **Reliability diagram**: A plot of the observed event rate against a monitor's or judge's predicted score, by score bin, used with recalibration on held-out data. For a calibrated score, about 70% of events scored 0.7 are real. See [/handbook/regime/production-monitoring](https://benchmaker.ai/handbook/regime/production-monitoring).
- **Replication universe**: The declared set of conditions that may legitimately vary across replications: tasks, rollouts, judges, raters, environment snapshots, harness versions, time. The intended claim decides which variation is measurement error and which is part of the target. See [/handbook/validation/reliability](https://benchmaker.ai/handbook/validation/reliability).
- **Rogan–Gladen correction**: Inverts a binary sensor model to estimate the true pass rate from an imperfect judge: p̂ = (p_obs − FPR) / (TPR − FPR), assuming error rates transport. See [/handbook/validation/judge-as-noisy-sensor](https://benchmaker.ai/handbook/validation/judge-as-noisy-sensor).
- **Rule of three**: After n independent trials with zero failures, a rough 95% upper bound on the failure probability is 3/n. Zero failures is not zero risk. See [/handbook/validation/rare-events](https://benchmaker.ai/handbook/validation/rare-events).
- **Sensitivity and specificity (TPR, TNR, FPR)**: A binary evaluator's error rates against a reference: sensitivity (TPR) is the probability it accepts a true positive, specificity (TNR) the probability it rejects a true negative, and the false-positive rate FPR = 1 − TNR. See [/handbook/validation/judge-as-noisy-sensor](https://benchmaker.ai/handbook/validation/judge-as-noisy-sensor).
- **Statistical power**: The probability that a test detects a real difference of a stated size. A low-power comparison that comes back "not significant" is weak evidence of absence, not evidence of no effect. See [/handbook/validation/comparing-systems](https://benchmaker.ai/handbook/validation/comparing-systems).
- **Stratified estimate**: A pass rate combined from per-stratum estimates with the target population's weights, Σ w_h p̂_h. It lets a suite over-sample informative slices and still report the deployment mix. See [/handbook/validation/estimands-and-intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals).
- **VaR / CVaR**: Tail-risk functionals: the α-quantile of a loss, and the expected loss beyond it. Useful when severe outcomes matter more than the mean. See [/handbook/instrument/aggregation-geometry](https://benchmaker.ai/handbook/instrument/aggregation-geometry).
- **Variance components**: The shares of observed score variance attributable to each facet (tasks, rollouts, judges, raters, environment snapshots) and their interactions, estimated with mixed-effects models. Published per score, they form a variance budget. See [/handbook/validation/reliability](https://benchmaker.ai/handbook/validation/reliability).
- **Wilson interval**: A confidence interval for a binomial proportion that behaves well near 0 and 1 and at small n. The default interval for pass rates. See [/handbook/validation/estimands-and-intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals).

### Optimization and failure dynamics

- **Adversary regime**: Who chooses the inputs a falsification search targets. Against nature (ordinary users), weight the search by plausibility; against a strategic adversary (jailbreakers, injectors), drop the weight, because the existence of a worst case is what matters. See [/handbook/pressure/falsification](https://benchmaker.ai/handbook/pressure/falsification).
- **Anchor burn**: Loss of a holdout's or anchor channel's independence as its information is fed back into optimization. Track it in an anchor-burn ledger and refresh exposed items. See [/handbook/lifecycle/channels-and-independence](https://benchmaker.ai/handbook/lifecycle/channels-and-independence).
- **Anchor leakage**: Hidden or certification evidence repeatedly flows back into optimization, eroding its independence. Keep an anchor-burn ledger and refresh the anchor channel once it is no longer independent. See [/handbook/practice/anti-patterns#anchor-leakage](https://benchmaker.ai/handbook/practice/anti-patterns#anchor-leakage).
- **Channel coupling**: Dependence among supposedly independent assurance channels, measured by joint miss rates: does one channel miss what the other misses? High coupling means the Swiss-cheese holes line up. See [/handbook/lifecycle/channels-and-independence](https://benchmaker.ai/handbook/lifecycle/channels-and-independence).
- **Common-mode assurance**: Supposedly independent evaluators share a model family, data, prompts, or evidence and therefore fail together. Measure channel coupling and diversify mechanisms across development and certification. See [/handbook/practice/anti-patterns#common-mode-assurance](https://benchmaker.ai/handbook/practice/anti-patterns#common-mode-assurance).
- **Correlated trials**: Shared infrastructure, dependencies, or state make trial outcomes co-move, invalidating independent-sample uncertainty calculations. Isolate trials and resample at the task level. See [/handbook/practice/anti-patterns#correlated-trials](https://benchmaker.ai/handbook/practice/anti-patterns#correlated-trials).
- **Divergence tripwire**: A predeclared monitor that fires on a named warning pattern: O improving while A is flat or falling (proxy corruption), O and A improving while P is not (construct mismatch), saturation, or repeated success on work that no longer matters. See [/handbook/lifecycle/living-eval-suites](https://benchmaker.ai/handbook/lifecycle/living-eval-suites).
- **Diversity collapse**: Homogenization of behavior or tasks in self-play or self-generated-data loops, which reduces useful variation. Track diversity explicitly and refresh task sources. See [/handbook/pressure/goodhart-and-self-confirming-loops](https://benchmaker.ai/handbook/pressure/goodhart-and-self-confirming-loops).
- **Drift**: Change in task, environment, user, generator, or evaluator distribution that invalidates earlier calibration or validity evidence. See [/handbook/lifecycle/living-eval-suites](https://benchmaker.ai/handbook/lifecycle/living-eval-suites).
- **Eval theater**: Evaluation disconnected from decisions: scores are displayed but never change action. Write the decision first, and ask when the eval last changed one. See [/handbook/practice/anti-patterns#eval-theater](https://benchmaker.ai/handbook/practice/anti-patterns#eval-theater).
- **Exploitability curve**: Disagreement between an evaluator and an independent reference anchor as a function of optimization exposure b. b* is the largest tested exposure before divergence exceeds a predeclared tolerance. See [/handbook/pressure/adaptive-validity](https://benchmaker.ai/handbook/pressure/adaptive-validity).
- **Failure region**: The set of inputs and conditions around a found failure that also fail, described by the features that define it. Mapping the region, not collecting anecdotes, turns one counterexample into evidence about risk. See [/handbook/pressure/falsification](https://benchmaker.ai/handbook/pressure/falsification).
- **Falsification**: The search for inputs and disturbances that make a system, or its evaluator, violate the specification: can it fail at all, and where? A found failure is evidence of existence, not of frequency. See [/handbook/pressure/falsification](https://benchmaker.ai/handbook/pressure/falsification).
- **Frame lock-in**: A system keeps optimizing a stable criterion after the real objective has changed. It needs external frame review, not a better judge. See [/handbook/practice/anti-patterns#frame-lock-in](https://benchmaker.ai/handbook/practice/anti-patterns#frame-lock-in).
- **Generator coupling**: Dependence between the generator's failure modes and the evaluator's, measured as Pr[E errs | G errs] versus Pr[E errs], or by comparing false-accept rates on the generator's failures with those on other sources' failures. See [/handbook/pressure/goodhart-and-self-confirming-loops](https://benchmaker.ai/handbook/pressure/goodhart-and-self-confirming-loops).
- **Goodhart pressure**: Degradation of the proxy-to-construct relation caused by optimization against the proxy. See [/handbook/pressure/goodhart-and-self-confirming-loops](https://benchmaker.ai/handbook/pressure/goodhart-and-self-confirming-loops).
- **Grader gaming (bypass)**: A candidate satisfies the scorer without doing the intended work: hard-coded answers, fabricated citations, edited tests. Red-team the grader and constrain the routes to a pass. See [/handbook/practice/anti-patterns#grader-gaming](https://benchmaker.ai/handbook/practice/anti-patterns#grader-gaming).
- **Judge rot**: The judge's prompt, model, or settings change without versioning and recalibration, and every downstream number inherits the error. Version the whole measurement and revalidate on change. See [/handbook/practice/anti-patterns#judge-rot](https://benchmaker.ai/handbook/practice/anti-patterns#judge-rot).
- **Model collapse**: Recursive training on generated data degrades distributional quality or capability, starting with the tails of the distribution. Gate data and ground rewards externally. See [/handbook/pressure/goodhart-and-self-confirming-loops](https://benchmaker.ai/handbook/pressure/goodhart-and-self-confirming-loops).
- **One-number reporting**: An aggregate score without sample size, interval, slices, or trend, which hides the measurement process. Report n, intervals, slices, and trends. See [/handbook/practice/anti-patterns#one-number-reporting](https://benchmaker.ai/handbook/practice/anti-patterns#one-number-reporting).
- **Overoptimization**: Continued optimization that improves the proxy score after true or reference-anchor quality has plateaued or declined. Gao et al. demonstrated it for learned reward models. See [/handbook/pressure/goodhart-and-self-confirming-loops](https://benchmaker.ai/handbook/pressure/goodhart-and-self-confirming-loops).
- **Reward hacking**: Behavior that obtains high measured reward while violating the intended objective; a special case of evaluator exploitation. See [/handbook/pressure/goodhart-and-self-confirming-loops](https://benchmaker.ai/handbook/pressure/goodhart-and-self-confirming-loops).
- **Rigid trajectory grading**: Grading that requires one canonical sequence of steps when the process is not part of the construct, so valid novel strategies fail. Grade observable constraints, not one path. See [/handbook/practice/anti-patterns#rigid-trajectory-grading](https://benchmaker.ai/handbook/practice/anti-patterns#rigid-trajectory-grading).
- **Saturation**: An instrument ceiling where stronger systems can no longer be discriminated. Treat it as a maintenance event. See [/handbook/lifecycle/living-eval-suites](https://benchmaker.ai/handbook/lifecycle/living-eval-suites).
- **Saturation blindness**: A benchmark near its ceiling loses discrimination, and regressions hide in rounded aggregates. Treat saturation as a maintenance event, not a success. See [/handbook/practice/anti-patterns#saturation-blindness](https://benchmaker.ai/handbook/practice/anti-patterns#saturation-blindness).
- **Self-confirming loop**: Generator and evaluator share correlated blind spots, so selection or training amplifies common errors. It needs no adversary, only correlated bias. See [/handbook/pressure/goodhart-and-self-confirming-loops](https://benchmaker.ai/handbook/pressure/goodhart-and-self-confirming-loops).
- **Tested exposure** (b*): The largest optimization exposure tested before the evaluator and an independent reference anchor diverge beyond a predeclared tolerance. It makes adaptive validity expire: report it with the tested range, never as robustness beyond it. See [/handbook/pressure/adaptive-validity](https://benchmaker.ai/handbook/pressure/adaptive-validity).
- **Versioned dynamism**: Dynamic corpus, static run: refresh an eval suite deliberately across versions, but freeze every environment snapshot, tool contract, scorer, and dependency needed to reproduce a reported result. A new snapshot is a new measurement form. See [/handbook/lifecycle/living-eval-suites](https://benchmaker.ai/handbook/lifecycle/living-eval-suites).
- **Wireheading**: Extreme reward hacking in which the system manipulates the reward-producing process itself rather than doing the task. The defense is architectural: keep evaluator artifacts outside the modification surface. See [/handbook/pressure/goodhart-and-self-confirming-loops](https://benchmaker.ai/handbook/pressure/goodhart-and-self-confirming-loops).

### RSI and self-improvement

- **Auto Research**: AI carrying out the research process itself — proposing hypotheses, running experiments, discovering algorithms — including research that can improve methods, systems, and evaluators. See [/handbook/agents/self-improvement](https://benchmaker.ai/handbook/agents/self-improvement).
- **Bounded self-refinement**: Self-improvement against a fixed external evaluator, so the process is at least conceptually convergent and evaluable. See [/handbook/agents/self-improvement](https://benchmaker.ai/handbook/agents/self-improvement).
- **Closed loop**: A self-improvement loop in which the system generates, validates, and applies its own improvements without human review. Open-ended RSI is a closed loop that can also modify its own evaluator. See [/handbook/agents/self-improvement](https://benchmaker.ai/handbook/agents/self-improvement).
- **Deployment-time self-evolution**: Self-improvement during deployment: refining outputs with frozen weights, adapting weights per query (test-time training), or evolving the harness, skills, or memory. See [/handbook/agents/self-improvement](https://benchmaker.ai/handbook/agents/self-improvement).
- **Evaluator co-evolution**: Tests, rubrics, or judges that evolve together with the policy they supervise. Co-evolution may escape a self-confirming loop or add a second story to it, so it needs a fixed anchor channel outside the loop. See [/handbook/agents/self-improvement](https://benchmaker.ai/handbook/agents/self-improvement).
- **Frame review**: The governance process that asks whether the construct and objective themselves remain appropriate — logically prior to verifying performance against them. See [/handbook/lifecycle/frame-review](https://benchmaker.ai/handbook/lifecycle/frame-review).
- **Governance-grade measurement**: Auditable evidence about what a training or self-improvement loop is and is not improving, of the kind credible oversight and slowdown proposals require. The RSI survey finds almost no technical work on it. See [/handbook/agents/self-improvement](https://benchmaker.ai/handbook/agents/self-improvement).
- **Grounding exchange rate**: The minimum rate of external, grounded signal a self-improvement loop needs to keep improving rather than degrading. Pure closed loops degrade, but the rate has not been characterized; an open problem. See [/handbook/agents/self-improvement](https://benchmaker.ai/handbook/agents/self-improvement).
- **Human-in-the-loop**: A self-improvement loop in which a person reviews each change before it is applied. Contrast with human-on-the-loop (humans audit outcomes and gate deployment) and closed loop (no human review). See [/handbook/agents/self-improvement](https://benchmaker.ai/handbook/agents/self-improvement).
- **Human-on-the-loop**: An automatic signal drives the loop, while humans audit outcomes and gate deployment. Contrast with human-in-the-loop (a human reviews each change) and closed loop (no human review). See [/handbook/agents/self-improvement](https://benchmaker.ai/handbook/agents/self-improvement).
- **Non-verifiable frontier**: Tasks with no strong external verifier, such as creative quality, dialogue, research taste, and direction-setting, where improvement loops lack a reliable signal. Usually the ground truth is costly, plural, or unavailable, not unverifiable in principle. See [/handbook/agents/self-improvement](https://benchmaker.ai/handbook/agents/self-improvement).
- **O/A/P channels** (O / A / P): Three separated measurement channels: an optimized channel (O), cheap, dense, and allowed to co-evolve; an anchor channel (A) that certifies independently and stays out of training objectives; and a production channel (P) of real outcomes and incidents. See [/handbook/lifecycle/channels-and-independence](https://benchmaker.ai/handbook/lifecycle/channels-and-independence).
- **Open-ended RSI**: Recursive self-improvement that can modify not only the system but the machinery or criteria of improvement, so no external evaluator stays permanently fixed. See [/handbook/agents/self-improvement](https://benchmaker.ai/handbook/agents/self-improvement).
- **Self-evaluation**: Self-improvement whose object is the evaluator itself: designing, strengthening, or co-evolving the judges, verifiers, reward models, and rubrics that define "better" for other loops. See [/handbook/agents/self-improvement](https://benchmaker.ai/handbook/agents/self-improvement).
- **Self-improvement**: A system participating in producing a better version of itself or its outputs, where an evaluator defines "better". The evaluator is load-bearing: no improvement claim means anything without a criterion that separates better from worse. See [/handbook/agents/self-improvement](https://benchmaker.ai/handbook/agents/self-improvement).
- **Self-play**: Training in which the model generates its own tasks, opponents, or curricula, including proposer–solver loops and zero-data regimes that start from a base model with no external task set. See [/handbook/agents/self-improvement](https://benchmaker.ai/handbook/agents/self-improvement).
- **Test-time training (TTT)**: Updating model weights during deployment, conditioned on the current query or session, without a separate curated training phase. Distinct from inference-time refinement, where weights stay fixed. See [/handbook/agents/self-improvement](https://benchmaker.ai/handbook/agents/self-improvement).
- **Training-time self-iteration**: Self-improvement in which self-generated data, rewards, or teacher signals update the model's weights in a training phase. See [/handbook/agents/self-improvement](https://benchmaker.ai/handbook/agents/self-improvement).


---
Source: https://benchmaker.ai/handbook/reference/glossary