The validation profile and validity statement

The validation profile $\Pi = (N, E, T, G, C, D)$ is the measured record of how a configured evaluator errs: natural error, exploitability curve, transport, generator coupling, channel coupling, and drift profile. It replaces a hierarchy label with measurements, and a validity statement turns it into a falsifiable claim of scope.

The rung is a prior#

The verification hierarchy ranks mechanisms by expected grounding strength, from formal verification down to intrinsic signals (The verification hierarchy, demoted). The framework is explicit about what that ranking is worth: treat hierarchy position as a prior over expected error structure, not as the final verdict on evaluator quality. What matters operationally is the measured error profile of the configured evaluator in its intended regime.

The prior can mislead in both directions. A sound proof checker certifies whatever theorem statement it was given, and only relative to its configured trusted base; a learned judge two rungs lower can have an error profile measured well enough to support a reporting claim. The rung says where to look first; the profile says what you found.

Six components#

The framework asks for a measured profile beside every rung:

$$\Pi = (N, E, T, G, C, D)$$

where:

• $N$ is the natural error profile on the target distribution;
• $E$ is the exploitability curve: divergence between the evaluator and an independent reference anchor as optimization exposure grows;
• $T$ is transport of error rates across generators, slices, time, and environments;
• $G$ is generator coupling: whether the evaluator's errors concentrate on the generator's;
• $C$ is channel coupling: dependence among supposedly independent assurance channels;
• $D$ is the drift profile: how calibration and discrimination degrade over time, and when the instrument saturates.

In words: how the evaluator errs today, under optimization, elsewhere, alongside the system it judges, alongside its backups, and over time.

Each component has its own measurement; each name links to the chapter that explains it:

Transport#

Transport is whether an evaluator's measured error rates carry over to the places it is used: other generator families, slices, months, and environment versions. Judge calibration should test transport across generator families and time before the judge is trusted, and the noisy-sensor correction fails when calibration error rates do not transport. Measure it by re-estimating error rates on each slice the claim covers; a corrected pass rate that must be clipped to stay between 0 and 1 is one symptom that they do not carry over (The judge as a noisy sensor).

Transport evidence is what licenses the validity domain, which declares the conditions where the validation evidence is claimed to transport. Every condition in the domain should be one where transport was checked, or be marked as assumed.

The validity statement#

The framework asks every serious evaluator to ship with a falsifiable statement of scope, written to a template:

Evaluator system $E$ supports claim $C$ for consumer $\kappa$ over validity domain $D$, under optimizer/access regime $R$, up to tested exposure $b^{*}$, with measured validation profile $\Pi$, relative to reference authority/trust root $T$.

Each slot does a job:

This handbook adds a closing clause to the template: an explicit statement of what remains human judgment, naming the criteria, cases, and revisions the evaluator system does not decide.

Trust roots#

The last slot ends a regress: an evaluator is validated against something that needs validating too, and the trust root is where the chain stops. The framework names two kinds. For objective formal properties, the trust root may be a configured proof kernel plus axioms and hardware and software assumptions. For normative open-ended quality, it may be an expert panel with measured reliability and an explicit governance mandate.

Naming a panel as the trust root does not exempt it from measurement. Human judgment is both a measurement mechanism with ordinary error and the reference authority where the chain ends, and an expert panel can define a standard while still showing measurable disagreement. So the slot names the panel, its measured agreement, and its mandate: who gave it authority over the standard, and who may revise it (Frame review and governance).

Statements expire#

A validity statement is dated evidence, not a certificate. The tested exposure $b^{*}$ makes adaptive validity expire: a stronger optimizer, a larger budget, or a new policy checkpoint moves the system outside the tested range. Drift does the same to the rest of the profile, because a change in tasks, environment, users, generators, or the evaluator can invalidate earlier calibration and validity evidence. Optimization changes the instrument: leaderboard and release-gate pressure make validity a lifecycle property rather than a one-time certification.

So write the expiry down with the statement: revalidate after material model, judge, harness, task, environment, or user changes, and review the frame when metrics improve but real outcomes do not. If anyone starts optimizing against the signal, the statement's regime no longer holds, and the statement is void until adaptive validity has been measured (Use regimes: same judge, four jobs).

Three worked statements#

A statement written before the evidence exists is useful as a plan, provided it says so. A measured slot carries numbers with n and intervals; a planned slot says what will be measured and that it has not been. A statement that mixes the two without marking them reads as evidence it is not; the coding-agent case writes its statement as a plan.

No measurements exist for these three systems, so each statement below is a plan; the deployments are illustrative and follow this handbook's Cases A, C, and F and their Studio presets. Each statement covers one use; the case chapters (A, C, and F) state the statement for the whole system.

Math RLVR (optimize).

The RLVR reward and certification stack — a symbolic final-answer checker with answer normalization, and a proof assistant for the formalized subset — is designed to support the claim that rewarded and certified solutions are mathematically correct, not artifacts of formatting or checker quirks, for the RL trainer and the research leads' checkpoint-release decision, over English competition problems released after the training cutoff, with numeric or short symbolic answers or short formalizable proofs, under on-policy RL with the checker as reward and with the certification set and the checker's configuration outside the modification surface, up to a tested exposure $b^{*}$ that has not yet been measured (planned: the checker's disagreement with an independent reference anchor, re-estimated on policy outputs at every checkpoint), with a validation profile that is planned rather than measured (false accepts on adversarial normal-form cases; agreement with an independent computer algebra system on sampled policy outputs), relative to the proof kernel and its configured trusted base, and to two mathematicians who adjudicate notation and proof quality. Proof quality beyond formal validity, and whether each formal statement captures the intended problem, remain human judgment.

SQL agent (measure).

The SQL-agent rollout evaluator — execution on a pinned snapshot of the finance warehouse with semantic result-set equality and metamorphic tests, read-only credentials and row-level permissions, and a rubric-guided judge for ambiguity handling — is designed to support the claim that the agent answers analysts' questions correctly without unauthorized reads or writes, for the head of analytics' rollout decision, over English questions about the finance warehouse at schema v12 and the analyst permission level, under developer iteration on prompts and tools against a separate development split, with only aggregate scores from a hidden certification set queried once per release, up to a tested exposure $b^{*}$ that has not yet been measured (planned: development-split and certification scores compared after each release cycle), with a validation profile that is planned rather than measured (checker and judge error against SME adjudication, by criterion, with Wilson intervals; a check that the score is not mostly a proxy for memorized schema names or output formatting; rank stability across alternate snapshots), relative to a panel of finance subject-matter experts whose agreement is still to be measured. Business-semantic edge cases and changes to the access policy remain human judgment.

Safety monitor (monitor-control).

The support agent's safety stack — structural permissions that make unapproved destinations unreachable, a trajectory check that no message is sent and no refund above the limit is issued before a recorded confirmation, and a learned monitor for residual policy violations — is designed to support the claim that unauthorized external messages and refunds are prevented or flagged, with alerts that mark real violations at an acceptable false-alarm load, for the permission layer and on-call's block-or-allow decision, over English support conversations for the consumer product under the current tool and policy versions, under monitor-control with no optimizer by design, because the agent's training never sees monitor scores or alerts, and so with no tested exposure to report, with a validation profile that is planned rather than measured (recall and alert precision at the production base rate from adjudicated alerts; false-alarm cost; a conformal flag threshold with its exchangeability caveat; drift checks per tool version), relative to the trust-and-safety team's adjudication of flagged cases, with reviewer agreement still to be measured. Novel policy interpretations and every escalated incident remain human judgment.

Each statement carries its own expiry. The math statement's $b^{*}$ belongs to one optimizer and one training run; the SQL statement lapses when the schema moves past v12; the monitor's statement is void the day its alerts reach the agent's training, which moves the monitor into the optimize regime and calls for adaptive-validity tests.

The builder below loads a shorter SQL-agent statement with illustrative values. Like the one above, it marks $b^{*}$ and $\Pi$ planned, the honest state until the studies have run, so it reads "not yet tested" and "not yet measured". Flip either to measured and see how much more the sentence then claims, and what evidence would have to stand behind it.

In practice#

Write the validity statement and its profile together.

1. Name the configured system. List mechanisms, versions, prompts, thresholds, and trusted bases: everything one measurement version covers.
2. State the claim, consumer, and domain. Write the claim as a construct and a decision, name who consumes the signal, and draw the domain no wider than the conditions where error rates were checked.
3. Record the regime. Name the optimizer, what it can see and modify, and the unit of exposure.
4. Fill Π component by component. For each of the six, write the measurement with n and an interval, or "planned" and what will be measured.
5. Name the trust root. For a panel, its measured agreement and mandate; for a proof kernel, its configuration.
6. Close with the human-judgment clause. Say which criteria, cases, and revisions stay with people.
7. Attach expiry triggers. List what voids the statement: an optimizer or budget beyond $b^{*}$, drift past a threshold, a judge or harness change, or a regime migration.

The artifact is a validity statement with its Π table, every entry marked measured or planned. It is the core of the evaluator dossier (Templates) and the first thing to ask for when auditing an eval.

Common failure modes#

• Rung as verdict — "it is execution-based, so we trust it" → report the measured profile of this configuration in this regime. See The verification hierarchy, demoted.
• A plan read as evidence — the statement sounds measured while Π's cells are empty → mark every component measured or planned, with n and an interval on each measured one. See one-number reporting.
• A timeless statement — no $b^{*}$ or expiry triggers, so the statement outlives the optimizer, judge, and data behind it → record the tested range and revalidation triggers. See judge rot.
• An unmeasured trust root — "validated against experts", with no agreement figures or mandate → name the panel, measure its agreement, and record its authority.

Check yourself#

Analyze. A model card says that its safety classifier "has been validated, with 95% accuracy". Map the sentence onto the slots of a validity statement and list the slots it leaves empty.

Hint

95% accuracy is at most part of natural error, with no n, interval, slice, or base rate. Every other slot is empty.

Evaluate. A vendor sells a judge as "robust to Goodhart and validated against experts". Which slots could support each phrase, and what would you ask for?

Hint

"Robust to Goodhart" needs $b^{*}$ with its tested range, the optimizer, and the reference anchor. "Validated against experts" needs natural error by slice with n, plus the panel's own agreement and mandate.

Create. Write a validity statement for an evaluator you rely on. Mark every component of Π measured or planned, add the human-judgment clause, and list three events that would void the statement.

Hint

Start from the regime. If anything optimizes against the signal, the exploitability curve cannot be "not applicable".

Further reading#

• The validity argument — the six claims a profile's measurements support.
• Adaptive validity — measuring the exploitability curve and $b^{*}$.
• Templates — the copy-ready statement and the 16-field dossier.
• Auditing an eval — reading someone else's statement, or writing the missing one.