Templates

This page holds the three artifacts an evaluator system should ship with: a one-sentence validity statement, the sixteen-field evaluator dossier, and the fifteen-item minimum reporting checklist. The text explains how they fit together and how to fill them honestly; the templates themselves follow at the end, ready to copy as Markdown.

How the three fit together#

The evaluator dossier is the full record. It has sixteen fields that a professional evaluator system should ship with, from the claim and the regime to reproducibility and frame governance. Many chapters of this handbook end with an artifact that fills one of its fields.

The validity statement summarizes the dossier in one sentence. The framework asks every serious evaluator to ship with a falsifiable statement of scope, and each slot of the sentence condenses one or more dossier fields. Read the correspondence this way:

Write one statement per use. The same dossier can support a release gate and a training reward, but the consumer, the regime, and the tested exposure differ between them, so each use gets its own sentence (The One Spine).

The minimum reporting checklist audits a report. The framework asks every evaluator-system release or major revision to report at least its fifteen items, and each item checks that a report covers the matching dossier field. Two pairings are not obvious from the names: item 3, on environment, harness, and tool versions, draws on the reproducibility field, and item 15 covers both maintenance and frame governance. Use it on your own report before release, and on someone else's when auditing an eval.

Fill the statement honestly#

A validity statement written before its evidence exists is still useful, as a plan, provided it says so (The validation profile and validity statement). Five habits keep it honest:

1. Mark $b^{*}$ and $\Pi$ measured or planned. A measured slot carries numbers with n and an interval, as in an illustrative "72% (95% Wilson CI 63–80%, n = 100)". A planned slot says what will be measured and that it has not been. Until the profile is measured, drop "measured" from the template's "with measured validation profile".
2. State $b^{*}$ only inside the tested range. If the largest budget tested still held, the honest entry is "at least" that budget, not "robust" (Adaptive validity). For a use that nothing optimizes against, say so; if anyone starts, the statement is void until adaptive validity has been measured.
3. Give the trust root its evidence. Name the proof kernel and its configured trusted base, or the expert panel with its measured agreement and its mandate.
4. Close with what remains human judgment. End every statement with an explicit clause naming the criteria, cases, and revisions the evaluator system does not decide.
5. Date it and say what voids it. Material changes to the model, judge, harness, tasks, environment, or users call for revalidation, and drift or a new optimizer can retire the statement before anything is rebuilt.

The builder opens on the SQL-agent example with $b^{*}$ and $\Pi$ marked planned: replace each value with your own system's, switch a slot to measured only when you can attach n and an interval to it, and copy the result as Markdown.

Where other items fit#

Some items that other dossier and checklist formats list separately already have a home in the sixteen fields and the fifteen items.

In the dossier#

• Resolution has no field of its own; the checklist pairs it with signal semantics in item 7.
• The dossier records the hierarchy family of each mechanism only as a prior, not as a rung that settles trust, because the validation profile replaces hierarchy labels with measurements.
• Calibration and error model, known blind spots, adversarial tests, and versioning and provenance fall under natural validity, the validity domain, adaptive validity, and reproducibility and maintenance.
• What remains human judgment has no field either; the validity statement carries it.

In the checklist#

For a benchmark release, several of the fifteen items expand into sub-points:

• Item 2, population and sampling: the task source, strata and weights, inclusion and exclusion criteria, contamination controls, and version dates.
• Item 3, environment and versions: dependencies, simulated and live components, snapshots, permissions, and known drift surfaces; and for reproducibility, harness code, configurations, environment images, raw traces or privacy-compatible summaries, and exact model identifiers where possible.
• Item 4, specification: outcome and process criteria, negative constraints, abstention behavior, and the specification's owner.
• Item 5, anchors and evidence: the calibration and reference data.
• Item 6, mechanism and configuration: grader types, test coverage or state-check logic, the judge's model, prompt, and version, and adversarial tests of the scorer.
• Item 9, natural validity: convergent, discriminant, and criterion or predictive evidence where the claims require them.
• Item 10, reliability: the replication facets allowed, and the empirical stability across them.
• Item 11, statistics: sample sizes, intervals, the paired-comparison method, slice counts, weighting, and dependence handling.
• Items 12 and 13, adaptive validity and channels: known shortcuts and Goodhart surfaces, and which hidden or adversarial evidence stays outside the optimization loop.
• Item 15, maintenance: the versioning and refresh policy, linking tasks, the contamination response, and retirement and saturation criteria.

Open in the Studio#

The Studio builds the same record with you. It fills the same sixteen fields, writes one validity statement per use with every planned component marked, and audits the draft against the same fifteen checklist items, because it reads the same framework file this page renders. Each case study opens there with its own preset, such as the SQL agent of Case C.

The templates follow, generated from that file: the statement's slots with the reason each exists, the dossier's sixteen fields with their layers, and the checklist, which keeps your ticks when you copy it.