# Templates

> Copy-ready validity statement, 16-part evaluator dossier, and 15-item minimum reporting checklist.

This page holds the three artifacts an evaluator system should ship with: a one-sentence validity statement, the sixteen-field evaluator dossier, and the fifteen-item minimum reporting checklist. The text explains how they fit together and how to fill them honestly; the templates themselves follow at the end, ready to copy as Markdown.

## How the three fit together

The [evaluator dossier](https://benchmaker.ai/glossary/evaluator-dossier) is the full record. It has sixteen fields that a professional evaluator system should ship with, from the claim and the regime to reproducibility and frame governance. Many chapters of this handbook end with an artifact that fills one of its fields.

The validity statement summarizes the dossier in one sentence. The framework asks every serious evaluator to ship with a falsifiable statement of scope, and each slot of the sentence condenses one or more dossier fields. Read the correspondence this way:

| Statement slot | Dossier fields it condenses |
|---|---|
| Evaluator system | 4 Specification, 5 Evidence view and anchors, 6 Mechanisms, 7 Signal semantics: the configured system, not only its mechanism |
| Claim | 1 Claim |
| Consumer | 2 Regime: who or what acts on the signal |
| Validity domain | 13 Validity domain |
| Optimizer/access regime | 2 Regime: optimizer, exposure budget, modification surface, information access |
| Tested exposure $b^{*}$ | 10 Adaptive validity |
| Validation profile $\Pi$ | 8 Reliability, 9 Natural validity, 10 Adaptive validity, 11 Statistics, 12 External channels, and the drift monitoring in 14 Maintenance |
| Trust root | The reference authority behind 5 and 6, and the authority to revise the construct in 16 Frame governance |

Write one statement per use. The same dossier can support a release gate and a training reward, but the consumer, the regime, and the tested exposure differ between them, so each use gets its own sentence ([The One Spine](https://benchmaker.ai/handbook/practice/one-spine)).

The minimum reporting checklist audits a report. The framework asks every evaluator-system release or major revision to report at least its fifteen items, and each item checks that a report covers the matching dossier field. Two pairings are not obvious from the names: item 3, on environment, harness, and tool versions, draws on the reproducibility field, and item 15 covers both maintenance and frame governance. Use it on your own report before release, and on someone else's when [auditing an eval](https://benchmaker.ai/handbook/practice/auditing-an-eval).

## Fill the statement honestly

A [validity statement](https://benchmaker.ai/glossary/validity-statement) written before its evidence exists is still useful, as a plan, provided it says so ([The validation profile and validity statement](https://benchmaker.ai/handbook/pressure/validation-profile)). Five habits keep it honest:

1. **Mark $b^{*}$ and $\Pi$ measured or planned.** A measured slot carries numbers with n and an interval, as in an illustrative "72% (95% Wilson CI 63–80%, n = 100)". A planned slot says what will be measured and that it has not been. Until the profile is measured, drop "measured" from the template's "with measured validation profile".
2. **State $b^{*}$ only inside the tested range.** If the largest budget tested still held, the honest entry is "at least" that budget, not "robust" ([Adaptive validity](https://benchmaker.ai/handbook/pressure/adaptive-validity)). For a use that nothing optimizes against, say so; if anyone starts, the statement is void until adaptive validity has been measured.
3. **Give the trust root its evidence.** Name the proof kernel and its configured trusted base, or the expert panel with its measured agreement and its mandate.
4. **Close with what remains human judgment.** End every statement with an explicit clause naming the criteria, cases, and revisions the evaluator system does not decide.
5. **Date it and say what voids it.** Material changes to the model, judge, harness, tasks, environment, or users call for revalidation, and drift or a new optimizer can retire the statement before anything is rebuilt.

The builder opens on the SQL-agent example with $b^{*}$ and $\Pi$ marked planned: replace each value with your own system's, switch a slot to measured only when you can attach n and an interval to it, and copy the result as Markdown.

```widget
validity-statement-builder
preset: sql-agent
```

## Where other items fit

Some items that other dossier and checklist formats list separately already have a home in the sixteen fields and the fifteen items.

### In the dossier

- **Resolution** has no field of its own; the checklist pairs it with signal semantics in item 7.
- The dossier records the **hierarchy family** of each mechanism only as a prior, not as a rung that settles trust, because the validation profile replaces hierarchy labels with measurements.
- **Calibration and error model**, **known blind spots**, **adversarial tests**, and **versioning and provenance** fall under natural validity, the validity domain, adaptive validity, and reproducibility and maintenance.
- **What remains human judgment** has no field either; the validity statement carries it.

### In the checklist

For a benchmark release, several of the fifteen items expand into sub-points:

- **Item 2, population and sampling:** the task source, strata and weights, inclusion and exclusion criteria, contamination controls, and version dates.
- **Item 3, environment and versions:** dependencies, simulated and live components, snapshots, permissions, and known drift surfaces; and for reproducibility, harness code, configurations, environment images, raw traces or privacy-compatible summaries, and exact model identifiers where possible.
- **Item 4, specification:** outcome and process criteria, negative constraints, abstention behavior, and the specification's owner.
- **Item 5, anchors and evidence:** the calibration and reference data.
- **Item 6, mechanism and configuration:** grader types, test coverage or state-check logic, the judge's model, prompt, and version, and adversarial tests of the scorer.
- **Item 9, natural validity:** convergent, discriminant, and criterion or predictive evidence where the claims require them.
- **Item 10, reliability:** the replication facets allowed, and the empirical stability across them.
- **Item 11, statistics:** sample sizes, intervals, the paired-comparison method, slice counts, weighting, and dependence handling.
- **Items 12 and 13, adaptive validity and channels:** known shortcuts and Goodhart surfaces, and which hidden or adversarial evidence stays outside the optimization loop.
- **Item 15, maintenance:** the versioning and refresh policy, linking tasks, the contamination response, and retirement and saturation criteria.

## Open in the Studio

The [Studio](https://benchmaker.ai/studio) builds the same record with you. It fills the same sixteen fields, writes one validity statement per use with every planned component marked, and audits the draft against the same fifteen checklist items, because it reads the same framework file this page renders. Each case study opens there with its own preset, such as the [SQL agent of Case C](https://benchmaker.ai/studio?preset=c).

The templates follow, generated from that file: the statement's slots with the reason each exists, the dossier's sixteen fields with their layers, and the checklist, which keeps your ticks when you copy it.

## Validity statement slots

Evaluator system [Evaluator system E] supports claim [Claim C] for consumer [Consumer κ] over validity domain [Validity domain D], under optimizer/access regime [Optimizer/access regime R], up to tested exposure [Tested exposure b*], with measured validation profile [Validation profile Π], relative to reference authority/trust root [Trust root T].

- **Evaluator system E**: Names the configured system, not just the mechanism.
- **Claim C**: Prevents metric worship.
- **Consumer κ**: Makes the causal use explicit.
- **Validity domain D**: Bounds generalization.
- **Optimizer/access regime R**: Records adaptive exposure.
- **Tested exposure b***: Makes adaptive validity expire rather than pretending robustness is timeless.
- **Validation profile Π**: Replaces hierarchy labels with measurements.
- **Trust root T**: Names where the evaluate-the-evaluator recursion stops.

## Evaluator dossier fields

1. **Claim** (claim): Construct, intended interpretation, excluded interpretations, target population, environment, and decision.
2. **Regime** (regime): Measure / optimize / monitor-control; consumer; optimizer; exposure budget; modification surface; information access.
3. **Task population** (claim): Source, sampling, strata, weights, inclusion/exclusion, contamination controls, version dates.
4. **Specification** (instrument): Criteria, outcome/process constraints, negative requirements, abstention/escalation, aggregation, owner.
5. **Evidence view and anchors** (instrument): What the evaluator can inspect, what reference roots the judgment, and how anchor independence is protected.
6. **Mechanisms** (instrument): Mechanism per criterion, hierarchy family, configuration/trusted base, model/prompt/version if learned.
7. **Signal semantics** (instrument): Boolean/scalar/vector/preference/critique; thresholds and operating points.
8. **Reliability** (validation): Replication facets, variance components, inter-rater agreement, repeatability.
9. **Natural validity** (validation): Task/outcome validity, convergent/discriminant/criterion evidence, calibration and slice error.
10. **Adaptive validity** (validation): Optimizer stress test, exploitability curve, generator coupling, b*.
11. **Statistics** (validation): Estimands, sample sizes, intervals, paired methods, rare-event procedures, dependence handling.
12. **External channels** (lifecycle): Development, certification, production; coupling analysis; anchor-burn ledger.
13. **Validity domain** (claim): Explicit conditions of justified use and out-of-domain limitations.
14. **Maintenance** (lifecycle): Drift monitoring, saturation thresholds, refresh policy, judge revalidation, retirement.
15. **Reproducibility** (lifecycle): Harness/configuration, environment snapshots, raw traces or privacy-compatible summaries, exact model identifiers.
16. **Frame governance** (lifecycle): Triggers for construct/objective review and who has authority to revise it.

## Minimum reporting checklist

- [ ] Construct, interpretation, decision, and excluded claims
- [ ] Target population and task sampling
- [ ] Environment, harness and tool versions
- [ ] Specification and aggregation geometry
- [ ] Anchors and evidence view
- [ ] Evaluator mechanism, configuration and trusted base
- [ ] Signal semantics and resolution
- [ ] Use regime, optimizer, exposure, modification and access surfaces
- [ ] Natural-validity evidence
- [ ] Reliability and calibration
- [ ] Statistical estimands and uncertainty
- [ ] Adaptive-validity evidence and exploitability tests
- [ ] Independent channels and anchor-burn controls
- [ ] Validity domain and known blind spots
- [ ] Maintenance, drift, saturation, and frame-review policy


---
Source: https://benchmaker.ai/handbook/reference/templates