Part 4 · The five layers
Validation
Does the score mean what we claim, and how sure are we?
- 4.1The validity argumentInteractiveValidity is a claim, not a sticker: a six-link argument from construct to decision, where any link can break.10 min read
- 4.2Task validity and outcome validityDoes success require the capability, and does a pass mean success? The two agent-era links, with checks by output type.11 min read
- 4.3Sources of validity evidenceFace, content, response-process, internal-structure, convergent, discriminant and criterion evidence, plus the multitrait–multimethod design.10 min read
- 4.4Reliability and generalizabilitySpecify what may legitimately vary, then measure how much tasks, seeds, judges and environments move the score.11 min read
- 4.5The judge as a noisy sensorInteractiveModel judge error like a diagnostic test: correct pass rates, carry calibration noise into the result, and audit error per system.10 min read
- 4.6Estimands, pass rates, and intervalsInteractiveWrite the estimand first, then report n, an honest interval and the right functional, from Wilson to pass^k to reweighted estimates.11 min read
- 4.7Comparing systemsInteractivePair tasks, count discordant pairs, size for the smallest difference that matters, and never let significance stand in for validity.10 min read
- 4.8Rare-event assuranceInteractiveZero failures in n trials is not zero risk. Bound it, then prevent, search, oversample and monitor.9 min read