Glossary · Reliability, calibration, and statistics
Replication universe
The declared set of conditions that may legitimately vary across replications: tasks, rollouts, judges, raters, environment snapshots, harness versions, time. The intended claim decides which variation is measurement error and which is part of the target.
Also called: domain of legitimate replications
Defined in
Handbook · ValidationReliability and generalizabilitySpecify what may legitimately vary, then measure how much tasks, seeds, judges and environments move the score.Related terms
- Reliability
- Stability of measurement across legitimate replications. Necessary but not sufficient: a perfectly repeatable evaluator can reliably measure the wrong construct.
- Variance components
- The shares of observed score variance attributable to each facet (tasks, rollouts, judges, raters, environment snapshots) and their interactions, estimated with mixed-effects models. Published per score, they form a variance budget.