Glossary · Reliability, calibration, and statistics
Variance components
The shares of observed score variance attributable to each facet (tasks, rollouts, judges, raters, environment snapshots) and their interactions, estimated with mixed-effects models. Published per score, they form a variance budget.
Also called: variance budget, variance decomposition
Defined in
Handbook · ValidationReliability and generalizabilitySpecify what may legitimately vary, then measure how much tasks, seeds, judges and environments move the score.Related terms
- Generalizability theory
- A variance-component framework that assigns measurement variance to facets such as task, judge, seed and environment, and their interactions.
- Replication universe
- The declared set of conditions that may legitimately vary across replications: tasks, rollouts, judges, raters, environment snapshots, harness versions, time. The intended claim decides which variation is measurement error and which is part of the target.
Where it comes up
- Comparing systemsHandbook · Validation
- Notation and formula cardHandbook · Reference