Glossary · Reliability, calibration, and statistics
Mixed-effects significance
Testing a difference between systems with a model in which tasks, rollouts, judges, and environment versions are variance components, asking whether the system effect survives the measured experimental variation.
Defined in
Handbook · ValidationComparing systemsPair tasks, count discordant pairs, size for the smallest difference that matters, and never let significance stand in for validity.Related terms
- Generalizability theory
- A variance-component framework that assigns measurement variance to facets such as task, judge, seed and environment, and their interactions.