Glossary · Objects and system context
Instrument chain
The links through which a score supports a decision: target construct, sampled tasks and environments, trials and trajectories, grader readings, an estimate with uncertainty, and the decision. Error and invalid inference can enter at every link.
Defined in
Handbook · FoundationsEvaluation is measurementA score is an observation, not a capability. Follow the instrument chain from construct to decision and the four places error gets in.Related terms
- Error families
- The four kinds of error kept apart along the instrument chain: representation error, behavioral or trial variation, scoring error, and estimation error. Wrong world and wrong property are structural failures, not error families.
- Estimand
- The exact quantity an evaluation sets out to estimate: the outcome, the population it ranges over, and the functional (single-trial rate, pass@k, pass^k, a tail quantile, a paired difference). Many statistical mistakes are estimand mistakes.