Glossary · Reliability, calibration, and statistics
Estimand
4ValidationThe exact quantity an evaluation sets out to estimate: the outcome, the population it ranges over, and the functional (single-trial rate, pass@k, pass^k, a tail quantile, a paired difference). Many statistical mistakes are estimand mistakes.
Defined in
Handbook · ValidationEstimands, pass rates, and intervalsWrite the estimand first, then report n, an honest interval and the right functional, from Wilson to pass^k to reweighted estimates.Related terms
- Pass rate
- The estimated probability that evaluated behavior satisfies the operational specification under a declared distribution.
- Instrument chain
- The links through which a score supports a decision: target construct, sampled tasks and environments, trials and trajectories, grader readings, an estimate with uncertainty, and the decision. Error and invalid inference can enter at every link.
Where it comes up
- Evaluation is measurementHandbook · Foundations
- Pass-rate intervals and sample sizeCalculator