Glossary · Reliability, calibration, and statistics
Estimation set
Tasks sampled, or deliberately over-sampled and then reweighted, to answer a population-level question such as the production pass rate. Kept apart from a regression set that over-samples known failures.
Defined in
Handbook · ClaimTask distributions and dataset engineeringA dataset is a model of a population. Grow it from real failures, sample where the information is, reweight to the world, and keep leaks out.Related terms
- Regression set
- Tasks that deliberately over-sample known failures to support fast iteration. Its pass rate is not a population estimate; keep it apart from an estimation set sampled or weighted to the population.
- Population
- The target task, user, or input distribution a claim is about. An eval set is a finite model of it; its headline estimates the population's pass rate only when the set's mix matches.