Glossary · Reliability, calibration, and statistics
Regression set
Tasks that deliberately over-sample known failures to support fast iteration. Its pass rate is not a population estimate; keep it apart from an estimation set sampled or weighted to the population.
Defined in
Handbook · ClaimTask distributions and dataset engineeringA dataset is a model of a population. Grow it from real failures, sample where the information is, reweight to the world, and keep leaks out.Related terms
- Estimation set
- Tasks sampled, or deliberately over-sampled and then reweighted, to answer a population-level question such as the production pass rate. Kept apart from a regression set that over-samples known failures.
- Error analysis
- Discovering failure modes from real traces: a domain expert open-codes what went wrong, axial-codes the notes into a counted failure taxonomy, and iterates until about twenty consecutive traces add no new category.