Glossary · Meaning and specification
Error analysis
Discovering failure modes from real traces: a domain expert open-codes what went wrong, axial-codes the notes into a counted failure taxonomy, and iterates until about twenty consecutive traces add no new category.
Defined in
Handbook · ClaimTask distributions and dataset engineeringA dataset is a model of a population. Grow it from real failures, sample where the information is, reweight to the world, and keep leaks out.Related terms
- Regression set
- Tasks that deliberately over-sample known failures to support fast iteration. Its pass rate is not a population estimate; keep it apart from an estimation set sampled or weighted to the population.
Where it comes up
- Constructs, facets, and nuisancesHandbook · Claim
- Estimands, pass rates, and intervalsHandbook · Validation
- Living eval suitesHandbook · Lifecycle