Part 1 · The five layers
Claim
Start with the inference and the decision, not the metric.
- 1.1Constructs, facets, and nuisancesDefine what the score should respond to, and just as carefully what it must ignore. Your nuisance list is your discriminant test plan.9 min read
- 1.2Population, environment, harness, and validity domainInteractiveEvery score was earned in some world. Declare the population, environment and harness, and say where the claim stops.10 min read
- 1.3Decision claimsInteractiveA score means something only when it changes a decision. Write the decision, the threshold and the cost of each error first.6 min read
- 1.4Task distributions and dataset engineeringInteractiveA dataset is a model of a population. Grow it from real failures, sample where the information is, reweight to the world, and keep leaks out.10 min read