Glossary · Objects and system context
PopulationP
1ClaimThe target task, user, or input distribution a claim is about. An eval set is a finite model of it; its headline estimates the population's pass rate only when the set's mix matches.
Also called: target population, task distribution
Defined in
Handbook · ClaimPopulation, environment, harness, and validity domainEvery score was earned in some world. Declare the population, environment and harness, and say where the claim stops.Related terms
- Validity domain
- The declared conditions and population over which an interpretation is justified. Outside it, a benchmark has made no justified claim.
- Claim scope
- Whether an evaluation supports claims about a specific capability, a specific application, or a general set of applications. Scope fixes the population; the broadest scope makes the strongest and most fragile claim.
Where it comes up
- Constructs, facets, and nuisancesHandbook · Claim
- Task distributions and dataset engineeringHandbook · Claim
- Notation and formula cardHandbook · Reference
- CrosswalkHandbook · Reference