Skip to content
Benchmaker

Glossary · Meaning and specification

Claim scope

1Claim

Whether an evaluation supports claims about a specific capability, a specific application, or a general set of applications. Scope fixes the population; the broadest scope makes the strongest and most fragile claim.

Defined in

Handbook · ClaimPopulation, environment, harness, and validity domainEvery score was earned in some world. Declare the population, environment and harness, and say where the claim stops.
Validity domain
The declared conditions and population over which an interpretation is justified. Outside it, a benchmark has made no justified claim.
Population
The target task, user, or input distribution a claim is about. An eval set is a finite model of it; its headline estimates the population's pass rate only when the set's mix matches.

Where it comes up

See it in the full glossary →