Skip to content
Benchmaker

Glossary · Validity and evidence

Circular feature

An input feature that deterministically defines the target, so the task reconstructs the annotation rule instead of evidencing the capability; for example, a task generator and its evaluator sharing a hidden template.

Defined in

Handbook · ClaimTask distributions and dataset engineeringA dataset is a model of a population. Grow it from real failures, sample where the information is, reweight to the world, and keep leaks out.
Bias feature
A legitimate, predictive cue in a benchmark, such as wording, answer length, or file naming, whose relation to the outcome does not hold across environments or challenge sets. One of Riezler and Hagmann's three validity pathologies.
Illegitimate feature
A feature carrying target information the deployed system should not have, such as a gold file reachable through tools or task IDs correlated with labels. It alone predicts the label, even after arbitrary re-encoding.

See it in the full glossary →