Glossary · Validity and evidence
Bias feature
A legitimate, predictive cue in a benchmark, such as wording, answer length, or file naming, whose relation to the outcome does not hold across environments or challenge sets. One of Riezler and Hagmann's three validity pathologies.
Defined in
Handbook · ClaimTask distributions and dataset engineeringA dataset is a model of a population. Grow it from real failures, sample where the information is, reweight to the world, and keep leaks out.Related terms
- Illegitimate feature
- A feature carrying target information the deployed system should not have, such as a gold file reachable through tools or task IDs correlated with labels. It alone predicts the label, even after arbitrary re-encoding.
- Circular feature
- An input feature that deterministically defines the target, so the task reconstructs the annotation rule instead of evidencing the capability; for example, a task generator and its evaluator sharing a hidden template.