Glossary · Validity and evidence
Illegitimate feature
A feature carrying target information the deployed system should not have, such as a gold file reachable through tools or task IDs correlated with labels. It alone predicts the label, even after arbitrary re-encoding.
Also called: leakage
Defined in
Handbook · ClaimTask distributions and dataset engineeringA dataset is a model of a population. Grow it from real failures, sample where the information is, reweight to the world, and keep leaks out.Related terms
- Contamination
- Leakage of evaluation items or close variants into training, prompts, tools, or fine-tuning, giving a construct-irrelevant advantage. Hygiene: task provenance, private or time-stamped holdouts, canaries, and post-cutoff splits.
- Bias feature
- A legitimate, predictive cue in a benchmark, such as wording, answer length, or file naming, whose relation to the outcome does not hold across environments or challenge sets. One of Riezler and Hagmann's three validity pathologies.
- Circular feature
- An input feature that deterministically defines the target, so the task reconstructs the annotation rule instead of evidencing the capability; for example, a task generator and its evaluator sharing a hidden template.