Skip to content
Benchmaker

Glossary · Validity and evidence

Contamination

1Claim

Leakage of evaluation items or close variants into training, prompts, tools, or fine-tuning, giving a construct-irrelevant advantage. Hygiene: task provenance, private or time-stamped holdouts, canaries, and post-cutoff splits.

Defined in

Handbook · ClaimTask distributions and dataset engineeringA dataset is a model of a population. Grow it from real failures, sample where the information is, reweight to the world, and keep leaks out.
Illegitimate feature
A feature carrying target information the deployed system should not have, such as a gold file reachable through tools or task IDs correlated with labels. It alone predicts the label, even after arbitrary re-encoding.
Construct-irrelevant variance
A validity threat in which factors outside the construct, such as output format, token budgets, tool latency, or answer length, drive the score, making it impure. The nuisance list guards against it.

Where it comes up

See it in the full glossary →