Task distributions and dataset engineering

Dataset engineering is the design of an eval set as a finite model of a task distribution: the population of situations over which a claim should generalize. It grows tasks from real failures, samples where the information is, reweights to the world the score must describe, and keeps leaks out.

A dataset is a model of a population#

An eval set is not merely a list. It stands in for the situations the claim is about, the way an electoral poll stands in for the electorate: a smaller well-designed sample can support a better population estimate than a huge convenience sample. Whatever the eval set's mix, its headline estimates success on that mix:

$$\mathbb{E}\big[\hat p\,\big] = \mathbb{E}_{x \sim Q}\big[\psi(S(x))\big] \neq \mathbb{E}_{x \sim P}\big[\psi(S(x))\big] = p$$

where:

• $\hat p$ is the eval set's pass rate over its $n$ tasks;
• $Q$ is the distribution the eval set actually samples, and $P$ is the target population named in the claim;
• $S(x)$ is the system's behavior on task $x$, and $\psi$ is the success property the specification defines;
• $p$ is the pass rate the claim is about.

In words: a headline pass rate measures success on the eval set's own mix, which is the claim's pass rate only when that mix matches the population. More tasks only make the wrong estimand more precise.

Under covariate shift, when the two distributions differ only in how often each kind of task occurs, weights $w(x) = P(x)/Q(x)$ can sometimes correct the mix. But weighting cannot repair missing support: if deployment contains task regions the benchmark never samples, no arithmetic creates evidence there. If a fifth of the live questions to DocsAgent, the documentation agent of Case G, concern a product version its suite never covers (illustrative), no weight recovers that fifth; only new tasks do. Strata come from the construct's facets, and experts and production traces justify their weights.

Grow the suite from real failures#

Early suites should grow from production traces, bug reports, red-team findings, and expert analysis rather than from generic taxonomies alone. The practitioner method this handbook adopts, from Husain (2025) and Anthropic (2026), is error analysis:

1. Collect real traces.
2. Open-code them: a domain expert writes free-form notes on what went wrong.
3. Axial-code the notes into a failure taxonomy whose categories are named and counted.
4. Iterate to saturation: start with on the order of a hundred traces, and stop a pass when about twenty consecutive traces yield no new category.

The counts, not anyone's intuition, then dictate where eval and engineering effort go. Saturation here means new traces stop producing new categories, not the saturation of a benchmark at its ceiling. For SQLAgent, the text-to-SQL agent of Case C, open codes such as "fiscal quarter read as a calendar quarter" or "guessed at an ambiguous metric instead of asking" (illustrative) would roll up into two of its facets, business-calendar semantics and ambiguity handling. Representative failures become regression cases; a handful is enough to start, because early effect sizes are large (Anthropic, 2026). Generic metrics such as "helpfulness" are no substitute: they give an unjustified illusion of confidence (Husain, 2025).

Error analysis does not replace writing evals before code. Requirements known from the product definition — the SQL must execute, citations must resolve, out-of-scope requests must be declined — deserve evals before the capability exists; failure modes nobody can predict, such as which joins confuse the model or what users actually type, come only from traces. The suite at any moment is the union of both streams.

What makes a task trustworthy#

Each task should be checkable, passable, and unambiguous (Anthropic, 2026), and each property has a test you can run before the task enters the suite:

Include negative-polarity tasks, in which the correct behavior is to decline, abstain, or do nothing, in realistic proportion, because one-sided suites train one-sided optimization (Anthropic, 2026). For SQLAgent these are unanswerable requests; for DocsAgent, questions the documentation does not answer. The Berkeley course's list of ways an evaluation goes wrong asks for the same discipline: test data accurate and diverse enough, tasks that reflect practitioners' real needs, no shortcuts, and test cases hard enough to challenge (Song, 2025, slide 46).

SWE-bench Verified shows what skipping these checks costs. Professional developers screened 1,699 samples for underspecified issues and for unit tests that could reject valid fixes, and 68.3% were filtered out, leaving 500 tasks; on those, GPT-4o resolved 33.2% against 16% on the original set with the best scaffold (OpenAI, 2024, via Song, 2025, slide 78). Invalid tasks can understate capability as well as overstate it. Task validity and outcome validity develops this repair and task validity in general; see also the SWE-bench card.

Regression sets and estimation sets#

A regression suite intentionally over-samples known failures, so it should not masquerade as an unbiased population estimate. Keep at least two sets apart: a regression set, designed to expose specific failures and support fast iteration, and an estimation set, sampled or weighted to answer a population question.

The estimation set can still over-sample on purpose, as long as the books stay honest: sample where the information is, then reweight to the population. Suppose DocsAgent's production questions are 70% how-to, 20% troubleshooting, and 10% conceptual (illustrative). A proportional suite of 200 tasks gives troubleshooting, where most failures are, only 40 tasks; 24 passes there support only 60% (95% Wilson CI 45–74%, n = 40). The team builds 400 tasks instead — 100 how-to, 200 troubleshooting, 100 conceptual — and 90, 120, and 70 pass. Troubleshooting tightens to 60% (95% Wilson CI 53–67%, n = 200). The raw pass rate, 70% (95% Wilson CI 65–74%, n = 400), describes a user base that is half troubleshooters; reweighted to the production mix, the same results give 82% (95% CI 78–86% from the stratified standard error, n = 400). The price is a Kish effective sample size of about 192: for the production-mix rate, the 400 tasks are worth that many equally weighted ones (Estimands, pass rates, and intervals gives the formulas).

The calculator opens on the 40-task troubleshooting slice; raise the number of tasks to 200 and watch the half-width fall from ±14.5 points to ±6.7, then switch to planning mode to see that a ±5-point claim about a slice near 60% needs 366 tasks.

Leaks: bias, illegitimate, and circular features#

Riezler and Hagmann (2024) organize validity problems in machine learning around three kinds of feature, and they transfer directly to agent evaluations:

Contrastive and out-of-domain evaluation is therefore validity evidence, not merely robustness theater, and leakage is worse than overfitting: it invalidates the interpretation of the score. Document how gold outcomes were constructed, what the agent's tools can reach, and which variables would make the score trivial if discovered.

Contamination hygiene#

Contamination, in which evaluation items or close variants leak into training, prompts, tools, or fine-tuning, is one special case of illicit information flow. Its symptoms are implausibly high scores and no transfer to production. The hygiene is to keep provenance for every task, maintain private or time-stamped holdouts, use canary-like artifacts where appropriate, and refuse to read a suspiciously easy result as capability evidence until leakage has been investigated. Post-cutoff splits apply the time stamp to the model: CyberGym runs a contamination analysis on vulnerabilities found after the models' knowledge-cutoff dates (Song, 2025, slide 53).

How real benchmarks build their tasks#

The table condenses the Berkeley course's descriptions of how several agent benchmarks build their data (Song, 2025, slides 47–63, 78): CyberGym (Wang et al., 2025), τ-bench (Yao et al., 2024), τ²-bench (Barres et al., 2025), GDPval (OpenAI, 2025), LegalAgentBench, and SWE-bench Verified.

Every pipeline pairs a source of tasks with a check that each task is well posed before it counts. Only two descriptions name a defense against contamination or saturation: CyberGym's post-cutoff analysis, and GDPval's expert-authored tasks tied to concrete work products, which the course calls low in contamination risk and hard to saturate (Song, 2025, slides 53, 59). Living eval suites covers keeping a suite useful as models improve.

In practice#

Write the task-population record for one eval set.

1. Define the population and its strata. Take strata from the construct's facets and weights from deployment traffic, and list the traffic the suite cannot yet sample.
2. Run error analysis. Open-code about 100 real traces, axial-code the notes into a counted taxonomy, and stop a pass when about 20 consecutive traces add no new category.
3. Write tasks from both streams. Turn known requirements into tasks before code, and discovered failures into regression tasks.
4. Vet every task. Check the verdict, the reference solution, and bypass routes; add negative-polarity tasks in realistic proportion.
5. Keep two sets. Over-sample known failures in a regression set, and report the estimation set reweighted to the population with its effective sample size.
6. Audit for leaks. Document how gold outcomes were built, what the agent's tools can reach, and which variables would make the score trivial.
7. Record provenance. Date and source every task, and keep private or time-stamped holdouts, canaries, and post-cutoff splits where they apply.

The artifact is the task-population record — source, sampling, strata, weights, inclusion and exclusion rules, contamination controls, and version dates — which is field 3 of the evaluator dossier.

Common failure modes#

• A regression set as the headline — a suite that over-samples known failures is quoted as how good the system is → report the estimation set, reweighted to the production mix, with n and an interval. See one-number reporting.
• Bypassable tasks — the agent can reach the answer key or edit the tests that grade it → constrain the routes to a pass, and audit what the tools can reach. See grader gaming.
• Contaminated items — a suspiciously easy score that does not transfer to production → check provenance, holdouts, and canaries before reading it as capability. See contamination.
• One-sided suites — no tasks in which the right action is to decline or do nothing → add negative-polarity tasks before optimization learns that acting always pays. See Goodharting.

Check yourself#

Analyze. Reconstruct the pipeline of a benchmark you use: where its tasks come from, what check each passed before it counted, and what defends it against contamination. Which of the task properties above does its paper demonstrate?

Hint

Compare it with the pipelines table. For a public benchmark, ask whether any split postdates the models you evaluate, as in CyberGym's post-cutoff analysis.

Evaluate. A team reports the unweighted pass rate of a 400-task suite in which half the tasks are troubleshooting questions, though troubleshooting is a fifth of production traffic. What does the number estimate, and what should they report instead?

Hint

Unweighted reporting of a deliberately unbalanced suite estimates a fictional user base. Reweight by stratum, and report the effective sample size with the interval.

Create. Plan version 1 of an eval set for an agent you build: strata and weights, the error-analysis plan, the regression and estimation sets, two negative-polarity tasks, and the contamination controls.

Hint

Size each stratum for the claim you want to make about it: in the calculator's planning mode, a ±5-point claim about a slice near 60% needs 366 tasks.

Further reading#

• Constructs, facets, and nuisances — where the strata come from.
• Estimands, pass rates, and intervals — stratified estimates, the Kish effective sample size, and covariate-shift weights.
• Task validity and outcome validity — shortcuts, impossible setups, and the SWE-bench Verified repair.
• Living eval suites — contamination, saturation, and drift across versions.
• Riezler and Hagmann (2024), Validity, Reliability, and Significance — the bias, illegitimate, and circular features.
• Husain (2025), "Evals FAQ," and Anthropic (2026), "Demystifying evals for AI agents" — the practitioner methods behind error analysis and task design.