Task validity and outcome validity
Task validity is the degree to which tasks elicit the target capability rather than shortcuts, impossible setups, or irrelevant bottlenecks. Outcome validity is the degree to which grading recognizes actual task success and failure. Agent benchmarks strain both, because they add complex interactive setups and unstructured outcomes.
Two links in the chain#
Zhu et al. separate what a benchmark does from what it is supposed to show (Zhu et al., 2025). The operational chain runs from an agent through tasks, environments, and tools to a task outcome (a text response, code, or a changed environment state), and through a benchmark evaluator (string matching, state matching, tests, or a custom metric) to a reported result. The conceptual chain runs from the target capability through task success to a positive result; task validity links the first two, outcome validity the last two (Song, 2025, slides 40–41).
For SQLAgent, the text-to-SQL agent of Case C, the operational chain runs from an analyst question, with schema-inspection and query tools, to a result set compared against a reference on a pinned snapshot of a finance warehouse. Task validity asks whether answering correctly requires the SQL competence the claim names; outcome validity asks whether the comparison passes exactly the correct answers.
Zhu et al. state both links as equivalences. This handbook treats them as probabilistic and places them inside the validity argument, where task validity mainly tests the representation and elicitation claims and outcome validity the scoring claim (The validity argument). In classical terms, task validity draws mostly on content, response-process, and discriminant evidence, and outcome validity on scoring and criterion evidence and scorer reliability; the mapping is many-to-many.
Task validity: success must require the capability#
A task can fail in two directions. Success can be reachable without the capability, through shortcuts, accidental hints, or exploitable setup. Or the capability can be blocked from producing success by impossible setups or unrelated constraints.
The first direction is easy to miss, because the agent's behavior can be correct while the interpretation is wrong. Suppose a task is meant to test whether an airline-service agent can interpret policy, look up a reservation, and take an allowed action. If the environment lets the agent read a database field that encodes the expected answer, task success no longer implies policy reasoning. Agent benchmarks offer many such routes: a hidden gold file reachable through tools, test cases stored in the same repository location, task IDs correlated with labels. Strong performance on leaked information invalidates the interpretation of the score; Task distributions and dataset engineering covers how to keep it out.
The second direction is capability–harness compatibility. Before interpreting low performance, check whether the harness makes the intended capability expressible: the tool descriptions the model needs, dynamic resource discovery where deployment requires it, a context budget adequate to the task, and no rate limits or browser quirks that add failure modes outside the construct. A strong model scoring near 0% on a task is overwhelmingly a task or harness bug, not a capability fact (Anthropic, 2026). Investigate the eval before the model.
SQLAgent's task-validity evidence covers both directions: remove task IDs and schema artifacts that leak table choice, verify that every task is solvable from the allowed schema and tools, and include adversarially similar schemas so that memorized table names stop working as a shortcut.
Outcome validity: a pass must mean success#
Outcome validity fails when the evaluator is fooled by weak tests, brittle string matching, incomplete state checks, or poorly calibrated judgment, and grading logic can then systematically over- or underestimate performance (Zhu et al., 2025). For SQLAgent, exact SQL-string match has poor outcome validity because many queries are semantically equivalent. The stronger design executes the query against a pinned snapshot, compares result sets semantically, adds property checks for forbidden writes and cross-tenant access, and uses metamorphic tests or several database instances where result-set equality could be accidental.
Checks by output type#
Zhu et al.'s Agentic Benchmark Checklist makes outcome-validity failures concrete for each kind of output, and the Berkeley lecture groups its checks by output type (Zhu et al., 2025, via Song, 2025, slides 42–45). The table paraphrases them and adds the error each group prevents:
| Checks | What the checks require | Error they prevent |
|---|---|---|
| O.a.1–2 Text, whole-string or substring match | Accept expressions semantically equivalent to the reference; tolerate redundant words in the answer | False fails on correct answers |
| O.b.1–3 Text, substring match | Handle negation; resist an answer that lists every possibility; make the reference complex enough that guessing rarely matches | False passes |
| O.c.1–2 Text, LLM judge | Document the judge's accuracy, self-consistency, and agreement with humans; design it to resist adversarial inputs and reward hacking | Unknown or exploitable judge error |
| O.d.1–2 Code, unit or end-to-end tests | Verify the tests' correctness and quality, for example by human review; measure test quality with objective metrics such as coverage and cyclomatic-complexity control | Tests that pass wrong code or fail right code |
| O.e.1–3 Code, fuzz testing | Address edge cases; cover the relevant input variations, such as data types, memory layouts, and value ranges; generate inputs the code is sensitive to | Bugs the hand-written tests miss |
| O.f.1–2 Code, end-to-end tests | Exercise all relevant parts of the code; prevent flaky, non-deterministic results | Untested paths, and noise read as signal |
| O.g.1–3 Environment state, state matching | Include every state reachable after success; check relevant and irrelevant state; make the target state complex enough to rule out trivial edits | False fails on valid alternatives; false passes with side effects or trivial edits |
| O.h.1–2 Multi-step reasoning, answer matching | State the required answer format in the task; minimize success by random guessing | Format errors read as reasoning errors; lucky guesses |
| O.i.1 Multi-step reasoning, quality metrics | Design metrics that resist exploitation, such as reward hacking | Scores earned without the work |
Read through this handbook's architecture, the checks become design rules. Decide first whether exact surface form is part of the construct. For code, the largest uncertainty may be test completeness rather than judge noise. Characterize the acceptable success states and the forbidden side effects before writing a state checker. For multi-step behavior, separate terminal success from trajectory constraints, check invariants and permissions directly, and keep process judges for the semantic residue.
O.c needs one addition. Documented accuracy, self-consistency, and human agreement are useful but incomplete: a judge can agree with itself and with a homogeneous panel while all of them apply the wrong criterion. For a binary verdict, measure the judge's true-positive and false-positive rates on adjudicated labels, per compared system; The judge as a noisy sensor shows how.
Test the evaluator before it tests the agent#
An evaluator is a program, and it needs test cases with known verdicts. The Berkeley lecture's checklist for building an evaluator asks for cases in which agents succeed or fail in different ways, each with its ground-truth verdict and with as many edge cases as possible, to check that the evaluator gives reliable results (Song, 2025, slide 85). For a ticket-booking agent, the lecture's cases are the first four rows below; the outcome-validity checks suggest the last four, which are this chapter's additions:
| Case | Expected verdict | What it tests |
|---|---|---|
| Books the requested ticket | Pass | The baseline success |
| Books the wrong ticket | Fail | A wrong outcome |
| Books a more expensive ticket than needed | Fail, or a lower score if price is graded | Every requirement on the end state |
| Never finds the booking site | Fail | A failure to act |
| Books the requested ticket through a different, allowed sequence of pages | Pass | Every success state is accepted (O.g.1) |
| Books the requested ticket and cancels an unrelated booking | Fail | Irrelevant state is checked (O.g.2) |
| Writes a confirmation record directly, without booking | Fail | Trivial state edits are ruled out (O.g.3) |
| Reports a booking that never happened | Fail | The verdict reads the booking system's state, not the agent's report |
What a benchmark audit can find#
Before adopting an existing benchmark, the Berkeley course asks for a quality analysis: check sampled data for correctness, clarity, and difficulty, confirm that metrics and judges align with true task success, and note biases and limitations (Song, 2025, slides 74–77). SWE-bench Verified is its example of what such an analysis finds.
Of 1,699 samples screened by 93 professional developers, 38.3% were flagged as underspecified and 61.1% for unit tests that could unfairly reject valid fixes; 68.3% were filtered out, and the released Verified set has 500 tasks. GPT-4o resolved 33.2% of them, against 16% on the original set with the best scaffold (OpenAI, 2024, via Song, 2025, slide 78).
Both links had failed, in the same direction. An underspecified issue can leave a capable agent unable to tell what counts as a fix, an impossible setup in task-validity terms; a test that rejects valid fixes is the outcome-validity failure O.d targets; and the lecture notes setup friction that sometimes caused false negatives (slide 78). All three push the score down: an invalid benchmark can understate capability as well as overstate it. The SWE-bench card records what the repaired subset does and does not support.
Necessary, not sufficient#
Task and outcome validity are crucial, but they are not the whole validity argument. A benchmark can have strong task and outcome validity while lacking representative sampling, transport evidence, or decision relevance. Zhu et al.'s framework is strongest at the middle of the inference chain; psychometrics adds the outer links: construct representation, relations to other measures, generalization to the intended population and use, and stability across legitimate replications.
SWE-bench Verified shows the gap. Its screening repaired tasks and tests; it did not make Python issues from well-maintained open-source projects a sample of every codebase, and the SWE-bench card marks transfer to other languages and closed codebases as untested. Sources of validity evidence and Reliability and generalizability take up the outer links.
In practice#
Audit task and outcome validity before you adopt, publish, or gate on a benchmark.
- Write the two links down. State the target capability and what counts as task success, and name the grader that decides success for each output type.
- Solve a sample yourself. Check sampled tasks for correctness, clarity, and difficulty, and confirm that each is solvable with the allowed tools and information.
- Hunt for shortcuts. List what the agent can reach through its tools — files, database fields, task IDs, test locations — and try to pass a task without the capability.
- Check the harness. Compare tool descriptions, context budget, rate limits, and resource discovery with the deployment interface, and treat a near-zero score from a strong model as a bug until the traces say otherwise.
- Run the output-type checks. Classify every grader by output type, and record evidence for each matching row of the O.a–O.i table.
- Test the evaluator. Build a known-verdict suite with successes, failures of different kinds, edge cases, and attempts to pass without the work, and rerun it on every grader change.
- Repair and re-score. Fix or filter flagged tasks and tests, and report the score before and after, each with n and an interval.
The artifact is a task-and-outcome validity audit: per-task flags (underspecified, unsolvable, leaky), a check table per grader with its evidence, the evaluator's known-verdict suite, and before-and-after scores. It supplies part of the natural-validity field of the evaluator dossier.
Common failure modes#
- Leaky environment — the agent can read the expected answer from state, files, or IDs → remove the route, then re-test with an agent that looks for shortcuts. See grader gaming.
- Brittle matching — exact strings reject correct paraphrases and equivalent queries → compare meaning, by execution, result-set equality, or a calibrated judge with an explicit equivalence rule. See specification.
- Untested tests — tests accept wrong patches, reject valid fixes, or flake → verify and measure the tests (O.d), fuzz the inputs (O.e), and control flakiness (O.f). See Case B.
- Near-zero read as incapacity — a strong model scores near 0% and the result is reported as a capability gap → treat it as a task or harness bug until the traces show otherwise. See harness architecture and reproducibility.
Check yourself#
Analyze. A web-shopping benchmark marks a task as passed when the orders table contains a row with the target product ID. List its task-validity and outcome-validity threats, and name the check from the O.a–O.i table that each outcome threat fails.
Hint
For task validity, ask whether the product ID can be read from the page source or a tool response. For outcome validity, ask whether a direct insert would pass (O.g.3), whether extra orders or a changed shipping address would go unnoticed (O.g.2), and whether an equally acceptable product counts as success (O.g.1).
Evaluate. A strong model scores near 0% on a new agent benchmark on which weaker models do better, and the authors report a surprising capability gap. Evaluate the claim, and say what you would check first.
Hint
A near-zero score from a strong model is overwhelmingly a task or harness bug. Read the failing traces; check tool descriptions, context budget, rate limits, and required answer formats (O.h.1); and look for a capability the harness never lets the model express.
Create. Write a known-verdict suite of at least eight cases for the grader of a calendar-scheduling agent. Cover every O.g check and at least one attempt to pass without doing the work.
Hint
Pair each success with its near-miss: a different free slot that satisfies every constraint (pass), the right slot with an unrelated meeting deleted (fail), an event created without the required invitations (fail), and a "done" message with no change of state (fail).
Further reading#
- The validity argument — where the two links sit among the six claims.
- Specification: what counts as success — the success condition the outcome checks test against.
- Reading agent benchmarks — benchmark-quality analysis before adoption, across six agent benchmarks.
- Harness architecture and reproducibility — the harness defects that masquerade as capability gaps.
- Zhu et al. (2025), "Establishing best practices for building rigorous agentic benchmarks" — the Agentic Benchmark Checklist behind O.a–O.i.
- Song (2025), UC Berkeley lecture on agent evaluation — slides 40–46 for the links and checks, 74–85 for benchmark audits and evaluator test cases.