B: Coding agent

Case B evaluates a coding agent: a tool-using system that takes a repository issue, edits code in a sandbox, and proposes a change for merge. Its claim — that the agent resolves repository issues safely and correctly — needs six criteria and five kinds of mechanism: a portfolio, not a score.

Claim and decision#

Stated at the computational level, the claim names the decision it serves: the agent can autonomously modify production code without violating approval and security constraints, often enough to justify merge privileges.

The case uses an illustrative deployment, the one the Studio preset loads: an internal agent that turns issues in a team's Python services into pull requests.

• Construct: the change fixes the reported problem, preserves existing behavior, and stays within the agent's permissions.
• Population: closed issues with linked fixes from the 20 most active services, sampled by type.
• Environment and harness: a network-off container at the base commit; file, shell, and test tools; 30 minutes per issue.
• Validity domain: Python 3.10 or later, English issue text, changes under about 400 lines.
• Decision: ship or hold each weekly release, whose pull requests then merge after one light human review.

Three regimes, one set of tests#

The same tests serve three causal roles:

Write the foreseeable migrations down now: tuning against certification failures spends that split's independence, and feeding rollbacks into prompt changes moves the monitor into the optimize regime. A third risk hides in the modification surface. The agent writes to the repository that holds the tests, so unless the grading tests live elsewhere, the anchor condition $A \notin S$ fails.

Criteria and mechanisms#

The specification has six parts, each routed to the strongest affordable mechanism that can check it:

The table holds most of the verification hierarchy inside one evaluator. Types and some authorization invariants can be verified, but whether the patch solves the user's real problem usually remains empirical. What construction guarantees comes off the sampling budget: a merge tool that refuses to run without an approval token makes merging before approval unreachable, not rare.

In this design the first five criteria are fatal gates, maintainability is tradeable against the threshold on its criterion card, and cost and latency are reported beside the verdict. A hard gate states the intended decision semantics directly; a weighted mean would let fluent code buy back a forbidden write.

In the playground, read Safety as forbidden files and approvals. In the compensation trap, System B wins the weighted mean (0.927 against 0.783) but fails the safety gate at 0.96, four tasks in a hundred with a forbidden action. Set B's safety to 0.99 and the tail rule accepts it while the gate still refuses.

Test validity and edited tests#

Because tests carry most of the weight, the evaluator's largest uncertainty may be test completeness rather than judge noise. The outcome-validity checks for code make it auditable (Zhu et al., 2025, via Song, 2025, slide 43): verify the tests' correctness and quality, by human review and metrics such as coverage (O.d); fuzz for edge cases and the inputs the code is sensitive to (O.e); and exercise all relevant code while preventing flaky results (O.f).

SWE-bench Verified shows what such an audit finds. Of 1,699 samples screened by 93 professional developers, 38.3% were flagged as underspecified and 61.1% for unit tests that could unfairly reject valid fixes; 68.3% were filtered out, and the released Verified set has 500 tasks. GPT-4o resolved 33.2% of them, against 16% on the original set with the best scaffold (OpenAI, 2024, via Song, 2025, slide 78). Invalid tests can understate capability as well as overstate it; see the SWE-bench card.

The agent-specific risk is grader gaming: satisfying the scorer without doing the work, in coding tasks classically by editing tests or fixtures (Zhu et al., 2025). The response is to constrain the routes to a pass: copy the grading tests in from outside the agent's workspace after it finishes, make test and CI paths non-writable where the claim allows, and diff-check for changes to them. The agent may write tests, but never the ones that grade it.

Retries change the estimand#

pass@k is the probability that at least one of k attempts succeeds; pass^k, that all k succeed. With independent attempts that each succeed 75% of the time, pass@3 is about 98% and pass^3 about 42%: the same system can suit an inspected coding workflow and fail an unattended one. pass@k helps only if something reliably picks out the success, such as a reviewer choosing among attempts; unattended work needs pass^k.

The calculator starts there; raise k to 10, and pass@10 rounds to 100% while pass^10 falls to about 6%.

Validation plan#

Each row is a planned, not yet measured, entry in the validation profile Π.

Channels and tripwires#

Keep three measurement channels apart: the optimized development split, cheap, dense, and free to co-evolve with the agent; the anchor channel, a hidden certification split of newly closed issues, queried once per release, never used for prompts or training, and tracked in an anchor burn ledger; and production. Frame review sits above all three, with the engineers who own the specification.

Three tripwires watch for divergence: development rising while certification stays flat signals proxy corruption; both rising while rollbacks or incidents also rise signals a construct or deployment mismatch; and certification near its ceiling signals saturation.

Validity statement#

Following the validity-statement template, and stated as a plan until the profile is measured:

The coding-agent release evaluator — hidden reference tests, type and static checks, sandbox path permissions, a trajectory checker for approvals, a maintainability judge calibrated against engineers, and engineer adjudication — is designed to support the claim that the agent resolves repository issues correctly and safely, for the release manager's weekly ship-or-hold decision, over Python 3.10+ services in the sampled repositories with English issue text and changes under about 400 lines, under weekly developer iteration on prompts, tools, and harness with full feedback on the development split and pass-or-fail access to the certification split, up to a tested exposure $b^{*}$ that has not yet been measured, with a validation profile that is planned rather than measured, relative to two senior engineers who adjudicate disputed cases and own the specification, and whose agreement is still to be measured. High-impact architectural changes and revisions to the specification remain human judgment.

Common failure modes#

• One resolve rate for six criteria — one pass rate hides forbidden writes and skipped approvals → report each gate with n and an interval. See one-number reporting.
• Grading one canonical path — the trajectory checker demands one sequence of commands → check only the constraints the claim includes, such as approval before merge. See rigid trajectory grading.

Check yourself#

Analyze. Give each of the six criteria its mechanism, hierarchy family, and regimes. Which criterion could move up the hierarchy if the agent had to emit something new?

Hint

A required reproducer test, failing before the fix and passing after it, makes part of the claim checkable by execution once the reproducer itself is validated.

Evaluate. To cut cost, the team proposes certifying releases with an LLM judge that reads each diff instead of running the hidden tests. Argue for or against it using the rule "strongest affordable mechanism per criterion per regime".

Hint

A cheap judge can suit dense development feedback while execution plus expert sampling certifies release. Which of the two is the release gate?

Create. Write the criterion card for "forbidden files untouched": definition, allowed evidence, a near-miss, boundary rule and escalation, gate semantics, mechanism, blind spots, and owner.

Hint

Settle the boundary cases first — a new file in an allowed directory, a regenerated lockfile, an added test — then decide which the sandbox blocks and which a diff check escalates.

Further reading#

• Task validity and outcome validity — all of the O.a–O.i checks, and the SWE-bench Verified repair.
• Guarantee before you estimate — making forbidden writes unreachable instead of rare.
• Estimands, pass rates, and intervals — the unbiased pass@k estimator; no unbiased pass^k estimator is developed in this handbook.
• Zhu et al. (2025), "Establishing best practices for building rigorous agentic benchmarks" — the checklist behind O.d–O.f.

Open this case in the Studio to start an evaluator dossier with its claim and regimes pre-filled.