Living eval suites

A living eval suite is an eval suite run as a living specification: its tasks change on purpose from version to version, every reported run is frozen and reproducible, and tripwires watch for the moment its score stops meaning what it used to. The asset is that machinery, not the score at any one time.

A suite is a living specification#

The life of a suite is a loop of nine steps:

1. Collect real traces and failure reports.
2. Taxonomize the errors.
3. Write or revise the specification.
4. Add tasks, adversarial probes, and slices.
5. Validate the graders.
6. Compare systems with uncertainty.
7. Gate releases on explicit decision rules.
8. Monitor production for drift, disagreement, and failures.
9. Feed new failures back into the suite.

The asset is not the score at time t; it is the institutional machinery that can notice when a score stops meaning what it used to mean. The One Spine ends in the same place, with step 9 operating tripwires and step 10 revalidating and reviewing the frame; The One Spine maps the whole loop onto its steps.

For DocsAgent, the documentation agent of Case G, tasks come from production query clusters and documented failure modes, judges are validated against double-labeled SME examples, offline scores are compared with fresh user-resolution outcomes, and drift in documents or queries produces a new benchmark version while frozen snapshots keep old results reproducible.

Two streams feed step 4 without conflict: requirements known from the product definition deserve evals before code, and failure modes nobody could predict come only from error analysis on real traces (see Task distributions and dataset engineering).

Ownership splits the same way: contribution of cases should be as open as possible, because cases are evidence, while arbitration of the specification should rest with one named owner, because specifications are decisions: the rule of "cases wide, specs narrow". Anyone may add a DocsAgent failure to the suite; only the spec owner decides whether it changes what counts as a grounded answer.

Three ways a static suite decays#

A suite that stays fixed while the world moves decays in three ways:

• Contamination: tasks become visible in training data or in developers' optimization loops.
• Saturation: the suite can no longer distinguish frontier systems in the region of interest.
• Drift: users, tools, documents, policies, or workflows change while the suite stays fixed.

Contamination's second route is easy to miss. A certification split whose failures developers study is contaminated without any training run, and when that split is the anchor channel, the loss is anchor burn (Channels, independence, and anchor burn). Hygiene for the first route, such as provenance, private holdouts, canaries, and post-cutoff splits, belongs to Task distributions and dataset engineering.

Saturation looks like success, which is what makes it dangerous: a suite pinned near 100% measures nothing — no headroom, no signal, only false comfort — while real regressions hide inside the rounding. Treat it as a maintenance event: harden the tasks, promote them to regression duty, and commission harder capability tasks. A suite for SQLAgent, the text-to-SQL agent of Case C, saturated on simple filters could never have supported a claim about enterprise SQL competence; hardening it means adding the facets that claim needs, such as join reasoning, business-calendar semantics, and ambiguity handling.

Some designs resist both. The Berkeley lecture holds up GDPval as a benchmark with low contamination risk that is hard to saturate: its tasks are written by vetted professionals averaging 14 years of experience, reviewed in five rounds, tied to concrete work products, and graded by blinded experts from the same occupations (Song, 2025, slide 59).

Drift is the quietest of the three. DocsAgent's documents are revised and its users bring new questions; the suite still runs, but on last year's product. Detecting live requests that fall outside the validity domain is a job for production monitoring; the suite's job is to version itself when the world moves.

Versioned dynamism#

The Berkeley lecture contrasts static benchmarks such as MMLU, whose fixed test cases allow direct, reproducible comparisons, with what it calls dynamic ones, such as LiveCodeBench, whose data are updated or regenerated over time so that they stay relevant and are harder to overfit or contaminate (Song, 2025, slide 23). This handbook treats the choice as a maintenance axis and notes that "dynamic" names two different things:

1. Maintenance dynamics: the corpus changes between versions.
2. Within-episode dynamics: the agent acts in a stateful world that changes during a trial.

This handbook always says which: a suite is dynamic in maintenance when its corpus is refreshed between versions, and an environment is stateful when it changes within an episode. The two are independent: a frozen suite can hold stateful environments, and a monthly-refreshed one can hold only single-turn questions.

The synthesis is versioned dynamism: dynamic corpus, static run. Refresh the suite deliberately across versions, but freeze every environment snapshot, tool contract, scorer, and dependency needed to reproduce a reported result, because a new snapshot is a new measurement form, not a silent mutation of the old one. The rule reconciles two correct warnings, one against uncontrolled live dependencies that silently change benchmark conditions (Zhu et al., 2025) and one against stale data. Harness architecture and reproducibility covers the mechanics of freezing a run.

Refreshing creates a comparability problem, because version 4 of a suite is a different form from version 3. Preserve some tasks across versions, maintain overlap samples run on both, and report within-version comparisons separately from the linking procedure that connects the forms. The preserved tasks are linking items; when DocsAgent's documentation moves to a new release, questions whose correct answers did not change can serve as linking items (illustrative).

Divergence tripwires#

A divergence tripwire is a predeclared monitor that fires on a named warning pattern, usually signals that should move together coming apart. The framework names four, stated over the O/A/P channels: the optimized channel O, the anchor channel A, and the production channel P of Channels, independence, and anchor burn.

The patterns and their hedged readings are the framework's; the responses follow its anti-pattern catalog and the One Spine.

The second wire is the only one that consults production. If DocsAgent's judged grounding score rises on both the development and the hidden split while fresh user-resolution outcomes stay flat, the benchmark may still measure something coherent, but it no longer supports the deployment interpretation.

Write down each wire's window, threshold, and owner before it fires, and let "rising" mean a change larger than its interval (Comparing systems). Measure exposure b in the optimizer's own unit, such as iterations, queries, or training steps.

The widget draws illustrative O, A, and P curves with a tolerance of 0.20 on the gap between O and A; at the default budget, b = 1,024, two wires have fired and $b^{*} = 56$. Step the largest budget tested down to b = 2, then back up. Through b = 16 only the mismatch wire fires, because O and A rise while P stays flat; at b = 32 the corruption wire fires while the gap, 0.16, is still inside the tolerance; the gap wire waits until b = 64. A trend wire can catch the turn before a level wire does, so wire both. Finally, drop the tolerance to 0.10 and watch b* fall to 12.

Saturation and frame lock-in need monitors of their own: saturation is a maintenance event, not a success, and lock-in goes to frame review.

Revalidation triggers#

A living suite also knows when its own evidence has expired. The One Spine's last step is to revalidate after material changes to the model, judge, harness, tasks, environment, or users, and to trigger frame review when metrics improve but real outcomes do not. For a learned judge, add production disagreement alarms to the list.

Revalidation is not a rerun. It re-measures the parts of the evidence a change could have moved, or runs an explicit bridging study when a load-bearing part of the measurement changes:

When SQLAgent's warehouse moves from schema v12 to v13, for example, the new snapshot makes a new measurement form, the result-set checker must be revalidated on the new schema, and linking items let releases be compared across the change (illustrative). The judge as a noisy sensor covers recalibration.

In practice#

Write a maintenance plan for one suite.

1. Version every run. Give each reported run a version that pins its tasks, environment snapshots, tool contracts, scorers, and dependencies.
2. Run the loop on a cadence. Schedule the nine steps, with named owners for the traces, the error taxonomy, and the specification.
3. Carry linking items. Keep a linking set and overlap samples in every new version, and report within-version results separately from cross-version links.
4. Set decay thresholds. Write down what counts as saturation, how you will look for contamination, and which drift signals trigger a new version.
5. Wire the four tripwires. For each, record the window, the threshold, the owner, and the first response.
6. Map the revalidation triggers. For each kind of change, list the evidence it reopens.
7. Split ownership. Open case contribution to everyone; give the specification one named owner.

The artifact is a maintenance plan, which fills the maintenance entry of the evaluator dossier (drift monitoring, saturation thresholds, refresh policy, judge revalidation, and retirement) and answers the reporting checklist's item on maintenance, drift, saturation, and frame-review policy.

Common failure modes#

• Saturation blindness — the suite sits near 100% and the dashboard celebrates → set a ceiling threshold, then harden, promote to regression duty, and commission new tasks. See saturation blindness.
• Contamination — scores arrive implausibly easily and do not transfer to production → keep dated provenance, private held-out sets, and canary strings. See contamination.
• Silent mutation — a live dependency or an edited scorer changes results between runs of "the same" version → freeze the run, and treat every new snapshot as a new version. See Harness architecture and reproducibility.
• Tripwires without owners — a wire fires and nothing changes → give each wire an owner and a first response before it fires. See eval theater.

Check yourself#

Analyze. A public coding benchmark has been fixed for two years, and the top five systems score between 94% and 96%. Which decay modes could produce this pattern, and what evidence would tell them apart?

Hint

Saturation predicts that the systems cannot be separated on fresh tasks of the same kind either; contamination predicts a drop on freshly written or post-cutoff tasks; drift predicts a gap between the benchmark and production outcomes.

Evaluate. A team regenerates every task in its suite from a template each month and reports one trend line across all versions. What can the trend support?

Hint

Each month is a new form. Without linking items or overlap samples, a move in the line may be a change in difficulty rather than in the system, and generation can introduce shortcuts of its own.

Create. Take a static benchmark that is probably contaminated and design a versioned replacement, dynamic in maintenance, that keeps longitudinal comparability without uncontrolled live websites.

Hint

Decide what is refreshed and what is frozen in each version, how the linking set is chosen and protected, which environment you snapshot or simulate in place of the live sites, and which tripwires watch the new versions.

Further reading#

• Channels, independence, and anchor burn — the O, A, and P channels that the tripwires compare.
• Task distributions and dataset engineering — error analysis, contamination hygiene, and where new tasks come from.
• Monitoring and control in production — detecting drift and out-of-domain requests at runtime.
• Frame review and governance — where the mismatch and lock-in wires lead.
• Song (2025), "Agent evaluation & project overview" — slide 23 on static benchmarks and those refreshed over time, slide 59 on GDPval.