# Self-improvement and evaluator co-evolution

> When a system can change what defines "better", the evaluator becomes the load-bearing wall. What changes, who validates, how loops fail.

**Self-improvement** means a system takes part in producing a better version of itself or its outputs, where an evaluator defines "better". Every self-improvement loop therefore rests on its evaluator, and the evaluation problem changes with what the loop may modify and with who checks each change.

> [!KEY] In one sentence
> When a system can change what defines "better", the evaluator becomes the load-bearing wall, so classify every loop by what it changes and who validates the change, and keep a fixed anchor channel that the loop cannot reach.

## Terms the loop depends on

The recursive self-improvement (RSI) literature uses its words loosely, so the framework fixes them (following [Chen, Wang & Qu, 2026](https://benchmaker.ai/handbook/reference/sources#chen-wang-qu-2026), §2.1):

- An [agent](https://benchmaker.ai/glossary/agent) is an LLM-based system in a perceive–act loop; agency belongs to the loop, not the model.
- A [harness](https://benchmaker.ai/glossary/harness) is everything around the model that makes it an agent — instructions, tools, memory, skills, retrieval, orchestration, stopping rules — and is editable, potentially by the agent itself, so harness self-modification counts as self-modification even with frozen weights.
- An [evaluator](https://benchmaker.ai/glossary/evaluator) maps a candidate artifact to a quality signal; "verifier" means one with soundness guarantees, "judge" a learned or prompted one.
- [Test-time training](https://benchmaker.ai/glossary/test-time-training) updates weights during deployment, conditioned on the current query or session, with no separate curated training phase.
- [Self-play](https://benchmaker.ai/glossary/self-play) is training on tasks, opponents, or curricula the model generates itself, including zero-data regimes with no external task set.

The definition of [self-improvement](https://benchmaker.ai/glossary/self-improvement) in the lead carries the chapter: it makes the evaluator load-bearing, because no improvement claim is meaningful without a criterion that distinguishes better from worse. A coding agent that rewrites its own skill library, keeping a change only when its pass rate on a fixed benchmark rises, is self-improving with frozen weights — and the benchmark, not the agent, decides what counted as improvement.

## Bounded refinement and open-ended RSI

The survey's central cut separates two kinds of loop (Chen, Wang & Qu, 2026, §2.1). [Bounded self-refinement](https://benchmaker.ai/glossary/bounded-self-refinement) improves a system against a fixed external evaluator, so it is at least conceptually convergent and evaluable. [Open-ended RSI](https://benchmaker.ai/glossary/open-ended-rsi) can also modify the machinery or criteria of improvement, so no external reference stays permanently fixed, and the question changes from "can this system improve against a target?" to "what makes the changing target trustworthy?".

The skill-library agent stays bounded only while its benchmark sits out of reach; let it edit the tests or the judge that scores it, and the evaluator joins its modification surface, everything the loop can change. The framework requires an anchor channel outside that surface, with controlled information flow back to the optimizer (see [Exposure, modification, and access](https://benchmaker.ai/handbook/regime/exposure-and-surfaces)).

### What changes and who validates

The survey locates every method on two axes: what improves, and who validates each change (Chen, Wang & Qu, 2026, §2.2). [Deployment-time self-evolution](https://benchmaker.ai/glossary/deployment-time-self-evolution) improves outputs, per-query weights, or the harness while deployed; [training-time self-iteration](https://benchmaker.ai/glossary/training-time-self-iteration) updates weights from self-generated signals; [self-evaluation](https://benchmaker.ai/glossary/self-evaluation) improves the evaluators that define "better"; [Auto Research](https://benchmaker.ai/glossary/auto-research) automates research itself. Under [human-in-the-loop](https://benchmaker.ai/glossary/human-in-the-loop) closure a person reviews each change; under [human-on-the-loop](https://benchmaker.ai/glossary/human-on-the-loop) closure an automatic signal drives the loop while humans audit outcomes and gate deployment; a [closed loop](https://benchmaker.ai/glossary/closed-loop) has no human review.

| Who validates | Deployment-time self-evolution | Training-time self-iteration | Self-evaluation | Auto Research |
|---|---|---|---|---|
| Human-in-the-loop | A person reviews each revision or harness change | A person reviews the data or rewards before an update | Experts write and revise rubrics, judges, and benchmarks | People direct the research; AI carries out parts of it |
| Human-on-the-loop | Execution feedback or a checker drives revision; humans audit outcomes | A checker or judge scores self-generated data; humans gate releases | Judges and reward models improve automatically; humans audit them | Programs score candidates; humans decide what is adopted |
| Closed loop | The agent edits and validates its own harness, skills, or memory | The model sets its own tasks and rewards | The system rewrites its own definition of "better" | An AI runs research or model development end to end |

The cells paraphrase the survey's grid (Chen, Wang & Qu, 2026, §2.2, with §§3–6), whose corpus of 1,250 papers concentrates in the human-on-the-loop row. The most consequential cell is self-evaluation × closed loop, where the system can alter its own definition or machinery of "better" and ordinary iterative improvement shades into open-ended RSI. A closed loop alone is not open-ended: the skill-library agent runs one, yet stays bounded while its benchmark is out of reach.

## What the evidence says

The survey reports one regularity across all four categories: demonstrated self-improvement strength tracks the [verification hierarchy](https://benchmaker.ai/glossary/verification-hierarchy), from formal verifiers through execution feedback and learned judges down to [intrinsic signals](https://benchmaker.ai/glossary/intrinsic-signal) (Chen, Wang & Qu, 2026, §5.2). It is a qualitative pattern, not a measured law, and it does not imply construct validity. Program-discovery systems such as FunSearch and AlphaEvolve work at the top two rungs (Chen, Wang & Qu, 2026, §5.2).

The floor of the hierarchy has been measured. The Mirror Loop study ran models from three providers through ten rounds of ungrounded self-critique on four task families: informational change fell by 55% across the iterations, while a single verification step at the third iteration restored forward movement ([DeVilling, 2025](https://benchmaker.ai/handbook/reference/sources#devilling-2025), via Chen, Wang & Qu, 2026, §5.2).

The field has drawn the lesson. Since an influential result that models largely cannot self-correct their reasoning without external feedback, almost every new refinement system in the survey's corpus grounds its critique in execution, retrieval, a detector, or a solver: a retreat from closed-loop self-critique to human-on-the-loop verified refinement that improved reliability (Chen, Wang & Qu, 2026, §3.1). Training loops repeat the pattern: their ceiling is set by the verification signal (Chen, Wang & Qu, 2026, §4).

## How self-improvement loops fail

Three characteristic failure modes follow from violations of the hierarchy (Chen, Wang & Qu, 2026, §5.3); the responses come from the framework's anti-pattern catalog:

| Failure | How it arises | What to watch | Response |
|---|---|---|---|
| Self-confirming loop | Generator and evaluator share weights, data, or biases, so the loop rewards the errors both make, such as confident mistakes | Generator coupling: $\Pr[E \text{ errs} \mid G \text{ errs}]$ against $\Pr[E \text{ errs}]$ | Independent grounding channels |
| Model collapse | Recursive training on a model's own outputs loses the tails of the distribution | The share of externally grounded data and signal | Gate data; ground rewards externally |
| Diversity collapse | Proposers converge on the narrow band of problems the reward favors | Task and behavior diversity across iterations | Track diversity; refresh task sources |

A [self-confirming loop](https://benchmaker.ai/glossary/self-confirming-loop) needs no adversary, only correlated bias; reward hacking is the special case in which the exploitation is explicit. [Goodhart pressure and self-confirming loops](https://benchmaker.ai/handbook/pressure/goodhart-and-self-confirming-loops) develops the [generator coupling](https://benchmaker.ai/glossary/generator-coupling) diagnostic.

On collapse, the survey's synthesis is that pure closed loops degrade, but no practical system needs to run one; nobody knows how little external grounding suffices (Chen, Wang & Qu, 2026, §5.3). For self-play, one systematic study finds two separate levers, data gating and reward grounding, with collapse the default when either fails (Chen, Wang & Qu, 2026, §4.4).

## Co-evolving evaluators need a fixed anchor channel

If a static evaluator cannot supervise an improving system, the evaluator has to improve too. The survey finds the field converging on [evaluator co-evolution](https://benchmaker.ai/glossary/evaluator-co-evolution) — tests, rubrics, or judges that evolve with the policy they supervise — with at least five independent instances in 2026, such as unit tests co-evolved with the code they judge (Chen, Wang & Qu, 2026, §8).

Co-evolution cuts both ways: an improving judge can keep pace with an improving policy, but a co-evolving biased judge can also create a more sophisticated self-confirming ecosystem. For the survey, whether co-evolution escapes the self-confirming loop or only builds another floor onto it is the pivotal empirical question (Chen, Wang & Qu, 2026, §8). And a loop that evolves its rubric is revising its specification, and with it the operational construct: a question for [frame review](https://benchmaker.ai/glossary/frame-review), not a better judge.

The framework lets one channel co-evolve and holds another fixed: an optimized channel (O) free to co-evolve, an anchor channel (A) queried sparsely and kept out of training objectives, and a production channel (P) of real outcomes, with alarms on divergence (see [Channels, independence, and anchor burn](https://benchmaker.ai/handbook/lifecycle/channels-and-independence)).

The first field hint of escape comes from a closed loop. A-Evolve-Training ran the whole post-training loop of a 30B model for four rounds with no human in the loop, reaching near-parity with the top human submission on a public leaderboard (0.86 against 0.87); its authors claim only the first publicly reported autonomous post-training run at that scale, not a match for human researchers ([Shi et al., 2026](https://benchmaker.ai/handbook/reference/sources#shi-2026), via Chen, Wang & Qu, 2026, §6.2). Mid-run it noticed candidates driving its development metric to record highs without moving the external target, and began treating that proxy as evidence against a candidate: a loop detecting the corruption of its own signal, though within a human-specified objective and search space (Chen, Wang & Qu, 2026, §6.2, §8).

In channel terms, O rose while A stayed flat. In the illustrative curves below, at a tolerance of 0.10, O stops tracking A after $b^{*}$ = 12, and by $b$ = 1,024 the gap is 0.67 and the proxy-corruption wire has fired. Lower the largest budget tested to $b$ = 8: O and A still rise together while production stays flat, so only the mismatch wire fires.

```widget
exploitability-curve
preset: tripwire
tolerance: 0.1
```

Whatever the loop spends — candidates, iterations, rounds of tuning — report the tested range and the failure boundary, never "robust to Goodhart" (see [Adaptive validity](https://benchmaker.ai/handbook/pressure/adaptive-validity)).

## The non-verifiable frontier

Where no checkable signal exists — creative writing, dialogue, research taste — the hierarchy bottoms out; meta-evaluation, verifier-free intrinsic rewards, and claim-level auditability are workarounds, and none yet approaches the reliability of execution feedback (Chen, Wang & Qu, 2026, §5.4). Research execution is increasingly automated, while choosing which directions deserve pursuit is the paradigm non-verifiable task and the place humans remain. The survey concludes that the direction-setting and verification bottlenecks are one bottleneck: the [non-verifiable frontier](https://benchmaker.ai/glossary/non-verifiable-frontier) (Chen, Wang & Qu, 2026, §5.4).

The framework draws the same line: research execution can be progressively instrumented, while direction-setting includes a frame-selection problem that the verification hierarchy alone does not solve. So instrument what can be checked and keep the residue as explicit judgment ([Verifiability engineering](https://benchmaker.ai/handbook/practice/verifiability-engineering)), and give someone authority over the objective itself: frame review, a governance process that can revise what is being measured (see [Frame review and governance](https://benchmaker.ai/handbook/lifecycle/frame-review)).

> [!CONVENTION] Human judgment is not a rung
> The survey draws human research judgment, direction-setting included, as the top rung of its verification hierarchy (Chen, Wang & Qu, 2026, §5.2, Figure 5). This handbook places human judgment outside the ladder, as the external frontier and trust root with two roles: a measurement mechanism with ordinary error, and the reference authority where the evaluate-the-evaluator chain stops.

> [!OPEN] Five open problems
> The survey closes with five open problems, in rough order of leverage (Chen, Wang & Qu, 2026, §8):
>
> 1. **[The exchange rate of grounding](https://benchmaker.ai/glossary/grounding-exchange-rate).** Pure closed loops degrade, but nobody has characterized how much external signal keeps a loop improving.
> 2. **Verifying the non-verifiable.** Reliable evaluation of research taste, creative quality, and direction-setting would loosen the constraint on every loop; meta-evaluation and auditability by construction are early steps.
> 3. **Stability engineering.** Rise-and-collapse, diversity collapse, and safety drift are rediscovered paradigm by paradigm, with no general account of when loops converge, oscillate, or collapse.
> 4. **Trustworthy accumulation.** Persistent self-modifications, such as skills and memory, lack the verification story that weight training has.
> 5. **[Governance-grade measurement](https://benchmaker.ai/glossary/governance-grade-measurement).** Credible slowdown proposals need auditable evidence of what a training loop is and is not improving, and almost no technical literature provides it.

## In practice

Before you let a loop modify itself, write a loop charter.

1. **Place the loop in the grid.** Name what it changes and who validates each change; flag any closed-loop reach into self-evaluation.
2. **Write the modification surface.** List what the loop can change, from prompts to tests, rubric, and judge; confirm the anchor channel is outside it, and meter what flows back.
3. **Locate the improvement signal.** Place it on the verification hierarchy and estimate its generator coupling on the loop's own outputs.
4. **Separate the channels.** Let O co-evolve; keep A fixed, sparsely queried, and logged in an exposure ledger; add P.
5. **Set tripwires before the run.** Declare the tolerances and the divergence between channels that stops the loop.
6. **Budget grounding.** Gate training data, ground rewards externally, and track diversity across iterations.
7. **Assign frame review.** Name who may revise the objective and what triggers a review.

The artifact is a loop charter: grid cell, modification surface, anchor channel and ledger, channels and tripwires, grounding budget, and frame-review owner. It supplies the regime and lifecycle terms of a [validity statement](https://benchmaker.ai/glossary/validity-statement).

## Common failure modes

- **Self-confirming loop** — the judge shares the generator's model family and improves in lockstep with it → measure generator coupling and add an independent grounding channel. See [self-confirming loop](https://benchmaker.ai/handbook/practice/anti-patterns#self-confirming-loop).
- **Evaluator inside the surface** — the loop can edit the tests, rubric, or judge that score it → move them outside the modification surface and meter what flows back. See [wireheading](https://benchmaker.ai/handbook/practice/anti-patterns#wireheading).
- **Ungrounded closed loop** — the model trains on its own outputs with no external signal → gate data and ground rewards externally. See [model collapse](https://benchmaker.ai/handbook/practice/anti-patterns#model-collapse).
- **Frame lock-in** — the loop keeps improving a criterion after the real objective has moved → schedule frame review; a better judge will not catch it. See [frame lock-in](https://benchmaker.ai/handbook/practice/anti-patterns#frame-lock-in).

## Check yourself

**Analyze.** Place three loops in the grid and name each one's evaluator and anchor channel, if it has one: an agent that keeps a skill-library change only when a fixed coding benchmark improves; a model that judges its own responses for training; and an AI-scientist pipeline scored by an LLM reviewer.

> [!DETAILS] Hint
> The first is a closed loop that stays bounded while the benchmark is out of reach; the second shares weights between generator and evaluator, the classic self-confirming setup; the third sits at the non-verifiable frontier.

**Evaluate.** A team proposes letting its reward judge co-evolve with the policy "so the reward keeps pace". Argue for or against, and say what evidence would show co-evolution escaping self-confirmation.

> [!DETAILS] Hint
> It is defensible only with a fixed anchor channel outside the modification surface. Escape looks like the anchor and production channels rising with the optimized one within tolerance, while the judge's generator coupling on current-policy outputs stays low.

**Create.** Design an evaluator co-evolution scheme for a coding agent that writes its own tests, with a fixed external anchor channel that prevents unconstrained drift.

> [!DETAILS] Hint
> Let the agent's tests be the optimized channel. Keep hidden reference tests outside its workspace as the anchor channel, query them once per release, log each query, and stop when its own pass rate rises while the hidden tests stay flat.

## Further reading

- [Exposure, modification, and access](https://benchmaker.ai/handbook/regime/exposure-and-surfaces) — the modification surface and the anchor condition.
- [Channels, independence, and anchor burn](https://benchmaker.ai/handbook/lifecycle/channels-and-independence) — O, A, and P channels, tripwires, and the exposure ledger.
- [Goodhart pressure and self-confirming loops](https://benchmaker.ai/handbook/pressure/goodhart-and-self-confirming-loops) — generator coupling and collapse in depth.
- [Frame review and governance](https://benchmaker.ai/handbook/lifecycle/frame-review) — who decides whether a loop still pursues the right objective.
- Chen, Wang & Qu (2026), "Recursive self-improvement in AI: From bounded self-refinement to autonomous research loops" — §2 for the grid, §5 for self-evaluation and its failures, §8 for co-evolution and open problems.

---
Source: https://benchmaker.ai/handbook/agents/self-improvement