Self-improvement and evaluator co-evolution

Self-improvement means a system takes part in producing a better version of itself or its outputs, where an evaluator defines "better". Every self-improvement loop therefore rests on its evaluator, and the evaluation problem changes with what the loop may modify and with who checks each change.

Terms the loop depends on#

The recursive self-improvement (RSI) literature uses its words loosely, so the framework fixes them (following Chen, Wang & Qu, 2026, §2.1):

• An agent is an LLM-based system in a perceive–act loop; agency belongs to the loop, not the model.
• A harness is everything around the model that makes it an agent — instructions, tools, memory, skills, retrieval, orchestration, stopping rules — and is editable, potentially by the agent itself, so harness self-modification counts as self-modification even with frozen weights.
• An evaluator maps a candidate artifact to a quality signal; "verifier" means one with soundness guarantees, "judge" a learned or prompted one.
• Test-time training updates weights during deployment, conditioned on the current query or session, with no separate curated training phase.
• Self-play is training on tasks, opponents, or curricula the model generates itself, including zero-data regimes with no external task set.

The definition of self-improvement in the lead carries the chapter: it makes the evaluator load-bearing, because no improvement claim is meaningful without a criterion that distinguishes better from worse. A coding agent that rewrites its own skill library, keeping a change only when its pass rate on a fixed benchmark rises, is self-improving with frozen weights — and the benchmark, not the agent, decides what counted as improvement.

Bounded refinement and open-ended RSI#

The survey's central cut separates two kinds of loop (Chen, Wang & Qu, 2026, §2.1). Bounded self-refinement improves a system against a fixed external evaluator, so it is at least conceptually convergent and evaluable. Open-ended RSI can also modify the machinery or criteria of improvement, so no external reference stays permanently fixed, and the question changes from "can this system improve against a target?" to "what makes the changing target trustworthy?".

The skill-library agent stays bounded only while its benchmark sits out of reach; let it edit the tests or the judge that scores it, and the evaluator joins its modification surface, everything the loop can change. The framework requires an anchor channel outside that surface, with controlled information flow back to the optimizer (see Exposure, modification, and access).

What changes and who validates#

The survey locates every method on two axes: what improves, and who validates each change (Chen, Wang & Qu, 2026, §2.2). Deployment-time self-evolution improves outputs, per-query weights, or the harness while deployed; training-time self-iteration updates weights from self-generated signals; self-evaluation improves the evaluators that define "better"; Auto Research automates research itself. Under human-in-the-loop closure a person reviews each change; under human-on-the-loop closure an automatic signal drives the loop while humans audit outcomes and gate deployment; a closed loop has no human review.

The cells paraphrase the survey's grid (Chen, Wang & Qu, 2026, §2.2, with §§3–6), whose corpus of 1,250 papers concentrates in the human-on-the-loop row. The most consequential cell is self-evaluation × closed loop, where the system can alter its own definition or machinery of "better" and ordinary iterative improvement shades into open-ended RSI. A closed loop alone is not open-ended: the skill-library agent runs one, yet stays bounded while its benchmark is out of reach.

What the evidence says#

The survey reports one regularity across all four categories: demonstrated self-improvement strength tracks the verification hierarchy, from formal verifiers through execution feedback and learned judges down to intrinsic signals (Chen, Wang & Qu, 2026, §5.2). It is a qualitative pattern, not a measured law, and it does not imply construct validity. Program-discovery systems such as FunSearch and AlphaEvolve work at the top two rungs (Chen, Wang & Qu, 2026, §5.2).

The floor of the hierarchy has been measured. The Mirror Loop study ran models from three providers through ten rounds of ungrounded self-critique on four task families: informational change fell by 55% across the iterations, while a single verification step at the third iteration restored forward movement (DeVilling, 2025, via Chen, Wang & Qu, 2026, §5.2).

The field has drawn the lesson. Since an influential result that models largely cannot self-correct their reasoning without external feedback, almost every new refinement system in the survey's corpus grounds its critique in execution, retrieval, a detector, or a solver: a retreat from closed-loop self-critique to human-on-the-loop verified refinement that improved reliability (Chen, Wang & Qu, 2026, §3.1). Training loops repeat the pattern: their ceiling is set by the verification signal (Chen, Wang & Qu, 2026, §4).

How self-improvement loops fail#

Three characteristic failure modes follow from violations of the hierarchy (Chen, Wang & Qu, 2026, §5.3); the responses come from the framework's anti-pattern catalog:

A self-confirming loop needs no adversary, only correlated bias; reward hacking is the special case in which the exploitation is explicit. Goodhart pressure and self-confirming loops develops the generator coupling diagnostic.

On collapse, the survey's synthesis is that pure closed loops degrade, but no practical system needs to run one; nobody knows how little external grounding suffices (Chen, Wang & Qu, 2026, §5.3). For self-play, one systematic study finds two separate levers, data gating and reward grounding, with collapse the default when either fails (Chen, Wang & Qu, 2026, §4.4).

Co-evolving evaluators need a fixed anchor channel#

If a static evaluator cannot supervise an improving system, the evaluator has to improve too. The survey finds the field converging on evaluator co-evolution — tests, rubrics, or judges that evolve with the policy they supervise — with at least five independent instances in 2026, such as unit tests co-evolved with the code they judge (Chen, Wang & Qu, 2026, §8).

Co-evolution cuts both ways: an improving judge can keep pace with an improving policy, but a co-evolving biased judge can also create a more sophisticated self-confirming ecosystem. For the survey, whether co-evolution escapes the self-confirming loop or only builds another floor onto it is the pivotal empirical question (Chen, Wang & Qu, 2026, §8). And a loop that evolves its rubric is revising its specification, and with it the operational construct: a question for frame review, not a better judge.

The framework lets one channel co-evolve and holds another fixed: an optimized channel (O) free to co-evolve, an anchor channel (A) queried sparsely and kept out of training objectives, and a production channel (P) of real outcomes, with alarms on divergence (see Channels, independence, and anchor burn).

The first field hint of escape comes from a closed loop. A-Evolve-Training ran the whole post-training loop of a 30B model for four rounds with no human in the loop, reaching near-parity with the top human submission on a public leaderboard (0.86 against 0.87); its authors claim only the first publicly reported autonomous post-training run at that scale, not a match for human researchers (Shi et al., 2026, via Chen, Wang & Qu, 2026, §6.2). Mid-run it noticed candidates driving its development metric to record highs without moving the external target, and began treating that proxy as evidence against a candidate: a loop detecting the corruption of its own signal, though within a human-specified objective and search space (Chen, Wang & Qu, 2026, §6.2, §8).

In channel terms, O rose while A stayed flat. In the illustrative curves below, at a tolerance of 0.10, O stops tracking A after $b^{*}$ = 12, and by $b$ = 1,024 the gap is 0.67 and the proxy-corruption wire has fired. Lower the largest budget tested to $b$ = 8: O and A still rise together while production stays flat, so only the mismatch wire fires.

Whatever the loop spends — candidates, iterations, rounds of tuning — report the tested range and the failure boundary, never "robust to Goodhart" (see Adaptive validity).

The non-verifiable frontier#

Where no checkable signal exists — creative writing, dialogue, research taste — the hierarchy bottoms out; meta-evaluation, verifier-free intrinsic rewards, and claim-level auditability are workarounds, and none yet approaches the reliability of execution feedback (Chen, Wang & Qu, 2026, §5.4). Research execution is increasingly automated, while choosing which directions deserve pursuit is the paradigm non-verifiable task and the place humans remain. The survey concludes that the direction-setting and verification bottlenecks are one bottleneck: the non-verifiable frontier (Chen, Wang & Qu, 2026, §5.4).

The framework draws the same line: research execution can be progressively instrumented, while direction-setting includes a frame-selection problem that the verification hierarchy alone does not solve. So instrument what can be checked and keep the residue as explicit judgment (Verifiability engineering), and give someone authority over the objective itself: frame review, a governance process that can revise what is being measured (see Frame review and governance).

In practice#

Before you let a loop modify itself, write a loop charter.

1. Place the loop in the grid. Name what it changes and who validates each change; flag any closed-loop reach into self-evaluation.
2. Write the modification surface. List what the loop can change, from prompts to tests, rubric, and judge; confirm the anchor channel is outside it, and meter what flows back.
3. Locate the improvement signal. Place it on the verification hierarchy and estimate its generator coupling on the loop's own outputs.
4. Separate the channels. Let O co-evolve; keep A fixed, sparsely queried, and logged in an exposure ledger; add P.
5. Set tripwires before the run. Declare the tolerances and the divergence between channels that stops the loop.
6. Budget grounding. Gate training data, ground rewards externally, and track diversity across iterations.
7. Assign frame review. Name who may revise the objective and what triggers a review.

The artifact is a loop charter: grid cell, modification surface, anchor channel and ledger, channels and tripwires, grounding budget, and frame-review owner. It supplies the regime and lifecycle terms of a validity statement.

Common failure modes#

• Self-confirming loop — the judge shares the generator's model family and improves in lockstep with it → measure generator coupling and add an independent grounding channel. See self-confirming loop.
• Evaluator inside the surface — the loop can edit the tests, rubric, or judge that score it → move them outside the modification surface and meter what flows back. See wireheading.
• Ungrounded closed loop — the model trains on its own outputs with no external signal → gate data and ground rewards externally. See model collapse.
• Frame lock-in — the loop keeps improving a criterion after the real objective has moved → schedule frame review; a better judge will not catch it. See frame lock-in.

Check yourself#

Analyze. Place three loops in the grid and name each one's evaluator and anchor channel, if it has one: an agent that keeps a skill-library change only when a fixed coding benchmark improves; a model that judges its own responses for training; and an AI-scientist pipeline scored by an LLM reviewer.

Hint

The first is a closed loop that stays bounded while the benchmark is out of reach; the second shares weights between generator and evaluator, the classic self-confirming setup; the third sits at the non-verifiable frontier.

Evaluate. A team proposes letting its reward judge co-evolve with the policy "so the reward keeps pace". Argue for or against, and say what evidence would show co-evolution escaping self-confirmation.

Hint

It is defensible only with a fixed anchor channel outside the modification surface. Escape looks like the anchor and production channels rising with the optimized one within tolerance, while the judge's generator coupling on current-policy outputs stays low.

Create. Design an evaluator co-evolution scheme for a coding agent that writes its own tests, with a fixed external anchor channel that prevents unconstrained drift.

Hint

Let the agent's tests be the optimized channel. Keep hidden reference tests outside its workspace as the anchor channel, query them once per release, log each query, and stop when its own pass rate rises while the hidden tests stay flat.

Further reading#

• Exposure, modification, and access — the modification surface and the anchor condition.
• Channels, independence, and anchor burn — O, A, and P channels, tripwires, and the exposure ledger.
• Goodhart pressure and self-confirming loops — generator coupling and collapse in depth.
• Frame review and governance — who decides whether a loop still pursues the right objective.
• Chen, Wang & Qu (2026), "Recursive self-improvement in AI: From bounded self-refinement to autonomous research loops" — §2 for the grid, §5 for self-evaluation and its failures, §8 for co-evolution and open problems.