I Tried 4 Models to Save My Self-Improving Agent. All 4 Failed.

Description and use

All four LLM models failed to improve their own prompts due to a local‑optimum search strategy. The pipeline is validated, but the bottleneck is the lack of diverse candidate generation.

Problem

The self‑editing agent could not produce promotable edits because its search strategy stuck in local optima, regardless of model size.

Outcome

All four tested models failed to produce promotable edits; the issue was the search strategy, not model size. The agent’s pipeline was proven, but the bottleneck was identified and a concrete fix path defined.

Workflow

• 1. Generate one edit proposal from the analyzer. 2. Run A/B test against a failure batch. 3. If the edit passes the gate, promote; otherwise repeat. 4. Repeat until a promotable edit is found or resources exhausted.
• 5. In v0.3.0, modify step 1 to generate 10 diverse candidates. 6. Apply cheap scoring on the failure batch. 7. Select the best candidate for the A/B test. 8. Iterate until promotion or timeout.

Evidence

None of them did.

Source

The search strategy was.

Source

The analyzer generates one proposal, sees the same failures next iteration, and proposes a slightly different version of the same failed edit.

Source