I Built an AI Agent That Optimizes Code — But Benchmarks Decide What Survives

Description and use

An autonomous agent proposes code optimizations, which are objectively evaluated by frozen benchmarks and tests to determine if they truly improve performance and correctness.

Problem

AI coding agents can propose optimizations that look good but may not actually improve performance or correctness.

Outcome

The system automates performance optimization by letting an AI agent iterate through changes, with a benchmark and test suite acting as an external judge to ensure only genuinely faster and correct code is retained.

Workflow

• Agent proposes an optimization
• Agent applies the change
• Run unit tests
• Run frozen benchmark
• Measure performance
• If performance improves and tests pass, KEEP the change; otherwise DISCARD
• Repeat

Evidence

The agent does not decide whether its own change is good. Instead, every experiment is evaluated by a frozen benchmark and test harness.

Source

If the change improves the measured result and still passes the required checks, it can survive.

Source