Asserts Passed, RSS Doubled: An rusage Envelope for AI C++ Patches

Description and use

AI‑generated patches can pass all functional tests while silently increasing memory usage. Adding a JSON envelope with RSS, CPU, and timeout limits lets evaluation harnesses detect these regressions.

Problem

AI‑generated C++ patches can pass all functional tests while silently increasing memory usage, leading to production regressions that are hard to detect with standard test harnesses.

Outcome

The resource envelope turned a silent memory regression into a detectable failure, ensuring AI patches respect performance contracts.

Workflow

• 1. Write functional tests and run them. 2. Add a resource envelope JSON alongside fixtures, specifying max_rss_kb, max_user_us, timeout_s. 3. Run the evaluation harness, which measures rusage and compares against envelope ceilings. 4. If any ceiling is exceeded, the patch is rejected. 5. Repeat until both functional and resource constraints are satisfied.

Evidence

An eval harness that only checks exit codes will score the heavy patch as a win.

Source

The envelope is JSON next to the fixtures. It records ceilings, not averages.

Source