Asserts Passed, RSS Doubled: An rusage Envelope for AI C++ Patches

Описание и применение

Патчи, генерируемые ИИ, могут проходить все функциональные тесты, но увеличивать потребление памяти. Добавление JSON‑оболочки с ограничениями по RSS, CPU и таймауту позволяет обнаруживать такие регрессии.

Задача

AI‑generated C++ patches can pass all functional tests while silently increasing memory usage, leading to production regressions that are hard to detect with standard test harnesses.

Результат

The resource envelope turned a silent memory regression into a detectable failure, ensuring AI patches respect performance contracts.

Рабочий процесс

• 1. Write functional tests and run them. 2. Add a resource envelope JSON alongside fixtures, specifying max_rss_kb, max_user_us, timeout_s. 3. Run the evaluation harness, which measures rusage and compares against envelope ceilings. 4. If any ceiling is exceeded, the patch is rejected. 5. Repeat until both functional and resource constraints are satisfied.

Доказательства

An eval harness that only checks exit codes will score the heavy patch as a win.

Источник

The envelope is JSON next to the fixtures. It records ceilings, not averages.

Источник