Mutation-Score the Grader: A Prompt Lockfile for AI C++ Patch Evals

Описание и применение

Патчинг кода C++ с помощью ИИ может привести к скрытым регрессиям из‑за дрейфа подсказок и ораклов. Хранилище lockfile фиксирует хэши подсказок, а набор тестов с мутацией проверяет, что любые изменения вызывают сбой, предотвращая проблемы в продакшене.

Задача

AI‑generated C++ patches can silently drift in prompt, oracle, or test coverage, leading to regressions that unit tests miss.

Результат

The harness detects prompt drift, oracle rot, and patches that only satisfy existing tests, preventing silent regressions in production.

Рабочий процесс

• 1. Create a prompt.lock.json that hashes the system prompt, user template, and style rules.
• 2. Add golden case files in cases/ that specify source files, forbidden diffs, compile flags, and behavior commands.
• 3. Write mutate_grader.py to weaken oracles deliberately and expect harness failures.
• 4. Implement run_eval.sh to compile with sanitizers and run the grader.
• 5. Add a pre‑commit hook that recomputes hashes and fails if any mismatch is detected.
• 6. Run the harness after each AI patch to verify prompt integrity and oracle correctness.

Доказательства

A mid‑size systems team merged an AI‑written patch into a lock‑free queue. Unit tests stayed green. Two weeks later a prompt edit asked the model to prefer simpler atomics. The new patch compiled.

Источник

The harness does not fail. It had never been mutation‑scored, and the prompt had never been pinned.

Источник

The artifact has four files: prompt.lock.json, cases/, mutate_grader.py, run_eval.sh.

Источник