Mutation-Score the Grader: A Prompt Lockfile for AI C++ Patch Evals

说明与用途

AI 生成的 C++ 补丁可能因提示和或acles 漂移而导致隐藏回归。锁文件固定提示哈希，黄金案例作为合同，变异测试评估器可在生产前捕获漂移。

问题

AI‑generated C++ patches can silently drift in prompt, oracle, or test coverage, leading to regressions that unit tests miss.

结果

The harness detects prompt drift, oracle rot, and patches that only satisfy existing tests, preventing silent regressions in production.

工作流

• 1. Create a prompt.lock.json that hashes the system prompt, user template, and style rules.
• 2. Add golden case files in cases/ that specify source files, forbidden diffs, compile flags, and behavior commands.
• 3. Write mutate_grader.py to weaken oracles deliberately and expect harness failures.
• 4. Implement run_eval.sh to compile with sanitizers and run the grader.
• 5. Add a pre‑commit hook that recomputes hashes and fails if any mismatch is detected.
• 6. Run the harness after each AI patch to verify prompt integrity and oracle correctness.

证据

A mid‑size systems team merged an AI‑written patch into a lock‑free queue. Unit tests stayed green. Two weeks later a prompt edit asked the model to prefer simpler atomics. The new patch compiled.

来源

The harness does not fail. It had never been mutation‑scored, and the prompt had never been pinned.

来源

The artifact has four files: prompt.lock.json, cases/, mutate_grader.py, run_eval.sh.

来源