Mutation-Score the Grader: A Prompt Lockfile for AI C++ Patch Evals

Description and use

AI‑generated C++ patches can silently drift in prompt and oracle, causing regressions that unit tests miss. A lockfile pins the prompt, golden cases act as contracts, and mutation testing of the grader catches any drift before production deployment.

Problem

AI‑generated C++ patches can silently drift in prompt, oracle, or test coverage, leading to regressions that unit tests miss.

Outcome

The harness detects prompt drift, oracle rot, and patches that only satisfy existing tests, preventing silent regressions in production.

Workflow

• 1. Create a prompt.lock.json that hashes the system prompt, user template, and style rules.
• 2. Add golden case files in cases/ that specify source files, forbidden diffs, compile flags, and behavior commands.
• 3. Write mutate_grader.py to weaken oracles deliberately and expect harness failures.
• 4. Implement run_eval.sh to compile with sanitizers and run the grader.
• 5. Add a pre‑commit hook that recomputes hashes and fails if any mismatch is detected.
• 6. Run the harness after each AI patch to verify prompt integrity and oracle correctness.

Evidence

A mid‑size systems team merged an AI‑written patch into a lock‑free queue. Unit tests stayed green. Two weeks later a prompt edit asked the model to prefer simpler atomics. The new patch compiled.

Source

The harness does not fail. It had never been mutation‑scored, and the prompt had never been pinned.

Source

The artifact has four files: prompt.lock.json, cases/, mutate_grader.py, run_eval.sh.

Source