Mutate Golden Cases Before You Trust the Eval

Description and use

Add negative control cases to prompt evals to catch silent regressions and improve model reliability.

Prompt evaluations that only test happy‑path golden cases miss silent regressions, leading to undetected model drift and costly endpoint changes.

The harness detects silent regressions by ensuring mutated golden cases fail, improving model stability and reducing costly endpoint swaps.

1. Create a YAML case file with expect and mutate sections.
2. Implement a deterministic grader that checks tool name, required arguments, and forbidden substrings.
3. Write a runner that calls the live endpoint and generates a synthetic mutated completion.
4. Grade the live completion against expect and the mutated completion against mutate.
5. If the mutated completion passes, flag a harness bug and fail the CI job.
6. Integrate the runner into the CI pipeline so every commit triggers the evaluation.

Store a negative control beside each happy-path fixture, then refuse the suite when that control still passes.

The grader should stay boring, because string search, key presence, and enum equality do not flake with temperature.