Mutate Golden Cases Before You Trust the Eval

说明与用途

在提示评估中添加负面控制案例，以捕获隐藏回归并提高模型可靠性。

问题

Prompt evaluations that only test happy‑path golden cases miss silent regressions, leading to undetected model drift and costly endpoint changes.

结果

The harness detects silent regressions by ensuring mutated golden cases fail, improving model stability and reducing costly endpoint swaps.

工作流

• 1. Create a YAML case file with expect and mutate sections.
• 2. Implement a deterministic grader that checks tool name, required arguments, and forbidden substrings.
• 3. Write a runner that calls the live endpoint and generates a synthetic mutated completion.
• 4. Grade the live completion against expect and the mutated completion against mutate.
• 5. If the mutated completion passes, flag a harness bug and fail the CI job.
• 6. Integrate the runner into the CI pipeline so every commit triggers the evaluation.

证据

Store a negative control beside each happy-path fixture, then refuse the suite when that control still passes.

来源

The grader should stay boring, because string search, key presence, and enum equality do not flake with temperature.

来源