Same Goldens, New Question: A Prompt‑Pin Manifest for AI C++ Evals

说明与用途

通过引入身份清单，固定系统提示、黄金案例、评估器和工具链，防止提示漂移导致的误导性稳定性报告。

问题

AI C++ evaluation harnesses often miss prompt drift, causing stability reports to be misleading when the prompt changes but golden cases remain the same.

结果

By pinning the prompt and related artifacts, the evaluation harness now detects prompt drift, preventing misleading stability reports and ensuring each run is a distinct experiment.

工作流

• 1. Create eval_manifest.json with empty hash placeholders.
• 2. Run freeze.py to compute SHA‑256 hashes for prompt.txt, golden directory, grader tree, and toolchain.txt.
• 3. Update eval_manifest.json with the computed hashes.
• 4. Commit the manifest to version control.
• 5. In CI, run check_identity.py before scoring to compare live hashes against the manifest.
• 6. If any hash mismatches, fail the run (on_mismatch: fail_closed).
• 7. Only after identity passes, record candidate patch hash and proceed to scoring.

证据

The harness refuses to score a candidate unless a manifest of hashes matches the files on disk.

来源