Same Goldens, New Question: A Prompt‑Pin Manifest for AI C++ Evals

Description and use

Implementing an identity manifest pins the prompt, golden cases, grader, and toolchain, preventing misleading stability reports caused by prompt drift.

Problem

AI C++ evaluation harnesses often miss prompt drift, causing stability reports to be misleading when the prompt changes but golden cases remain the same.

Outcome

By pinning the prompt and related artifacts, the evaluation harness now detects prompt drift, preventing misleading stability reports and ensuring each run is a distinct experiment.

Workflow

• 1. Create eval_manifest.json with empty hash placeholders.
• 2. Run freeze.py to compute SHA‑256 hashes for prompt.txt, golden directory, grader tree, and toolchain.txt.
• 3. Update eval_manifest.json with the computed hashes.
• 4. Commit the manifest to version control.
• 5. In CI, run check_identity.py before scoring to compare live hashes against the manifest.
• 6. If any hash mismatches, fail the run (on_mismatch: fail_closed).
• 7. Only after identity passes, record candidate patch hash and proceed to scoring.

Evidence

The harness refuses to score a candidate unless a manifest of hashes matches the files on disk.

Source