Same Goldens, New Question: A Prompt‑Pin Manifest for AI C++ Evals

Описание и применение

Внедрение манифеста идентичности позволяет фиксировать изменения в системном запросе и других компонентах, предотвращая ложные отчёты о стабильности при изменении вопроса.

Задача

AI C++ evaluation harnesses often miss prompt drift, causing stability reports to be misleading when the prompt changes but golden cases remain the same.

Результат

By pinning the prompt and related artifacts, the evaluation harness now detects prompt drift, preventing misleading stability reports and ensuring each run is a distinct experiment.

Рабочий процесс

• 1. Create eval_manifest.json with empty hash placeholders.
• 2. Run freeze.py to compute SHA‑256 hashes for prompt.txt, golden directory, grader tree, and toolchain.txt.
• 3. Update eval_manifest.json with the computed hashes.
• 4. Commit the manifest to version control.
• 5. In CI, run check_identity.py before scoring to compare live hashes against the manifest.
• 6. If any hash mismatches, fail the run (on_mismatch: fail_closed).
• 7. Only after identity passes, record candidate patch hash and proceed to scoring.

Доказательства

The harness refuses to score a candidate unless a manifest of hashes matches the files on disk.

Источник