The Diff Wasn't the Model: Fingerprint the Environment Before You Replay a Trace

Описание и применение

Добавление простого манифеста в трассы позволяет точно определить изменения окружения, предотвращая ложные тревоги и экономя время инженеров.

Replay failures were blamed on model changes, but the real cause was missing environment fingerprints, leading to wasted engineering time.

Adding a single manifest line to traces prevented false alarms and saved half a day of engineering effort.

1. Compute SHA‑256 hash of each tool schema file and combine them.
2. Compute SHA‑256 hash of all prompt files (content‑hashed with path).
3. Record model requested, model returned, temperature, max tokens, and tool timeout.
4. Hash the lockfile and interpreter version.
5. Assemble these into a manifest dictionary and compute a short manifest_id.
6. Prepend the manifest as a run_manifest line to each trace file.
7. During replay, compare the manifest to detect environment changes before executing the trace.

The run diff was clear and useless. The model had not changed.

The fix is cheap. Write one manifest line at the top of each trace.

Add one field that most harnesses skip: model_returned. Copy it from the response metadata, not from your config.