The Diff Wasn't the Model: Fingerprint the Environment Before You Replay a Trace

说明与用途

在跟踪中添加简单的清单可准确检测环境变化，防止误报并节省工程师时间。

问题

Replay failures were blamed on model changes, but the real cause was missing environment fingerprints, leading to wasted engineering time.

结果

Adding a single manifest line to traces prevented false alarms and saved half a day of engineering effort.

工作流

• 1. Compute SHA‑256 hash of each tool schema file and combine them.
• 2. Compute SHA‑256 hash of all prompt files (content‑hashed with path).
• 3. Record model requested, model returned, temperature, max tokens, and tool timeout.
• 4. Hash the lockfile and interpreter version.
• 5. Assemble these into a manifest dictionary and compute a short manifest_id.
• 6. Prepend the manifest as a run_manifest line to each trace file.
• 7. During replay, compare the manifest to detect environment changes before executing the trace.

证据

The run diff was clear and useless. The model had not changed.

来源

The fix is cheap. Write one manifest line at the top of each trace.

来源

Add one field that most harnesses skip: model_returned. Copy it from the response metadata, not from your config.

来源