48-Hour Field Notes: My Replay Cassette Kept Passing While the Live Call Failed

说明与用途

使用录制和回放 LLM 请求的胶囊文件，正确的键归一化和严格的 JSON 模式验证，解决了本地和生产测试之间的不一致。

问题

Tests that call an LLM were slow, flaky, and failed in production while the test suite passed locally.

结果

The harness correctly cached LLM responses, and strict validation prevented silent drift, aligning test outcomes with production.

工作流

• 1. Design cassette key using model_tag, prompt, and params, excluding non-reproducible fields.
• 2. Implement record_or_replay function to read or write cassette JSON files.
• 3. Add normalize() to strip transport-only fields from params before hashing.
• 4. Add validator to ensure cassette payload matches expected JSON schema.
• 5. Replace loose validation with jsonschema-based strict validation.
• 6. Re-run tests to confirm replay matches live calls and passes in production.

证据

My test suite was green for eleven hours while the same code failed in production

来源

I had built a record-and-replay harness for LLM-backed functions so my tests would stop depending on a live call

来源

The first version keyed each cassette on a canonical hash of model tag, prompt, and parameters

来源

I initially hashed params straight from a dict that included a caller-supplied request_id. Every replay missed, every run recorded a fresh cassette

来源

The model occasionally returned each item's tags as a comma-joined string instead of an array. Downstream, for tag in item["tags"] happily iterated characters, produced nonsense labels, and never raised

来源