48-Hour Field Notes: My Replay Cassette Kept Passing While the Live Call Failed

Description and use

Recording and replaying LLM requests with cassette files, proper key normalization, and strict JSON schema validation resolved discrepancies between local and production tests.

Problem

Tests that call an LLM were slow, flaky, and failed in production while the test suite passed locally.

Outcome

The harness correctly cached LLM responses, and strict validation prevented silent drift, aligning test outcomes with production.

Workflow

• 1. Design cassette key using model_tag, prompt, and params, excluding non-reproducible fields.
• 2. Implement record_or_replay function to read or write cassette JSON files.
• 3. Add normalize() to strip transport-only fields from params before hashing.
• 4. Add validator to ensure cassette payload matches expected JSON schema.
• 5. Replace loose validation with jsonschema-based strict validation.
• 6. Re-run tests to confirm replay matches live calls and passes in production.

Evidence

My test suite was green for eleven hours while the same code failed in production

Source

I had built a record-and-replay harness for LLM-backed functions so my tests would stop depending on a live call

Source

The first version keyed each cassette on a canonical hash of model tag, prompt, and parameters

Source

I initially hashed params straight from a dict that included a caller-supplied request_id. Every replay missed, every run recorded a fresh cassette

Source

The model occasionally returned each item's tags as a comma-joined string instead of an array. Downstream, for tag in item["tags"] happily iterated characters, produced nonsense labels, and never raised

Source