48-Hour Field Notes: My Replay Cassette Kept Passing While the Live Call Failed

Описание и применение

Запись и воспроизведение запросов к LLM с помощью кассет, правильная нормализация ключей и строгая валидация JSON‑схемы устранили несоответствия между локальными и продакшн‑тестами.

Задача

Tests that call an LLM were slow, flaky, and failed in production while the test suite passed locally.

Результат

The harness correctly cached LLM responses, and strict validation prevented silent drift, aligning test outcomes with production.

Рабочий процесс

• 1. Design cassette key using model_tag, prompt, and params, excluding non-reproducible fields.
• 2. Implement record_or_replay function to read or write cassette JSON files.
• 3. Add normalize() to strip transport-only fields from params before hashing.
• 4. Add validator to ensure cassette payload matches expected JSON schema.
• 5. Replace loose validation with jsonschema-based strict validation.
• 6. Re-run tests to confirm replay matches live calls and passes in production.

Доказательства

My test suite was green for eleven hours while the same code failed in production

Источник

I had built a record-and-replay harness for LLM-backed functions so my tests would stop depending on a live call

Источник

The first version keyed each cassette on a canonical hash of model tag, prompt, and parameters

Источник

I initially hashed params straight from a dict that included a caller-supplied request_id. Every replay missed, every run recorded a fresh cassette

Источник

The model occasionally returned each item's tags as a comma-joined string instead of an array. Downstream, for tag in item["tags"] happily iterated characters, produced nonsense labels, and never raised

Источник