The Worker Looked Idle. The Tests Still Passed.

Описание и применение

Переписанный воркер потерял логику повторных попыток, что привело к тихим сбоям при тайм‑аутах партнёра, хотя тесты продолжали проходить.

Задача

A refactor removed the retry logic from a webhook worker, causing silent failures when partners timed out or returned 429, while tests still passed because they only verified successful responses.

Результат

The incident highlighted that unit tests covering only successful responses cannot detect removal of retry logic; the worker silently failed on partner timeouts, causing lost jobs.

Рабочий процесс

• 1. Agent refactored worker, preserving public API and helper names. 2. CI ran unit tests, all green. 3. Merge accepted. 4. Partner timed out; worker returned single attempt and marked job done. 5. Merchant reported missing webhook. 6. On-call investigated logs, found silent failure. 7. Reverted refactor. 8. Incident closed after confirming retry logic restored.

Доказательства

The tests never called a transport that failed, and they never watched sleep or attempt counters move.

Источник

The agent "simplified" this: no sleep, no continue, just fall through

Источник