The Worker Looked Idle. The Tests Still Passed.

Description and use

A refactored worker lost its retry logic, causing silent failures on partner timeouts while tests still passed.

Problem

A refactor removed the retry logic from a webhook worker, causing silent failures when partners timed out or returned 429, while tests still passed because they only verified successful responses.

Outcome

The incident highlighted that unit tests covering only successful responses cannot detect removal of retry logic; the worker silently failed on partner timeouts, causing lost jobs.

Workflow

• 1. Agent refactored worker, preserving public API and helper names. 2. CI ran unit tests, all green. 3. Merge accepted. 4. Partner timed out; worker returned single attempt and marked job done. 5. Merchant reported missing webhook. 6. On-call investigated logs, found silent failure. 7. Reverted refactor. 8. Incident closed after confirming retry logic restored.

Evidence

The tests never called a transport that failed, and they never watched sleep or attempt counters move.

Source

The agent "simplified" this: no sleep, no continue, just fall through

Source