Anatomy of a Retry Storm: A Postmortem That Changes Code

Description and use

Three independent retry layers unaware of each other caused a retry storm, overwhelming the system and exhausting the budget. Implementing a single guard with budget ceiling, circuit breaker, and idempotency key prevented future incidents.

Problem

Three independent retry layers composed without awareness of each other caused a retry storm, overwhelming the system and exhausting the budget.

Outcome

Implemented a single retry layer with budget check, circuit breaker, and idempotency key to prevent future retry storms.

Workflow

• Upstream provider returned 429 on one model (01:58).
• Job-level retry re-enqueued failed jobs (02:06).
• SDK-level retry multiplied in-flight calls (02:14).
• Client-level retry added another multiplication (02:22).
• Circuit never opened; budget alarm fired (02:31).
• Manual stop of worker pool (02:40).
• Drain with idempotency keys cleared backlog (03:15).

Evidence

Three independent retry layers had been composed by people who each assumed the other two did not exist.

Source

One logical call passed through three retry layers at three attempts each, which is twenty-seven HTTP requests per unit of work; four hundred backlogged jobs became roughly eleven thousand requests in under twenty minutes.

Source

The guard below does three things: it charges the budget pessimistically before the call instead of after, it opens a circuit after consecutive failures, and it derives an idempotency key from the payload so a retried job cannot write twice.

Source