Anatomy of a Retry Storm: A Postmortem That Changes Code

Описание и применение

Три независимых слоя повторных попыток, не учитывающие друг друга, привели к «шторму повторных попыток», перегрузив систему и исчерпав бюджет. Внедрение единого слоя с бюджетным ограничением, переключателем цепи и ключом идемпотентности предотвратило повторные инциденты.

Задача

Three independent retry layers composed without awareness of each other caused a retry storm, overwhelming the system and exhausting the budget.

Результат

Implemented a single retry layer with budget check, circuit breaker, and idempotency key to prevent future retry storms.

Рабочий процесс

• Upstream provider returned 429 on one model (01:58).
• Job-level retry re-enqueued failed jobs (02:06).
• SDK-level retry multiplied in-flight calls (02:14).
• Client-level retry added another multiplication (02:22).
• Circuit never opened; budget alarm fired (02:31).
• Manual stop of worker pool (02:40).
• Drain with idempotency keys cleared backlog (03:15).

Доказательства

Three independent retry layers had been composed by people who each assumed the other two did not exist.

Источник

One logical call passed through three retry layers at three attempts each, which is twenty-seven HTTP requests per unit of work; four hundred backlogged jobs became roughly eleven thousand requests in under twenty minutes.

Источник

The guard below does three things: it charges the budget pessimistically before the call instead of after, it opens a circuit after consecutive failures, and it derives an idempotency key from the payload so a retried job cannot write twice.

Источник