Anatomy of a Retry Storm: A Postmortem That Changes Code

说明与用途

三个独立的重试层相互不知情导致重试风暴，压垮系统并耗尽预算。实施单一的重试守卫，包含预算上限、熔断器和幂等键，防止了未来的类似事件。

问题

Three independent retry layers composed without awareness of each other caused a retry storm, overwhelming the system and exhausting the budget.

结果

Implemented a single retry layer with budget check, circuit breaker, and idempotency key to prevent future retry storms.

工作流

• Upstream provider returned 429 on one model (01:58).
• Job-level retry re-enqueued failed jobs (02:06).
• SDK-level retry multiplied in-flight calls (02:14).
• Client-level retry added another multiplication (02:22).
• Circuit never opened; budget alarm fired (02:31).
• Manual stop of worker pool (02:40).
• Drain with idempotency keys cleared backlog (03:15).

证据

Three independent retry layers had been composed by people who each assumed the other two did not exist.

来源

One logical call passed through three retry layers at three attempts each, which is twenty-seven HTTP requests per unit of work; four hundred backlogged jobs became roughly eleven thousand requests in under twenty minutes.

来源

The guard below does three things: it charges the budget pessimistically before the call instead of after, it opens a circuit after consecutive failures, and it derives an idempotency key from the payload so a retried job cannot write twice.

来源