The Workflow That Retried a Dead Provider 200 Times: Backoff, Jitter, and Circuit Breakers in n8n

说明与用途

在 n8n 中，供应商宕机导致 200 次重试，浪费配额并延迟作业。解决方案包括将指数退避限制为 5 分钟，尊重 Retry-After，添加抖动，并在连续五次失败后打开断路器，将作业推迟到断路器关闭后再执行。

问题

A provider outage caused a workflow to retry the same failed call 200 times, burning quota, filling logs, and delaying queued jobs.

结果

The workflow now limits retries to five attempts, spreads retries with jitter, respects provider hints, and stops retrying after a circuit breaker opens, preventing 200 redundant calls and preserving resources.

工作流

• Enable retry-on-failure in n8n
• Configure exponential backoff: 1s, 2s, 4s, 8s, up to 5min cap
• Add logic to honor Retry-After header if present
• Implement full jitter: random wait between 0 and backoff cap for each attempt
• Persist backoff state per job so it does not reset on new execution
• Add circuit breaker: after 5 consecutive retryable failures, open breaker for a fixed window
• When breaker is open, route jobs to deferred queue with reason "provider circuit open"
• After window, allow one probe call; if success, close breaker, else reopen
• Ensure breaker state lives in Postgres table shared across executions
• Add alerts for breaker opening
• Classify failures: retryable (429, 5xx, timeouts) vs non-retryable (4xx, 401/403)
• Test with mock provider that fails for 10 minutes then recovers, asserting limits and behavior

证据

By the time the provider came back, the workflow had retried the same failed call two hundred times.

来源

The first fix is backoff: each attempt waits longer than the last. Mine doubles the wait — 1 second, 2, 4, 8 — up to a cap of five minutes.

来源

The fix is jitter: add a random spread to each wait, so retries land across a window instead of on a tick.

来源

After a set number of consecutive failures — mine is five for provider calls — the breaker opens, and the workflow stops calling that provider entirely for a fixed window.

来源

None of this applies to every error. A retry policy that treats all failures the same will retry the ones that can never succeed.

来源