I Added Retries to My Automation and It Doubled the Failures

说明与用途

在 Playwright 自动化管道中添加通用重试导致错误率翻倍并产生重复操作。通过将步骤分为安全和不安全，写操作使用幂等键，并采用带随机抖动的指数退避，错误率从 6.7% 降至 2.9%，并消除了重复。

问题

Adding blanket retries to a Playwright automation pipeline increased the failure rate from 3.1% to 6.7% and caused silent duplicate actions.

结果

Retry strategy refined to only retry idempotent actions and to use idempotency keys for writes, combined with jittered exponential backoff, reduced failure rate from 6.7% to 2.9% and eliminated duplicate writes and retry storms.

工作流

• 1. Identify safe-to-retry steps (reads, GETs, checks).
• 2. Identify unsafe-to-retry steps (writes, posts, sends).
• 3. Add an idempotency key to unsafe steps (account_id + action + date).
• 4. Apply retry only to safe steps or to unsafe steps with idempotency key.
• 5. Implement exponential backoff with jitter (0–500 ms) to spread retries.
• 6. Monitor metrics and adjust as needed.

证据

Retries made my Playwright jobs worse, not better: failure rate went from 3.1% to 6.7% in one week.

来源

In two days I double-posted to the same account twice and sent a duplicate signup to one platform.

来源

The write steps got an explicit key (account_id + action + date) so a replay would be rejected instead of duplicated.

来源

Adding jitter — a random 0–500 ms — spread them out and cut the second‑order failures by more than half.

来源

Failure rate 6.7% → 2.9%, Duplicate writes 5 in 2 days → 0 in 7 days, Retry storms ~2/day → 0.

来源