I Added Retries to My Automation and It Doubled the Failures

Описание и применение

Добавление универсальных повторных попыток в автоматизацию Playwright удвоило количество ошибок и привело к дублированию действий. Разделив шаги на безопасные и небезопасные, добавив ключи идемпотентности и применив экспоненциальный откат с джиттером, удалось снизить частоту ошибок с 6,7 % до 2,9 % и устранить дублирования.

Задача

Adding blanket retries to a Playwright automation pipeline increased the failure rate from 3.1% to 6.7% and caused silent duplicate actions.

Результат

Retry strategy refined to only retry idempotent actions and to use idempotency keys for writes, combined with jittered exponential backoff, reduced failure rate from 6.7% to 2.9% and eliminated duplicate writes and retry storms.

Рабочий процесс

• 1. Identify safe-to-retry steps (reads, GETs, checks).
• 2. Identify unsafe-to-retry steps (writes, posts, sends).
• 3. Add an idempotency key to unsafe steps (account_id + action + date).
• 4. Apply retry only to safe steps or to unsafe steps with idempotency key.
• 5. Implement exponential backoff with jitter (0–500 ms) to spread retries.
• 6. Monitor metrics and adjust as needed.

Доказательства

Retries made my Playwright jobs worse, not better: failure rate went from 3.1% to 6.7% in one week.

Источник

In two days I double-posted to the same account twice and sent a duplicate signup to one platform.

Источник

The write steps got an explicit key (account_id + action + date) so a replay would be rejected instead of duplicated.

Источник

Adding jitter — a random 0–500 ms — spread them out and cut the second‑order failures by more than half.

Источник

Failure rate 6.7% → 2.9%, Duplicate writes 5 in 2 days → 0 in 7 days, Retry storms ~2/day → 0.

Источник