Our retry logic was collecting data and then deleting it

Описание и применение

Внедрение логирования количества попыток в цикле повторов позволило обнаруживать постепенное ухудшение и почти‑неудачные операции, которые ранее скрывались в «зеленых» логах.

Задача

Retry loops erased evidence of retries, making it impossible to detect gradual degradation or near‑miss failures in unattended runs.

Результат

By persisting retry counts, the team gained visibility into performance degradation and near‑miss failures, allowing proactive maintenance before failures occurred.

Рабочий процесс

• 1. Modify retry loop to record each attempt count. 2. Append attempt count to log entry regardless of success. 3. Store both outcome and attempt count. 4. Update monitoring to analyze attempt counts for trend detection. 5. Adjust alerting to flag high retry rates rather than only failures.

Доказательства

We built ours to solve a problem, not to report one, and for months it did exactly what we asked: it caught a failure, waited, tried again, and got through.

Источник

The fix is not clever. Count the attempts, write the count next to the outcome, and keep the count even when the outcome is good.

Источник