Retry Policy Refactor to Reduce Unnecessary API Calls

Description and use

Rewrote retry logic for external API calls to retry only on server‑side errors and honor Retry‑After, reducing batch latency to 90 s and stopping backlog growth.

Problem

A retry helper was blindly retrying failed API calls, causing unnecessary quota consumption and backlog growth.

Outcome

Batch processing time reduced from ~4 min to ~90 s, quota consumption stabilized, and backlog growth halted.

Workflow

• 1. Identify retry helper logic. 2. Replace loop with predicate that checks status code and headers. 3. Retry only on 429 or 5xx responses. 4. Honor Retry‑After header. 5. Cap attempts and return original error on first non‑retryable response. 6. Deploy change and monitor batch latency and quota usage.

Evidence

The old helper was a loop and a sleep: up to four tries, a flat two-second wait, no branching on the response.

Source

The replacement is a small predicate that reads the status code and the headers before deciding anything.

Source

The batch settled back to about ninety seconds and the quota survived into the evening.

Source