Anthropic SDK max_retries Stacked on My Retries: 36 Calls per Job

Description and use

Multiple retry layers multiplied to 36 HTTP calls per failure, causing a retry storm that overwhelmed the API and broke live traffic. Removing redundant retries and separating clients by traffic class restored normal operation.

Problem

Multiple retry layers multiplied, causing 36 HTTP attempts per failed call and a retry storm that overwhelmed the API and broke live traffic.

Outcome

By eliminating redundant retries and separating traffic classes, the retry storm was prevented, live traffic remained unaffected, and background job throughput returned to normal.

Workflow

• 1. Identify all retry layers: SDK, tenacity, RQ job retry. 2. Remove tenacity wrapper around API calls. 3. Create separate Anthropic clients for background and live traffic. 4. Set SDK max_retries to 0 for background client or keep default for live client. 5. Cap background job queue size and add rate limiting. 6. Add httpx event hook to count real HTTP requests. 7. Monitor ratio of HTTP requests to app calls. 8. Adjust retry strategy based on observed ratios.

Evidence

The Anthropic Python SDK retries a failed request twice by default, so one client.messages.create() can become 3 HTTP requests.

Source

If you wrap messages.create() in tenacity, and your job queue also retries, the attempts multiply, not add. Mine was 3 × 4 × 3 = 36.

Source

140 report jobs, which should have been about 420 requests (3 calls each) – 1,302 calls to call_claude() according to my app logs – 3,904 actual HTTP requests according to the httpx hook.

Source