Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

My AI Agent Hit a Rate Limit at Midnight and Retried 11,000 Times. My "Smart" Retry Logic Was the Dumbest Code I've Shipped.

Description and use

A Raspberry Pi agent using simple exponential backoff turned a single rate limit into 11,000 requests, causing an account block. Implementing APICaller with jitter, a retry budget, and a circuit breaker fixed the issue.

Problem

A Python agent on a Raspberry Pi retried API calls too aggressively, turning a single rate‑limit event into an 11,000‑request denial‑of‑service attack on its vendor.

Outcome

The agent now limits retries globally, uses jittered backoff, and stops retrying when the vendor is rate‑limited, preventing repeated 11,000‑request spikes and avoiding account suspension.

Workflow

  • 1. Detect rate‑limit (429) from any API call.
  • 2. Use APICaller.call to retry with jittered exponential backoff.
  • 3. Track failures and maintain a rolling retry budget of 20 attempts per minute.
  • 4. Open circuit breaker after 10 consecutive failures for 5 minutes.
  • 5. On sustained rate‑limit, write a cooling‑off file and exit with success code.
  • 6. Configure systemd StartLimitIntervalSec/StartLimitBurst to avoid rapid restarts.

Evidence

By 6 AM: ~11,000 requests, account flagged, three hours of my morning spent on a support call proving I wasn’t a crypto scraper.

Source

Bug 1: Backoff without jitter. Every retry used exactly 2^attempt seconds.

Source

  • Record date: 2026-09-21T17:00:25.861Z