My AI Agent Hit a Rate Limit at Midnight and Retried 11,000 Times. My "Smart" Retry Logic Was the Dumbest Code I've Shipped.
说明与用途
在Raspberry Pi上运行的代理使用简单的指数退避导致一次限流事件产生11,000次请求,导致账户被封。通过实现带有抖动、重试预算和熔断器的APICaller解决了问题。
问题
A Python agent on a Raspberry Pi retried API calls too aggressively, turning a single rate‑limit event into an 11,000‑request denial‑of‑service attack on its vendor.
结果
The agent now limits retries globally, uses jittered backoff, and stops retrying when the vendor is rate‑limited, preventing repeated 11,000‑request spikes and avoiding account suspension.
工作流
- 1. Detect rate‑limit (429) from any API call.
- 2. Use APICaller.call to retry with jittered exponential backoff.
- 3. Track failures and maintain a rolling retry budget of 20 attempts per minute.
- 4. Open circuit breaker after 10 consecutive failures for 5 minutes.
- 5. On sustained rate‑limit, write a cooling‑off file and exit with success code.
- 6. Configure systemd StartLimitIntervalSec/StartLimitBurst to avoid rapid restarts.
证据
By 6 AM: ~11,000 requests, account flagged, three hours of my morning spent on a support call proving I wasn’t a crypto scraper.
Bug 1: Backoff without jitter. Every retry used exactly 2^attempt seconds.
- 记录日期: 2026-09-21T17:00:25.861Z