What To Do When a Free LLM API Says 429 (A Field Playbook)

Description and use

Developers using free LLM APIs often hit 429 errors due to multiple limits. The playbook outlines detecting the specific limit via headers, honoring retry-after, exponential backoff with jitter, distributing load across providers, and logging traffic.

Problem

Free LLM APIs frequently return HTTP 429 errors, but the cause varies across rate limits and providers, leading to ineffective retry strategies.

Outcome

By detecting the specific rate limit hit, honoring retry-after, using exponential backoff, and distributing load across multiple free-tier providers, developers can reduce 429 errors, improve reliability, and make informed decisions about scaling or paid tiers.

Workflow

• Identify which limit triggered the 429 by inspecting response body and headers (e.g., x-ratelimit-remaining-requests, x-ratelimit-remaining-tokens, retry-after, quotaId).
• If retry-after is present, wait that duration before retrying; otherwise apply exponential backoff with jitter (1s, 2s, 5s, 10s, capped).
• Limit retry attempts to 3–5 for daily caps to surface errors instead of looping indefinitely.
• Count tokens rather than requests when retrying to avoid exceeding TPM limits.
• Maintain a pool of OpenAI-compatible keys from multiple providers and fail over to the next provider when a window closes.
• Route requests by limit shape (RPD, RPM, TPM) to match job requirements.
• Log per-request metadata (provider, model, latency, status, rate‑limit headers) for analysis.
• Use a router or build a provider abstraction to centralize limit tracking, queuing, and health checks.

Evidence

Read the response body and headers before you touch your code. Groq sends x-ratelimit-remaining-requests (daily) and x-ratelimit-remaining-tokens (per minute) plus retry-after, so the 429 tells you which window is empty and when to come back.

Source

Honor retry-after when present. It is the provider telling you the truth; guessing is worse.

Source

Keep several OpenAI-compatible keys (Gemini, Groq, NVIDIA NIM, OpenRouter, Cloudflare Workers AI, Cerebras and others each publish their own caps) and fail a request over to the next provider when one window closes.

Source