Your Free Endpoint Is Fine. Your Retry Loop Is the Anti-Pattern.

Description and use

In free environments, LLM usage often leads to endless retry storms and inaccurate model assessment. A centralized retry budget, jittered backoff, and circuit breaker eliminate retry storms, while repeated calls enable proper distribution evaluation.

Problem

Workers on free servers calling OpenAI‑compatible APIs get stuck in endless retry storms and noisy single‑sample validation, wasting time and tokens.

Outcome

Centralizing retry logic with a global budget and jittered backoff eliminates runaway retry storms. Repeating calls for validation captures distribution noise, leading to more reliable prompt assessment and efficient token usage.

Workflow

• 1. Define a global MAX_ATTEMPTS (e.g., 4) for all retry layers.
• 2. Implement call_with_budget(fn, attempts=MAX_ATTEMPTS) that retries once per attempt, applies full jitter backoff, and raises after the last attempt.
• 3. Wrap the LLM call in call_with_budget to centralize retry logic.
• 4. Add a circuit breaker: after N consecutive failures, pause for 60 seconds and log the job as failed.
• 5. For validation, set N (e.g., 20) and repeat the prompt N times, collecting responses.
• 6. Parse each response to ensure it matches the expected JSON schema.
• 7. Aggregate results (e.g., success rate, latency statistics) to evaluate model reliability.
• 8. Use the aggregated data to decide whether the prompt is stable or needs redesign.

Evidence

Retry logic lives at three layers. The HTTP client, the SDK, and your loop each think they own the problem. Nobody caps the total.

Source

Repeat the call N times and record what actually came back.

Source