Your Free Endpoint Is Fine. Your Retry Loop Is the Anti-Pattern.

说明与用途

在免费环境中，LLM 调用往往导致无休止的重试风暴和模型评估不准确。集中式重试预算、抖动退避和断路器可消除重试风暴，而多次调用可实现正确的分布评估。

问题

Workers on free servers calling OpenAI‑compatible APIs get stuck in endless retry storms and noisy single‑sample validation, wasting time and tokens.

结果

Centralizing retry logic with a global budget and jittered backoff eliminates runaway retry storms. Repeating calls for validation captures distribution noise, leading to more reliable prompt assessment and efficient token usage.

工作流

• 1. Define a global MAX_ATTEMPTS (e.g., 4) for all retry layers.
• 2. Implement call_with_budget(fn, attempts=MAX_ATTEMPTS) that retries once per attempt, applies full jitter backoff, and raises after the last attempt.
• 3. Wrap the LLM call in call_with_budget to centralize retry logic.
• 4. Add a circuit breaker: after N consecutive failures, pause for 60 seconds and log the job as failed.
• 5. For validation, set N (e.g., 20) and repeat the prompt N times, collecting responses.
• 6. Parse each response to ensure it matches the expected JSON schema.
• 7. Aggregate results (e.g., success rate, latency statistics) to evaluate model reliability.
• 8. Use the aggregated data to decide whether the prompt is stable or needs redesign.

证据

Retry logic lives at three layers. The HTTP client, the SDK, and your loop each think they own the problem. Nobody caps the total.

来源

Repeat the call N times and record what actually came back.

来源