Your Free Endpoint Is Fine. Your Retry Loop Is the Anti-Pattern.

Описание и применение

В бесплатных средах работа с LLM часто приводит к бесконечным циклам повторов и неверной оценке модели. Централизованный бюджет повторов, jitter‑backoff и circuit breaker устраняют «штормы» повторов, а многократные вызовы позволяют корректно оценить распределение ответов.

Задача

Workers on free servers calling OpenAI‑compatible APIs get stuck in endless retry storms and noisy single‑sample validation, wasting time and tokens.

Результат

Centralizing retry logic with a global budget and jittered backoff eliminates runaway retry storms. Repeating calls for validation captures distribution noise, leading to more reliable prompt assessment and efficient token usage.

Рабочий процесс

• 1. Define a global MAX_ATTEMPTS (e.g., 4) for all retry layers.
• 2. Implement call_with_budget(fn, attempts=MAX_ATTEMPTS) that retries once per attempt, applies full jitter backoff, and raises after the last attempt.
• 3. Wrap the LLM call in call_with_budget to centralize retry logic.
• 4. Add a circuit breaker: after N consecutive failures, pause for 60 seconds and log the job as failed.
• 5. For validation, set N (e.g., 20) and repeat the prompt N times, collecting responses.
• 6. Parse each response to ensure it matches the expected JSON schema.
• 7. Aggregate results (e.g., success rate, latency statistics) to evaluate model reliability.
• 8. Use the aggregated data to decide whether the prompt is stable or needs redesign.

Доказательства

Retry logic lives at three layers. The HTTP client, the SDK, and your loop each think they own the problem. Nobody caps the total.

Источник

Repeat the call N times and record what actually came back.

Источник