Stop Retrying Blind: A Failure-Bucket FAQ for Free Model Endpoints

Описание и применение

Баги при работе с бесплатными моделями фиксируются в «bucket»‑логах, после чего решения о повторных попытках принимаются по таблице действий, что снижает ненужные запросы и улучшает UX.

Задача

Free model endpoints often return timeouts, rate limits, or server errors, yet developers retry blindly without understanding failure patterns, leading to wasted retries and degraded user experience.

Результат

By bucketing failures before retrying, developers can see which errors are rare, avoid unnecessary retries, and improve overall latency and reliability of free model endpoints.

Рабочий процесс

• Measure token rates (prefill and decode) locally and set constants.
• Define timeout_budget() to compute per-request budget based on token counts and overhead.
• Implement bucket() to classify responses into failure buckets.
• Implement decide() to map bucket kinds to actions (return, give up, retry with backoff).
• Wrap HTTP client calls in a loop that respects total deadline, attempts, and applies decide() actions.
• Log each bucket outcome for monitoring and future tuning.

Доказательства

Everyone adds exponential backoff. Almost nobody tags failures first.

Источник

A bucket is one log line: TIMEOUT, HTTP_429, HTTP_503, HTTP_5XX, CONN_RESET, BAD_RESPONSE, OK.

Источник