Stop Retrying Blind: A Failure-Bucket FAQ for Free Model Endpoints

说明与用途

在免费模型端点上记录失败桶，并根据表格做重试决策，减少不必要的请求并提升用户体验。

问题

Free model endpoints often return timeouts, rate limits, or server errors, yet developers retry blindly without understanding failure patterns, leading to wasted retries and degraded user experience.

结果

By bucketing failures before retrying, developers can see which errors are rare, avoid unnecessary retries, and improve overall latency and reliability of free model endpoints.

工作流

• Measure token rates (prefill and decode) locally and set constants.
• Define timeout_budget() to compute per-request budget based on token counts and overhead.
• Implement bucket() to classify responses into failure buckets.
• Implement decide() to map bucket kinds to actions (return, give up, retry with backoff).
• Wrap HTTP client calls in a loop that respects total deadline, attempts, and applies decide() actions.
• Log each bucket outcome for monitoring and future tuning.

证据

Everyone adds exponential backoff. Almost nobody tags failures first.

来源

A bucket is one log line: TIMEOUT, HTTP_429, HTTP_503, HTTP_5XX, CONN_RESET, BAD_RESPONSE, OK.

来源