Stop Retrying Blind: A Failure-Bucket FAQ for Free Model Endpoints

Description and use

Failures on free model endpoints are logged into buckets, and retry decisions are made from a table, reducing unnecessary requests and improving UX.

Problem

Free model endpoints often return timeouts, rate limits, or server errors, yet developers retry blindly without understanding failure patterns, leading to wasted retries and degraded user experience.

Outcome

By bucketing failures before retrying, developers can see which errors are rare, avoid unnecessary retries, and improve overall latency and reliability of free model endpoints.

Workflow

• Measure token rates (prefill and decode) locally and set constants.
• Define timeout_budget() to compute per-request budget based on token counts and overhead.
• Implement bucket() to classify responses into failure buckets.
• Implement decide() to map bucket kinds to actions (return, give up, retry with backoff).
• Wrap HTTP client calls in a loop that respects total deadline, attempts, and applies decide() actions.
• Log each bucket outcome for monitoring and future tuning.

Evidence

Everyone adds exponential backoff. Almost nobody tags failures first.

Source

A bucket is one log line: TIMEOUT, HTTP_429, HTTP_503, HTTP_5XX, CONN_RESET, BAD_RESPONSE, OK.

Source