Your Timeout Is Local: Five Myths About Model Calls You Retry

Description and use

Myths about timeouts and retries in model calls that cause duplicate work and hangs. Solution: classify outcomes, use separate idle and total timeouts for streaming, and fingerprint model, prompt, and params for caching.

Problem

Retry logic for model calls often misinterprets timeouts, HTTP status, and streaming behavior, leading to duplicate work, wasted retries, and stalled workers.

Outcome

The probe revealed that many retries were unnecessary because timeouts were local and the remote call had already succeeded or partially succeeded. Adjusting retry logic reduced duplicate work and prevented workers from hanging.

Workflow

• 1. Wrap model call with a deadline-aware function that returns a verdict.
• 2. Inspect verdict: if DEADLINE, log warning and decide whether to retry; if EMPTY or TRUNCATED, handle accordingly; if OK, proceed.
• 3. For streaming calls, enforce both idle and total timeouts to avoid stalled workers.
• 4. Generate a fingerprint of model, prompt, and params to use as cache key.
• 5. Store outcomes in a JSONL ledger for analysis and tuning.

Evidence

Your deadline is measured on your machine. The request may already sit in a queue. Or it may already be done.

Source

A 200 says the transport worked. It does not say the text is usable.

Source

Every retry is another call. Another unit of time. Another chance to hit a limit.

Source

Outputs are snapshots, not constants. The same prompt can return different text on two runs.

Source

Use two clocks: Idle timeout – no bytes for N seconds, give up. Total timeout – hard ceiling for the whole call.

Source