Your Timeout Is Local: Five Myths About Model Calls You Retry

说明与用途

关于模型调用超时和重试的误区，导致重复工作和挂起。解决方案：对结果进行分类，使用独立的空闲和总超时，并对模型、提示和参数进行哈希以实现缓存。

问题

Retry logic for model calls often misinterprets timeouts, HTTP status, and streaming behavior, leading to duplicate work, wasted retries, and stalled workers.

结果

The probe revealed that many retries were unnecessary because timeouts were local and the remote call had already succeeded or partially succeeded. Adjusting retry logic reduced duplicate work and prevented workers from hanging.

工作流

• 1. Wrap model call with a deadline-aware function that returns a verdict.
• 2. Inspect verdict: if DEADLINE, log warning and decide whether to retry; if EMPTY or TRUNCATED, handle accordingly; if OK, proceed.
• 3. For streaming calls, enforce both idle and total timeouts to avoid stalled workers.
• 4. Generate a fingerprint of model, prompt, and params to use as cache key.
• 5. Store outcomes in a JSONL ledger for analysis and tuning.

证据

Your deadline is measured on your machine. The request may already sit in a queue. Or it may already be done.

来源

A 200 says the transport worked. It does not say the text is usable.

来源

Every retry is another call. Another unit of time. Another chance to hit a limit.

来源

Outputs are snapshots, not constants. The same prompt can return different text on two runs.

来源

Use two clocks: Idle timeout – no bytes for N seconds, give up. Total timeout – hard ceiling for the whole call.

来源