Your Timeout Is Local: Five Myths About Model Calls You Retry

Описание и применение

Мифы о тайм‑ауте и повторных попытках при вызовах моделей, которые приводят к дублированию работы и зависаниям. Решение: классифицировать результаты, использовать отдельные тайм‑ауты для потоков и хешировать все параметры для кэша.

Задача

Retry logic for model calls often misinterprets timeouts, HTTP status, and streaming behavior, leading to duplicate work, wasted retries, and stalled workers.

Результат

The probe revealed that many retries were unnecessary because timeouts were local and the remote call had already succeeded or partially succeeded. Adjusting retry logic reduced duplicate work and prevented workers from hanging.

Рабочий процесс

• 1. Wrap model call with a deadline-aware function that returns a verdict.
• 2. Inspect verdict: if DEADLINE, log warning and decide whether to retry; if EMPTY or TRUNCATED, handle accordingly; if OK, proceed.
• 3. For streaming calls, enforce both idle and total timeouts to avoid stalled workers.
• 4. Generate a fingerprint of model, prompt, and params to use as cache key.
• 5. Store outcomes in a JSONL ledger for analysis and tuning.

Доказательства

Your deadline is measured on your machine. The request may already sit in a queue. Or it may already be done.

Источник

A 200 says the transport worked. It does not say the text is usable.

Источник

Every retry is another call. Another unit of time. Another chance to hit a limit.

Источник

Outputs are snapshots, not constants. The same prompt can return different text on two runs.

Источник

Use two clocks: Idle timeout – no bytes for N seconds, give up. Total timeout – hard ceiling for the whole call.

Источник