Free Inference Has a Tail. I Probe It Before I Ship.

说明与用途

尾部延迟探测揭示LLM推理中的最差1％队列延迟，帮助选择提供商并调整重试策略。

问题

Free inference latency is fine on average but the worst 1% causes run failures due to queue delays.

结果

The probe reveals tail latency and failure types, allowing developers to avoid providers with high queue delays and to tailor retry strategies.

工作流

• 1. Run a quick curl benchmark: 10 identical requests, record connect, ttfb, total times.
• 2. Inspect stability: if connect is stable but ttfb varies 10x, queue issues exist.
• 3. Execute the probe_tail.py script: set CONC (concurrency), N (number of requests), TTFT_BUDGET (acceptable first token time).
• 4. The script streams requests, records ttft, total, status, errors, and usage per request.
• 5. Analyze results: identify tail latency percentiles and failure taxonomy (e.g., 429 vs silent stall).
• 6. Use findings to decide whether to switch providers or adjust retry logic.

证据

Free inference is not slow — its average is fine. Its worst 1% is where your run dies, and the average hides it perfectly.

来源

I probe three numbers and one taxonomy before I let anything depend on a provider.

来源