Free Inference Has a Tail. I Probe It Before I Ship.

Описание и применение

Проба tail‑latency выявляет худшие 1 % задержек в LLM‑инференсах, позволяя выбирать провайдеров и настраивать политику повторных запросов.

Free inference latency is fine on average but the worst 1% causes run failures due to queue delays.

The probe reveals tail latency and failure types, allowing developers to avoid providers with high queue delays and to tailor retry strategies.

1. Run a quick curl benchmark: 10 identical requests, record connect, ttfb, total times.
2. Inspect stability: if connect is stable but ttfb varies 10x, queue issues exist.
3. Execute the probe_tail.py script: set CONC (concurrency), N (number of requests), TTFT_BUDGET (acceptable first token time).
4. The script streams requests, records ttft, total, status, errors, and usage per request.
5. Analyze results: identify tail latency percentiles and failure taxonomy (e.g., 429 vs silent stall).
6. Use findings to decide whether to switch providers or adjust retry logic.

Free inference is not slow — its average is fine. Its worst 1% is where your run dies, and the average hides it perfectly.

I probe three numbers and one taxonomy before I let anything depend on a provider.