The Free Tier Is a Queue, Not a Machine: A Fit-Test FAQ

Описание и применение

Разработчики часто ошибочно интерпретируют метрики бесплатного уровня LLM, полагая, что быстрый первый байт означает быстрый сервер. Систематический тест, измеряющий пропускную способность и вариацию, позволяет корректно оценить очередь и оптимизировать клиентскую реализацию.

Задача

Developers often misinterpret free tier performance metrics, leading to incorrect assumptions about latency and throughput when integrating LLM endpoints.

Результат

The fit test revealed that free tier performance is highly variable; single‑run latency was misleading. After adopting connection reuse and backoff, throughput doubled and error rates dropped. The free tier was correctly classified as a queue, guiding workload matching.

Рабочий процесс

• 1. Write a probe script (e.g., fitcheck.py) that sends repeated streaming requests to the free endpoint.
• 2. Run the probe at three concurrency levels (1, 4, 8) for a defined number of rounds (e.g., 10).
• 3. Record TTFB, total time, token count, and error occurrences for each call.
• 4. Aggregate results, compute statistics (mean, variance, tokens per second).
• 5. Classify the endpoint using a decision table that maps latency, throughput, and error rate to workload fit.
• 6. Adjust client implementation: enable connection pooling, add timeout handling, use exponential backoff with jitter.
• 7. Re‑run the probe to confirm improved stability and throughput.

Доказательства

TTFB measures queue position, not capacity.

Источник

Run the same probe at least ten times, across different hours.

Источник