The Free Tier Is a Queue, Not a Machine: A Fit-Test FAQ

说明与用途

开发者常误解免费层LLM的性能指标，认为首字节快即服务器快。通过测量吞吐量和变异性的系统化适配测试，可正确评估队列并优化客户端实现。

问题

Developers often misinterpret free tier performance metrics, leading to incorrect assumptions about latency and throughput when integrating LLM endpoints.

结果

The fit test revealed that free tier performance is highly variable; single‑run latency was misleading. After adopting connection reuse and backoff, throughput doubled and error rates dropped. The free tier was correctly classified as a queue, guiding workload matching.

工作流

• 1. Write a probe script (e.g., fitcheck.py) that sends repeated streaming requests to the free endpoint.
• 2. Run the probe at three concurrency levels (1, 4, 8) for a defined number of rounds (e.g., 10).
• 3. Record TTFB, total time, token count, and error occurrences for each call.
• 4. Aggregate results, compute statistics (mean, variance, tokens per second).
• 5. Classify the endpoint using a decision table that maps latency, throughput, and error rate to workload fit.
• 6. Adjust client implementation: enable connection pooling, add timeout handling, use exponential backoff with jitter.
• 7. Re‑run the probe to confirm improved stability and throughput.

证据

TTFB measures queue position, not capacity.

来源

Run the same probe at least ten times, across different hours.

来源