The Free Tier Is a Queue, Not a Machine: A Fit-Test FAQ

Description and use

Developers often misinterpret free tier LLM metrics, assuming fast first byte equals fast server. A systematic fit test measuring throughput and variance enables proper queue assessment and client optimization.

Problem

Developers often misinterpret free tier performance metrics, leading to incorrect assumptions about latency and throughput when integrating LLM endpoints.

Outcome

The fit test revealed that free tier performance is highly variable; single‑run latency was misleading. After adopting connection reuse and backoff, throughput doubled and error rates dropped. The free tier was correctly classified as a queue, guiding workload matching.

Workflow

• 1. Write a probe script (e.g., fitcheck.py) that sends repeated streaming requests to the free endpoint.
• 2. Run the probe at three concurrency levels (1, 4, 8) for a defined number of rounds (e.g., 10).
• 3. Record TTFB, total time, token count, and error occurrences for each call.
• 4. Aggregate results, compute statistics (mean, variance, tokens per second).
• 5. Classify the endpoint using a decision table that maps latency, throughput, and error rate to workload fit.
• 6. Adjust client implementation: enable connection pooling, add timeout handling, use exponential backoff with jitter.
• 7. Re‑run the probe to confirm improved stability and throughput.

Evidence

TTFB measures queue position, not capacity.

Source

Run the same probe at least ten times, across different hours.

Source