Your p50 Is a Lie: Four Free-Tier Myths You Can Verify in One Hour

Description and use

Debunking free-tier LLM myths: same models, misleading p50 latency, ineffective retries, and misinterpreted streaming completion. Provides a probe script and best‑practice guidelines.

Problem

Developers using free-tier LLM endpoints face misconceptions about model quality, latency metrics, retry strategies, and completion signals, leading to inefficient usage and misinterpretation of service health.

Outcome

By following the workflow, developers can accurately assess free-tier LLM performance, avoid common pitfalls, and make informed decisions about retry strategies and latency expectations.

Workflow

• 1. Run identical prompts on free and paid endpoints with temperature 0 to compare outputs (Myth 1).
• 2. Measure request latency, compute p50, p95, p99, and stall rate to assess queue health (Myth 2).
• 3. Implement retry logic with jittered exponential backoff instead of immediate retries to avoid thundering herd (Myth 3).
• 4. Monitor streaming responses for first token arrival and final DONE marker, using read timeouts to detect stalls (Myth 4).
• 5. Use the provided myth_probe.py script to automate checks for myths 2‑4 on any OpenAI‑style endpoint.

Evidence

Run the same prompt with temperature 0. Free endpoint or paid endpoint, same text. Compare the outputs. They match token for token.

Source

Queue delay is bimodal. Half your requests land on a warm slot. Half sit behind a cold start or a crowded queue.

Source

Retry with jittered exponential backoff. Random sleep that grows each attempt.

Source

With streaming, 200 means the gate opened. Nothing more. The first token can still be seconds away.

Source