The Concurrency Myth: Why One Successful Request Is Not a Baseline

说明与用途

负载探测脚本显示，免费模型端点在单个请求时成功，但在并发请求下会出现排队、延迟增加和错误。

问题

Free model endpoints often appear reliable on a single request but fail or exhibit high latency under concurrent load, misleading users about performance.

结果

The script revealed that while the first request succeeded, higher concurrency caused queueing, increased latency, and occasional failures, demonstrating that single-request success is not a reliable performance indicator.

工作流

• 1. Install Python and httpx.
• 2. Create load_probe.py with the provided async script.
• 3. Set URL, TOKEN, and PROMPT variables.
• 4. Run script: python load_probe.py <concurrency> <total> (defaults 5 and 25).
• 5. Collect results: status counts, successful 200 responses, errors.
• 6. Analyze latency percentiles and error rates to assess concurrency behavior.

证据

At 10 concurrent requests, it becomes a different product.

来源

The first request warms a path. The 20th request may hit a cold path.

来源