The Concurrency Myth: Why One Successful Request Is Not a Baseline

Description and use

A load probe script demonstrated that free model endpoints, while successful on a single request, suffer from queueing, increased latency, and failures under concurrent load.

Problem

Free model endpoints often appear reliable on a single request but fail or exhibit high latency under concurrent load, misleading users about performance.

Outcome

The script revealed that while the first request succeeded, higher concurrency caused queueing, increased latency, and occasional failures, demonstrating that single-request success is not a reliable performance indicator.

Workflow

• 1. Install Python and httpx.
• 2. Create load_probe.py with the provided async script.
• 3. Set URL, TOKEN, and PROMPT variables.
• 4. Run script: python load_probe.py <concurrency> <total> (defaults 5 and 25).
• 5. Collect results: status counts, successful 200 responses, errors.
• 6. Analyze latency percentiles and error rates to assess concurrency behavior.

Evidence

At 10 concurrent requests, it becomes a different product.

Source

The first request warms a path. The 20th request may hit a cold path.

Source