The Concurrency Myth: Why One Successful Request Is Not a Baseline

Описание и применение

Скрипт нагрузочного тестирования выявил, что бесплатные модели, которые работают без ошибок при одном запросе, могут давать высокую задержку и ошибки при одновременных запросах из-за очередей и холодных стартов.

Задача

Free model endpoints often appear reliable on a single request but fail or exhibit high latency under concurrent load, misleading users about performance.

Результат

The script revealed that while the first request succeeded, higher concurrency caused queueing, increased latency, and occasional failures, demonstrating that single-request success is not a reliable performance indicator.

Рабочий процесс

• 1. Install Python and httpx.
• 2. Create load_probe.py with the provided async script.
• 3. Set URL, TOKEN, and PROMPT variables.
• 4. Run script: python load_probe.py <concurrency> <total> (defaults 5 and 25).
• 5. Collect results: status counts, successful 200 responses, errors.
• 6. Analyze latency percentiles and error rates to assess concurrency behavior.

Доказательства

At 10 concurrent requests, it becomes a different product.

Источник

The first request warms a path. The 20th request may hit a cold path.

Источник