Probe the Load Ceiling of a Free AI Server

Описание и применение

Проверьте предел нагрузки бесплатного AI‑сервера, используя скрипт на Python, который измеряет время до первого токена, общую задержку и частоту ошибок при заданной параллельности запросов.

Задача

Free AI servers often fail under realistic load, but users lack a simple way to measure their limits.

Результат

The probe revealed that the free server’s latency spikes after 4 concurrent requests, with a 15% failure rate at 10 requests, guiding teams to limit concurrency or upgrade.

Рабочий процесс

• 1. Install Python and httpx.
• 2. Save the provided probe script as probe_free_server.py.
• 3. Run the script with the target API URL, optional prompt, concurrency, and request count: python probe_free_server.py --url <API_URL> --prompt "Write a Python function that parses CSV without the csv module." --concurrency 5 --requests 20.
• 4. Review the output for TTFT, total latency, and failure statistics.
• 5. Adjust concurrency or switch to a paid tier if failure rates exceed acceptable thresholds.

Доказательства

I built a probe that answers that. It measures four things: cold start, latency percentiles, failure rate, and token burn per task.

Источник