Probe the Load Ceiling of a Free AI Server

Description and use

Test the load ceiling of a free AI server with a Python script that measures TTFT, total latency, and failure rate under controlled concurrency.

Problem

Free AI servers often fail under realistic load, but users lack a simple way to measure their limits.

Outcome

The probe revealed that the free server’s latency spikes after 4 concurrent requests, with a 15% failure rate at 10 requests, guiding teams to limit concurrency or upgrade.

Workflow

• 1. Install Python and httpx.
• 2. Save the provided probe script as probe_free_server.py.
• 3. Run the script with the target API URL, optional prompt, concurrency, and request count: python probe_free_server.py --url <API_URL> --prompt "Write a Python function that parses CSV without the csv module." --concurrency 5 --requests 20.
• 4. Review the output for TTFT, total latency, and failure statistics.
• 5. Adjust concurrency or switch to a paid tier if failure rates exceed acceptable thresholds.

Evidence

I built a probe that answers that. It measures four things: cold start, latency percentiles, failure rate, and token burn per task.

Source