LLM proxy benchmark: 200 duplicate calls became 1 (and a 30 s bug)

Description and use

The LLM proxy reduced 200 identical requests to one, cutting costs by 99.5 % without added latency. During provider outages it limited failed calls from 89 to 5. A bug causing sequential upstream calls was fixed, improving 20‑prompt throughput from 30.2 s to 3.0 s (default) and 1.5 s with higher concurrency.

Problem

Duplicate identical LLM requests can cause unnecessary paid calls and latency, especially during retry storms or agent loops.

Outcome

The proxy reduced 200 duplicate calls to 1, cutting cost by ~99.5% without adding latency. It also limited failed calls during outages from 89 to 5. A bug causing sequential upstream calls was fixed in v2.0.3, improving 20‑prompt throughput from 30.2 s to 3.0 s (default) and 1.5 s with higher concurrency.

Workflow

• 1. Configure the OpenAI SDK to use the proxy base URL. 2. Send 200 identical chat completion requests concurrently. 3. Proxy deduplicates identical in‑flight requests into a single upstream call. 4. Proxy applies circuit breaking during provider outages, limiting failed calls. 5. Measure upstream call count, latency, and deduplication headers.

Evidence

200 identical requests fired at once became 1 upstream call.

Source

During a 60 second provider outage, the circuit breaker cut the calls that hit the dead API from 89 to 5.

Source

The server sends one upstream call at a time, so 20 different prompts sent together took 30.2 s instead of 1.5 s.

Source

The same 20 prompts now take 3.0 s at the default settings and 1.5 s with --concurrency 32.

Source