LLM proxy benchmark: 200 duplicate calls became 1 (and a 30 s bug)

说明与用途

LLM 代理将 200 条相同请求压缩为一次调用，成本降低 99.5%，延迟无增。故障时将失败请求从 89 降至 5。修复了顺序调用错误，20 条提示的吞吐量从 30.2 秒降至 3.0 秒（默认）或 1.5 秒（高并发）。

问题

Duplicate identical LLM requests can cause unnecessary paid calls and latency, especially during retry storms or agent loops.

结果

The proxy reduced 200 duplicate calls to 1, cutting cost by ~99.5% without adding latency. It also limited failed calls during outages from 89 to 5. A bug causing sequential upstream calls was fixed in v2.0.3, improving 20‑prompt throughput from 30.2 s to 3.0 s (default) and 1.5 s with higher concurrency.

工作流

• 1. Configure the OpenAI SDK to use the proxy base URL. 2. Send 200 identical chat completion requests concurrently. 3. Proxy deduplicates identical in‑flight requests into a single upstream call. 4. Proxy applies circuit breaking during provider outages, limiting failed calls. 5. Measure upstream call count, latency, and deduplication headers.

证据

200 identical requests fired at once became 1 upstream call.

来源

During a 60 second provider outage, the circuit breaker cut the calls that hit the dead API from 89 to 5.

来源

The server sends one upstream call at a time, so 20 different prompts sent together took 30.2 s instead of 1.5 s.

来源

The same 20 prompts now take 3.0 s at the default settings and 1.5 s with --concurrency 32.

来源