LLM proxy benchmark: 200 duplicate calls became 1 (and a 30 s bug)

Описание и применение

Прокси LLM сократил 200 одинаковых запросов до одного, уменьшив расходы на 99,5 % без увеличения задержки. При отказе провайдера он ограничил неудачные запросы с 89 до 5. Исправлена ошибка последовательных вызовов, ускорив обработку 20 запросов с 30,2 с до 3,0 с (по умолчанию) и 1,5 с при повышенной конкуренции.

Задача

Duplicate identical LLM requests can cause unnecessary paid calls and latency, especially during retry storms or agent loops.

Результат

The proxy reduced 200 duplicate calls to 1, cutting cost by ~99.5% without adding latency. It also limited failed calls during outages from 89 to 5. A bug causing sequential upstream calls was fixed in v2.0.3, improving 20‑prompt throughput from 30.2 s to 3.0 s (default) and 1.5 s with higher concurrency.

Рабочий процесс

• 1. Configure the OpenAI SDK to use the proxy base URL. 2. Send 200 identical chat completion requests concurrently. 3. Proxy deduplicates identical in‑flight requests into a single upstream call. 4. Proxy applies circuit breaking during provider outages, limiting failed calls. 5. Measure upstream call count, latency, and deduplication headers.

Доказательства

200 identical requests fired at once became 1 upstream call.

Источник

During a 60 second provider outage, the circuit breaker cut the calls that hit the dead API from 89 to 5.

Источник

The server sends one upstream call at a time, so 20 different prompts sent together took 30.2 s instead of 1.5 s.

Источник

The same 20 prompts now take 3.0 s at the default settings and 1.5 s with --concurrency 32.

Источник