Concurrency Limits Aren’t Enough for LLM APIs

说明与用途

传统并发限制忽略请求大小，导致资源利用率低下。async-bulkhead-llm 同时限制并发请求数和在途令牌数，平衡 LLM API 的负载。

问题

Traditional concurrency limits treat every request as equal, which is ineffective for LLM workloads where request sizes vary widely in token count, leading to inefficient resource utilization.

结果

By limiting both concurrent requests and in-flight tokens, the system prevents large requests from monopolizing capacity, improving responsiveness for interactive workloads.

工作流

• Define maxConcurrent and maxInFlightTokens in AsyncBulkhead configuration.
• Wrap LLM API calls with the bulkhead instance.
• When a request arrives, check both concurrency and token budgets before admission.
• If budgets are exceeded, queue or reject the request.
• Monitor token usage to adjust maxInFlightTokens dynamically if needed.

证据

Instead of limiting only concurrent requests, it can bound two things at once: concurrent requests + estimated in-flight tokens

来源