Concurrency Limits Aren’t Enough for LLM APIs

Description and use

Traditional concurrency limits ignore request size, causing inefficient resource use. async-bulkhead-llm limits both concurrent requests and in-flight tokens, balancing load for LLM APIs.

Problem

Traditional concurrency limits treat every request as equal, which is ineffective for LLM workloads where request sizes vary widely in token count, leading to inefficient resource utilization.

Outcome

By limiting both concurrent requests and in-flight tokens, the system prevents large requests from monopolizing capacity, improving responsiveness for interactive workloads.

Workflow

• Define maxConcurrent and maxInFlightTokens in AsyncBulkhead configuration.
• Wrap LLM API calls with the bulkhead instance.
• When a request arrives, check both concurrency and token budgets before admission.
• If budgets are exceeded, queue or reject the request.
• Monitor token usage to adjust maxInFlightTokens dynamically if needed.

Evidence

Instead of limiting only concurrent requests, it can bound two things at once: concurrent requests + estimated in-flight tokens

Source