Fifty API keys, one queue, ten GPU workers: the parts of a shared LLM that are not the model

Description and use

The demo showed how fifty teams can share a single LLM endpoint, manage access via keys, a shared queue, and ten GPU workers, while tracking costs per key.

Problem

Managing a self‑hosted LLM for a large organization requires handling many teams, accounting, and scaling without changing client code.

Outcome

The demo proved that a single endpoint with per‑key scoping, a shared queue, and worker concurrency can support fifty teams while providing clear cost attribution and scalable scaling decisions.

Workflow

• Issue one API key per team and use a single endpoint
• Configure ten GPU workers with a concurrency of 8 each
• All keys feed a shared queue; workers claim jobs when slots free
• Monitor backlog and throughput via the panel to decide scaling
• Use the cost panel to split hourly worker cost by key tokens

Evidence

The demo issues a key per team, often per use case: roadmap, competitive intel, security policy Q&A, board reporting. Fifty in total, and each one is allowed on a single endpoint, nexora-private-gpt/enterprise-chat.

Source

Ten workers serve one workload, and each takes up to 8 requests at the same time (its concurrency), so the fleet has 80 slots.

Source

The panel prices reserved worker time and only uses tokens to split the total: project_cost = reserved_hours * hourly_price ... key_cost = key_tokens / project_tokens * project_cost

Source