Fifty API keys, one queue, ten GPU workers: the parts of a shared LLM that are not the model

Описание и применение

Демонстрация показала, как 50 команд могут совместно использовать один эндпоинт LLM, управляя доступом через ключи, очередь и 10 GPU‑рабочих, а также отслеживать затраты по ключу.

Задача

Managing a self‑hosted LLM for a large organization requires handling many teams, accounting, and scaling without changing client code.

Результат

The demo proved that a single endpoint with per‑key scoping, a shared queue, and worker concurrency can support fifty teams while providing clear cost attribution and scalable scaling decisions.

Рабочий процесс

• Issue one API key per team and use a single endpoint
• Configure ten GPU workers with a concurrency of 8 each
• All keys feed a shared queue; workers claim jobs when slots free
• Monitor backlog and throughput via the panel to decide scaling
• Use the cost panel to split hourly worker cost by key tokens

Доказательства

The demo issues a key per team, often per use case: roadmap, competitive intel, security policy Q&A, board reporting. Fifty in total, and each one is allowed on a single endpoint, nexora-private-gpt/enterprise-chat.

Источник

Ten workers serve one workload, and each takes up to 8 requests at the same time (its concurrency), so the fleet has 80 slots.

Источник

The panel prices reserved worker time and only uses tokens to split the total: project_cost = reserved_hours * hourly_price ... key_cost = key_tokens / project_tokens * project_cost

Источник