Building Sarrera: Self-Hosted Enterprise AI Inference Gateway with RBAC, Token Quotas & Telemetry

Описание и применение

Sarrera позволяет компаниям запускать генеративный ИИ локально, обеспечивая контроль доступа, квоты токенов и наблюдаемость, устраняя риски конфиденциальности и непредсказуемые расходы на облако.

Задача

Engineering teams face privacy, billing, and hardware fragmentation challenges when adopting generative AI.

Результат

Sarrera enables enterprises to run generative AI locally with fine‑grained RBAC, token quotas, and real‑time telemetry, eliminating cloud privacy risks and uncontrolled billing while efficiently utilizing heterogeneous GPU/CPU resources.

Рабочий процесс

• Deploy Sarrera via Docker Compose
• Configure Caddy for TLS and reverse proxy
• Set up LiteLLM with model routing and quota tiers
• Deploy Langfuse, Open WebUI, MinIO, PostgreSQL, and Ollama nodes
• Use Admin Control Panel to register nodes and issue API keys
• Integrate with VS Code by adding Sarrera model to config.json

Доказательства

Engineering teams worldwide face a common dilemma when adopting generative AI: Cloud AI privacy & security risks, uncontrolled cloud billing, hardware fragmentation

Источник

Sarrera decouples client IDEs from physical compute hardware, shielding your internal network behind a single perimeter reverse proxy

Источник

LiteLLM checks if the requested model is whitelisted for their tier. If a junior developer with tier-basic requests premium-reasoning, the gateway immediately returns HTTP 403 Forbidden in 10 milliseconds

Источник