Building Sarrera: Self-Hosted Enterprise AI Inference Gateway with RBAC, Token Quotas & Telemetry

Description and use

Sarrera enables enterprises to run generative AI locally with fine‑grained RBAC, token quotas, and real‑time telemetry, eliminating cloud privacy risks and uncontrolled billing while efficiently utilizing heterogeneous GPU/CPU resources.

Problem

Engineering teams face privacy, billing, and hardware fragmentation challenges when adopting generative AI.

Outcome

Sarrera enables enterprises to run generative AI locally with fine‑grained RBAC, token quotas, and real‑time telemetry, eliminating cloud privacy risks and uncontrolled billing while efficiently utilizing heterogeneous GPU/CPU resources.

Workflow

• Deploy Sarrera via Docker Compose
• Configure Caddy for TLS and reverse proxy
• Set up LiteLLM with model routing and quota tiers
• Deploy Langfuse, Open WebUI, MinIO, PostgreSQL, and Ollama nodes
• Use Admin Control Panel to register nodes and issue API keys
• Integrate with VS Code by adding Sarrera model to config.json

Evidence

Engineering teams worldwide face a common dilemma when adopting generative AI: Cloud AI privacy & security risks, uncontrolled cloud billing, hardware fragmentation

Source

Sarrera decouples client IDEs from physical compute hardware, shielding your internal network behind a single perimeter reverse proxy

Source

LiteLLM checks if the requested model is whitelisted for their tier. If a junior developer with tier-basic requests premium-reasoning, the gateway immediately returns HTTP 403 Forbidden in 10 milliseconds

Source