Ollama, vLLM & Private LLMs: Routing Your Own Models Next to Your Cloud Keys

Description and use

SentioGate unifies cloud and local LLMs behind a single endpoint, offering a single request/stream format, automatic model selection with failover, and unified telemetry.

Problem

Teams often split requests between cloud and local LLMs, leading to multiple SDKs, error shapes, streaming formats, no failover, and code changes for each model or outage.

Outcome

Teams can route both cloud and local LLM requests through a single endpoint, simplifying integration, adding failover, and providing unified telemetry and cost tracking.

Workflow

• Register a custom provider with provider_id, base_url, api_key, and model list via POST /custom-providers
• Ensure the endpoint is internet‑reachable, HTTPS, and protected with a key
• Use AutoRouterClient or AutoRouter SDK to call models with model="provider_id/model" or "auto"
• SentioGate selects model per request based on capability, context window, cost, and recent reliability
• If a provider fails, auto falls back to next candidate before client receives bytes
• Telemetry and cost data appear in the SentioGate dashboard
• Streaming responses use OpenAI‑style SSE, identical to cloud streams

Evidence

Most teams end up with something like this: if task_difficulty > 0.7: response = openai_client.chat(...) # cloud else: response = ollama_client.generate(...) # local

Source

A router fixes the plumbing: one endpoint, one request format, one stream format, and one place to see what ran and what it cost.

Source

With auto, SentioGate picks a model per request from everything you've enabled. It looks at whether a model is capable enough for the task, whether the conversation fits its context window, what it costs, and how fast and reliable it has been recently.

Source