Ollama, vLLM & Private LLMs: Routing Your Own Models Next to Your Cloud Keys

Description and use

Teams can route requests to cloud and local LLMs through a single SentioGate endpoint, simplifying code, enabling failover, and providing unified telemetry.

Problem

Teams often split logic between cloud and local LLMs, leading to multiple SDKs, error shapes, streaming formats, no failover, and code churn when models change or outages occur.

Outcome

Teams can now route all LLM requests through a single endpoint, simplifying code, enabling failover, and providing unified telemetry and cost control across cloud and local models.

Workflow

• Register a custom provider with provider_id, base_url, api_key, and model list via SentioGate API.
• Ensure the endpoint is internet‑reachable, secured with HTTPS and a bearer token.
• Call the router using the provider_id/model syntax (e.g., my‑vllm/qwen2.5:14b) from any SDK.
• Optionally use "model":"auto" to let the router pick the best model from the pool.
• Monitor routing decisions, costs, and telemetry in the SentioGate dashboard.
• Adjust provider enablement or spend caps in the Vault to control model selection and guardrails.

Evidence

Most teams end up with something like this: ... two SDKs, two error shapes, two streaming formats, no failover and no shared telemetry.

Source

A router fixes the plumbing: one endpoint, one request format, one stream format, and one place to see what ran and what it cost.

Source

With auto, SentioGate picks a model per request from everything you've enabled.

Source