Ollama, vLLM & Private LLMs: Routing Your Own Models Next to Your Cloud Keys

说明与用途

团队可以通过 SentioGate 的单一端点将请求路由到云端和本地 LLM，简化代码，实现故障转移，并提供统一的遥测。

问题

Teams often split logic between cloud and local LLMs, leading to multiple SDKs, error shapes, streaming formats, no failover, and code churn when models change or outages occur.

结果

Teams can now route all LLM requests through a single endpoint, simplifying code, enabling failover, and providing unified telemetry and cost control across cloud and local models.

工作流

• Register a custom provider with provider_id, base_url, api_key, and model list via SentioGate API.
• Ensure the endpoint is internet‑reachable, secured with HTTPS and a bearer token.
• Call the router using the provider_id/model syntax (e.g., my‑vllm/qwen2.5:14b) from any SDK.
• Optionally use "model":"auto" to let the router pick the best model from the pool.
• Monitor routing decisions, costs, and telemetry in the SentioGate dashboard.
• Adjust provider enablement or spend caps in the Vault to control model selection and guardrails.

证据

Most teams end up with something like this: ... two SDKs, two error shapes, two streaming formats, no failover and no shared telemetry.

来源

A router fixes the plumbing: one endpoint, one request format, one stream format, and one place to see what ran and what it cost.

来源

With auto, SentioGate picks a model per request from everything you've enabled.

来源