Ollama, vLLM & Private LLMs: Routing Your Own Models Next to Your Cloud Keys

说明与用途

SentioGate 将云端和本地 LLM 集成到单一端点，提供统一的请求/流格式、自动模型选择与故障转移，以及统一的遥测仪表盘。

问题

Teams often split requests between cloud and local LLMs, leading to multiple SDKs, error shapes, streaming formats, no failover, and code changes for each model or outage.

结果

Teams can route both cloud and local LLM requests through a single endpoint, simplifying integration, adding failover, and providing unified telemetry and cost tracking.

工作流

• Register a custom provider with provider_id, base_url, api_key, and model list via POST /custom-providers
• Ensure the endpoint is internet‑reachable, HTTPS, and protected with a key
• Use AutoRouterClient or AutoRouter SDK to call models with model="provider_id/model" or "auto"
• SentioGate selects model per request based on capability, context window, cost, and recent reliability
• If a provider fails, auto falls back to next candidate before client receives bytes
• Telemetry and cost data appear in the SentioGate dashboard
• Streaming responses use OpenAI‑style SSE, identical to cloud streams

证据

Most teams end up with something like this: if task_difficulty > 0.7: response = openai_client.chat(...) # cloud else: response = ollama_client.generate(...) # local

来源

A router fixes the plumbing: one endpoint, one request format, one stream format, and one place to see what ran and what it cost.

来源

With auto, SentioGate picks a model per request from everything you've enabled. It looks at whether a model is capable enough for the task, whether the conversation fits its context window, what it costs, and how fast and reliable it has been recently.

来源