Ollama, vLLM & Private LLMs: Routing Your Own Models Next to Your Cloud Keys

Описание и применение

Команда может маршрутизировать запросы к облачным и локальным LLM через один эндпоинт SentioGate, упрощая код, обеспечивая отказоустойчивость и единый телеметрический поток.

Задача

Teams often split logic between cloud and local LLMs, leading to multiple SDKs, error shapes, streaming formats, no failover, and code churn when models change or outages occur.

Результат

Teams can now route all LLM requests through a single endpoint, simplifying code, enabling failover, and providing unified telemetry and cost control across cloud and local models.

Рабочий процесс

• Register a custom provider with provider_id, base_url, api_key, and model list via SentioGate API.
• Ensure the endpoint is internet‑reachable, secured with HTTPS and a bearer token.
• Call the router using the provider_id/model syntax (e.g., my‑vllm/qwen2.5:14b) from any SDK.
• Optionally use "model":"auto" to let the router pick the best model from the pool.
• Monitor routing decisions, costs, and telemetry in the SentioGate dashboard.
• Adjust provider enablement or spend caps in the Vault to control model selection and guardrails.

Доказательства

Most teams end up with something like this: ... two SDKs, two error shapes, two streaming formats, no failover and no shared telemetry.

Источник

A router fixes the plumbing: one endpoint, one request format, one stream format, and one place to see what ran and what it cost.

Источник

With auto, SentioGate picks a model per request from everything you've enabled.

Источник