Ollama, vLLM & Private LLMs: Routing Your Own Models Next to Your Cloud Keys

Описание и применение

SentioGate объединяет облачные и локальные модели LLM в один эндпоинт, обеспечивая единый формат запросов и потоков, автоматический выбор модели и сброс ошибок, а также единый дашборд телеметрии.

Задача

Teams often split requests between cloud and local LLMs, leading to multiple SDKs, error shapes, streaming formats, no failover, and code changes for each model or outage.

Результат

Teams can route both cloud and local LLM requests through a single endpoint, simplifying integration, adding failover, and providing unified telemetry and cost tracking.

Рабочий процесс

• Register a custom provider with provider_id, base_url, api_key, and model list via POST /custom-providers
• Ensure the endpoint is internet‑reachable, HTTPS, and protected with a key
• Use AutoRouterClient or AutoRouter SDK to call models with model="provider_id/model" or "auto"
• SentioGate selects model per request based on capability, context window, cost, and recent reliability
• If a provider fails, auto falls back to next candidate before client receives bytes
• Telemetry and cost data appear in the SentioGate dashboard
• Streaming responses use OpenAI‑style SSE, identical to cloud streams

Доказательства

Most teams end up with something like this: if task_difficulty > 0.7: response = openai_client.chat(...) # cloud else: response = ollama_client.generate(...) # local

Источник

A router fixes the plumbing: one endpoint, one request format, one stream format, and one place to see what ran and what it cost.

Источник

With auto, SentioGate picks a model per request from everything you've enabled. It looks at whether a model is capable enough for the task, whether the conversation fits its context window, what it costs, and how fast and reliable it has been recently.

Источник