LiteLLM on the Homelab: One Proxy, Every Model, Waterfall Routing Included

Описание и применение

LiteLLM объединяет множество локальных и облачных LLM‑эндпоинтов в один прокси, обеспечивая автоматический откат, балансировку нагрузки, отслеживание использования и ограничение запросов.

Задача

Managing multiple LLM endpoints in a homelab caused code duplication, configuration hell, no fallback, no load balancing, manual usage tracking, and lack of rate limiting.

Результат

A single proxy unified endpoint reduced code complexity, enabled automatic fallback, load balancing across local instances, built‑in usage tracking, and rate limiting, improving reliability and cost visibility.

Рабочий процесс

• Configure LiteLLM with desired routing strategy (simple list, load balancing, or fallback chain).
• Deploy LiteLLM as a Docker container exposing port 4000.
• Update application code to point to http://homelab:4000/v1/* using any OpenAI SDK.
• Define model_list in YAML to specify providers, models, API bases, and limits.
• Run LiteLLM; it routes requests per the configured strategy, handling fallbacks and load balancing automatically.

Доказательства

Each requires: Different API keys, different endpoints, different SDKs

Источник

Single endpoint: All applications talk to http://homelab:4000/v1/*

Источник

docker run -d \
--name litellm \
--restart unless-stopped \
-p 4000:4000 \
-v /mnt/nvme-pool/apps/litellm:/litellm \
-e OPENAI_API_KEY

Источник