LiteLLM on the Homelab: One Proxy, Every Model, Waterfall Routing Included

说明与用途

LiteLLM 将多个本地和云端 LLM 端点整合为单一代理，提供自动回退、负载均衡、使用跟踪和速率限制。

问题

Managing multiple LLM endpoints in a homelab caused code duplication, configuration hell, no fallback, no load balancing, manual usage tracking, and lack of rate limiting.

结果

A single proxy unified endpoint reduced code complexity, enabled automatic fallback, load balancing across local instances, built‑in usage tracking, and rate limiting, improving reliability and cost visibility.

工作流

• Configure LiteLLM with desired routing strategy (simple list, load balancing, or fallback chain).
• Deploy LiteLLM as a Docker container exposing port 4000.
• Update application code to point to http://homelab:4000/v1/* using any OpenAI SDK.
• Define model_list in YAML to specify providers, models, API bases, and limits.
• Run LiteLLM; it routes requests per the configured strategy, handling fallbacks and load balancing automatically.

证据

Each requires: Different API keys, different endpoints, different SDKs

来源

Single endpoint: All applications talk to http://homelab:4000/v1/*

来源

docker run -d \
--name litellm \
--restart unless-stopped \
-p 4000:4000 \
-v /mnt/nvme-pool/apps/litellm:/litellm \
-e OPENAI_API_KEY

来源