Free Models and a Free Server: A 100-Line LLM Failover Proxy

Описание и применение

Небольшой прокси‑сервис на Python автоматически переключает запросы между бесплатным управляемым API MonkeyCode и локальной моделью, развернутой на бесплатном сервере, обеспечивая непрерывную работу при ограничениях API.

Задача

Free LLM APIs are unreliable due to rate limits; users need a seamless fallback to a self-hosted model without manual switching.

Результат

The proxy delivers low‑latency responses from the managed API during normal operation and automatically switches to a local model during spikes or rate limits, providing resilience without cost.

Рабочий процесс

• Provision a free server via the MonkeyCode console and install Docker.
• Deploy a local OpenAI‑compatible model (e.g., Ollama with qwen2.5:1.5b) on the server.
• Write a 100‑line Python FastAPI proxy that calls the free API first and retries with the local model on 429 or connection errors.
• Set environment variables for FREE_API_KEY, FREE_API_URL, LOCAL_API_URL, and LOCAL_MODEL.
• Run the proxy and point client applications to its OpenAI‑compatible endpoint.
• Monitor logs to verify failover behavior during rate limits.

Доказательства

A tiny proxy that routes between a managed free API and a self-hosted model on a free server turns intermittent rate limits into a non-event.

Источник

The proxy tries the managed API first, watches for failures, and fails over to the local model when needed.

Источник