Homelab census: 41 containers, one 6 GB GPU, and where my AI agents run

Описание и применение

Автор создал подробный учёт контейнеров и GPU‑памяти в домашней лаборатории, определил правила маршрутизации задач между локальными и облачными моделями, учитывая контекст окна и стоимость ввода, и внедрил мониторинг и автоматический переключатель.

Задача

The author needed to understand what AI workloads were running on which hardware, how GPU memory was being used, and how to route tasks between local and cloud models to balance cost, latency, and security.

Результат

The author achieved a clear mapping of AI workloads to hardware, identified GPU memory constraints, and established a cost‑aware routing strategy that balances local privacy with cloud speed. The system now automatically fails over to local or cloud models based on health checks and credit limits.

Рабочий процесс

• 1. Inventory all physical nodes and their hardware specs.
• 2. List all LXC and Docker containers on each node and their purposes.
• 3. Measure GPU memory usage for each loaded model (e.g., 27B uses 0.5 GB on GPU, 17.8 GB on CPU).
• 4. Calculate total CPU threads and RAM usage across all nodes.
• 5. Identify duplicate or stale containers (e.g., twin agent box).
• 6. Define routing rule: use local model only if it meets 64K context and fits GPU; otherwise use cloud.
• 7. Assign each job to a target (local Ollama, OpenRouter, OpenCode, ComfyUI, Whisper/Piper).
• 8. Implement watchdogs to monitor model health and credit balance, with failover logic.
• 9. Measure cost and performance of cloud models over 30 days to inform pricing decisions.
• 10. Optimize local workloads by unloading idle models and reducing prompt size.
• 11. Document findings and share lessons learned.

Доказательства

The 27B was "running on the GPU" in the same sense that I am running a marathon when I walk to the shop.

Источник

The rule the whole routing falls out of: the agent framework I run refuses any model with less than a 64K context window

Источник

On 7 September every paid model "failed" its health check at once, so the watchdog moved four agents back to the local model.

Источник

The agent moved there for the extra RAM, and it was the right call, but every reboot is a queue of cold‑starting databases fighting over one spindle.

Источник