Homelab census: 41 containers, one 6 GB GPU, and where my AI agents run

Description and use

The author performed a detailed census of containers and GPU memory in a home lab, defined routing rules between local and cloud models based on context window and input cost, and implemented monitoring and failover mechanisms.

Problem

The author needed to understand what AI workloads were running on which hardware, how GPU memory was being used, and how to route tasks between local and cloud models to balance cost, latency, and security.

Outcome

The author achieved a clear mapping of AI workloads to hardware, identified GPU memory constraints, and established a cost‑aware routing strategy that balances local privacy with cloud speed. The system now automatically fails over to local or cloud models based on health checks and credit limits.

Workflow

• 1. Inventory all physical nodes and their hardware specs.
• 2. List all LXC and Docker containers on each node and their purposes.
• 3. Measure GPU memory usage for each loaded model (e.g., 27B uses 0.5 GB on GPU, 17.8 GB on CPU).
• 4. Calculate total CPU threads and RAM usage across all nodes.
• 5. Identify duplicate or stale containers (e.g., twin agent box).
• 6. Define routing rule: use local model only if it meets 64K context and fits GPU; otherwise use cloud.
• 7. Assign each job to a target (local Ollama, OpenRouter, OpenCode, ComfyUI, Whisper/Piper).
• 8. Implement watchdogs to monitor model health and credit balance, with failover logic.
• 9. Measure cost and performance of cloud models over 30 days to inform pricing decisions.
• 10. Optimize local workloads by unloading idle models and reducing prompt size.
• 11. Document findings and share lessons learned.

Evidence

The 27B was "running on the GPU" in the same sense that I am running a marathon when I walk to the shop.

Source

The rule the whole routing falls out of: the agent framework I run refuses any model with less than a 64K context window

Source

On 7 September every paid model "failed" its health check at once, so the watchdog moved four agents back to the local model.

Source

The agent moved there for the extra RAM, and it was the right call, but every reboot is a queue of cold‑starting databases fighting over one spindle.

Source