I Ran AI Offline for 7 Days. Was It Worth It?

说明与用途

用户在低功耗迷你电脑上离线运行两款开源 LLM，配置 Ollama 和防火墙，获得可接受的性能并确保无外部网络访问。

问题

Running large language models offline on low‑power hardware without internet access to evaluate performance and security.

结果

The setup delivered 1–2 tokens per second on CPU and ~4–5 tokens per second on the integrated GPU, with acceptable latency for casual use. The offline configuration prevented any outbound data leakage, satisfying privacy concerns. The user concluded that the low cost and low power consumption made the experiment worthwhile.

工作流

• 1. Install Ollama on the mini PC (requires internet for installation).
• 2. Install Vulkan drivers to enable the integrated GPU.
• 3. Pull Gemma 4 E4B (6.6 GB) and Qwen2.5‑Coder 7B (4.7 GB).
• 4. Configure Ollama systemd override to listen on 0.0.0.0:11434, limit to one loaded model, single parallel request, keep‑alive 15m, enable iGPU.
• 5. Verify API accessibility from the Mac via curl.
• 6. (Optional) Run Open WebUI in Docker on the same host for a browser chat interface.
• 7. Configure a firewall (ufw or Proxmox) to deny all outgoing traffic and allow only LAN access to ports 22, 11434, and 3000.
• 8. Test offline status by attempting an internet request and confirming failure.

证据

The mini PC is on my home network, but it can't reach the internet.

来源

OLLAMA_MAX_LOADED_MODELS=1: both models together don't fit in 10 GB, so only one stays in memory.

来源

ufw default deny outgoing

来源