I Ran AI Offline for 7 Days. Was It Worth It?

Description and use

A user ran two open LLMs on a cheap mini‑PC offline, configured Ollama and a firewall, and achieved acceptable performance while ensuring no outbound traffic.

Problem

Running large language models offline on low‑power hardware without internet access to evaluate performance and security.

Outcome

The setup delivered 1–2 tokens per second on CPU and ~4–5 tokens per second on the integrated GPU, with acceptable latency for casual use. The offline configuration prevented any outbound data leakage, satisfying privacy concerns. The user concluded that the low cost and low power consumption made the experiment worthwhile.

Workflow

• 1. Install Ollama on the mini PC (requires internet for installation).
• 2. Install Vulkan drivers to enable the integrated GPU.
• 3. Pull Gemma 4 E4B (6.6 GB) and Qwen2.5‑Coder 7B (4.7 GB).
• 4. Configure Ollama systemd override to listen on 0.0.0.0:11434, limit to one loaded model, single parallel request, keep‑alive 15m, enable iGPU.
• 5. Verify API accessibility from the Mac via curl.
• 6. (Optional) Run Open WebUI in Docker on the same host for a browser chat interface.
• 7. Configure a firewall (ufw or Proxmox) to deny all outgoing traffic and allow only LAN access to ports 22, 11434, and 3000.
• 8. Test offline status by attempting an internet request and confirming failure.

Evidence

The mini PC is on my home network, but it can't reach the internet.

Source

OLLAMA_MAX_LOADED_MODELS=1: both models together don't fit in 10 GB, so only one stays in memory.

Source

ufw default deny outgoing

Source