Qwen3.8-27B on 2 RTX 3090: My First Local Model I Actually Trust

Description and use

The author deployed a local Qwen3.8‑27B LLM on two RTX 3090 GPUs, splitting inference and auxiliary services across two machines and using vLLM for efficient serving. This enabled reliable, private, and cost‑controlled daily workflows without external APIs.

Problem

Running a large language model locally for daily work tasks without relying on external APIs, while managing hardware constraints and privacy concerns.

Outcome

The author achieved a reliable, private, and cost‑effective local LLM setup that handles real work tasks, reducing dependence on cloud APIs and improving privacy.

Workflow

• Set up a hub machine (ArDi) with 56‑core CPU, 64 GB RAM, 52 Docker containers for assistant core, Whisper ASR, embeddings, and homelab‑monitor.
• Deploy a separate inference box (Vader) with 2× RTX 3090, 48 GB VRAM, running vLLM with tensor‑parallelism and port 8010.
• Connect the two boxes over Tailscale for secure communication.
• Use vLLM’s OpenAI‑compatible API to serve Qwen3.8‑27B (262K context) to the assistant core.
• Monitor VRAM and power consumption per process with homelab‑monitor, calculating energy cost.
• Run daily tasks: email summarization, compliance table creation, net‑worth rollup, and homelab benchmarking.
• Iterate on prompts and chain complexity, stopping to double‑check only the most agentic chains.

Evidence

I stopped worrying. My assistant, Jarvis, runs its daily tasks on a 27B model on my own hardware.

Source

Under load Vader draws ~685 W at 80 °C. It's a server, not a laptop.

Source

The whole hub ran 4.97 kWh this week: 1.30 BGN / 7 days.

Source