Qwen3.8-27B on 2 RTX 3090: My First Local Model I Actually Trust

Описание и применение

Автор реализовал локальный LLM Qwen3.8‑27B на двух RTX 3090, разделив задачи между двумя машинами и используя vLLM для высокопроизводительного обслуживания. Это обеспечило надёжную работу без внешних API, сохранив конфиденциальность и контроль затрат.

Задача

Running a large language model locally for daily work tasks without relying on external APIs, while managing hardware constraints and privacy concerns.

Результат

The author achieved a reliable, private, and cost‑effective local LLM setup that handles real work tasks, reducing dependence on cloud APIs and improving privacy.

Рабочий процесс

• Set up a hub machine (ArDi) with 56‑core CPU, 64 GB RAM, 52 Docker containers for assistant core, Whisper ASR, embeddings, and homelab‑monitor.
• Deploy a separate inference box (Vader) with 2× RTX 3090, 48 GB VRAM, running vLLM with tensor‑parallelism and port 8010.
• Connect the two boxes over Tailscale for secure communication.
• Use vLLM’s OpenAI‑compatible API to serve Qwen3.8‑27B (262K context) to the assistant core.
• Monitor VRAM and power consumption per process with homelab‑monitor, calculating energy cost.
• Run daily tasks: email summarization, compliance table creation, net‑worth rollup, and homelab benchmarking.
• Iterate on prompts and chain complexity, stopping to double‑check only the most agentic chains.

Доказательства

I stopped worrying. My assistant, Jarvis, runs its daily tasks on a 27B model on my own hardware.

Источник

Under load Vader draws ~685 W at 80 °C. It's a server, not a laptop.

Источник

The whole hub ran 4.97 kWh this week: 1.30 BGN / 7 days.

Источник