Qwen3.8-27B on 2 RTX 3090: My First Local Model I Actually Trust

说明与用途

作者在两块RTX 3090上部署了本地Qwen3.8‑27B LLM，使用vLLM进行高效推理，并将推理与辅助服务拆分到两台机器上，实现了可靠、私密且成本可控的日常工作流程，无需外部API。

问题

Running a large language model locally for daily work tasks without relying on external APIs, while managing hardware constraints and privacy concerns.

结果

The author achieved a reliable, private, and cost‑effective local LLM setup that handles real work tasks, reducing dependence on cloud APIs and improving privacy.

工作流

• Set up a hub machine (ArDi) with 56‑core CPU, 64 GB RAM, 52 Docker containers for assistant core, Whisper ASR, embeddings, and homelab‑monitor.
• Deploy a separate inference box (Vader) with 2× RTX 3090, 48 GB VRAM, running vLLM with tensor‑parallelism and port 8010.
• Connect the two boxes over Tailscale for secure communication.
• Use vLLM’s OpenAI‑compatible API to serve Qwen3.8‑27B (262K context) to the assistant core.
• Monitor VRAM and power consumption per process with homelab‑monitor, calculating energy cost.
• Run daily tasks: email summarization, compliance table creation, net‑worth rollup, and homelab benchmarking.
• Iterate on prompts and chain complexity, stopping to double‑check only the most agentic chains.

证据

I stopped worrying. My assistant, Jarvis, runs its daily tasks on a 27B model on my own hardware.

来源

Under load Vader draws ~685 W at 80 °C. It's a server, not a laptop.

来源

The whole hub ran 4.97 kWh this week: 1.30 BGN / 7 days.

来源