The $2,000 Inference Server: Standing Up Local AI on Ten-Year-Old Hardware

Описание и применение

Сервер инференса на старом оборудовании стоимостью 2000 долларов обеспечивает тысячи запросов в день, достигая 41 токен/сек на модели 26 B и 15 токен/сек на модели 27 B, при этом полностью автономен от облачных API.

Задача

Running a high‑throughput local inference server on affordable, legacy hardware while avoiding costly cloud API bills.

Результат

A $2,000 inference server on ten‑year‑old hardware achieved production‑grade throughput, proving that legacy GPUs can serve large LLM workloads when tuned for their constraints.

Рабочий процесс

• 1. Procure legacy GPU and CPU components from surplus market
• 2. Assemble server with new case, PSU, and cooler
• 3. Install 128 GB DDR4 ECC RAM and 1 TB NVMe SSD
• 4. Install NVIDIA drivers and CUDA 11.0 (compatible with Pascal)
• 5. Build and benchmark LLMs (Gemma 4 26B‑A4B, 27B Q6_K) with quantization and DP4A kernels
• 6. Configure single‑stream and parallel‑slot inference pipelines
• 7. Deploy OpenAI‑compatible endpoint via Portainer API
• 8. Monitor performance, swap models, and log metrics
• 9. Iterate on flags, precision, and context size based on measured results

Доказательства

The machine paid for itself before I finished tuning it.

Источник

A 26B mixture-of-experts model decodes at 41 tokens/sec.

Источник

Over one measured 40‑hour production window the stack processed 8,129 requests with a 0.17% failure rate and zero manual interventions.

Источник