The $2,000 Inference Server: Standing Up Local AI on Ten-Year-Old Hardware

Description and use

A $2,000 inference server built on ten‑year‑old hardware handles thousands of daily requests, delivering 41 tokens/sec on a 26B model and 15 tokens/sec on a 27B model, entirely self‑hosted without cloud APIs.

Problem

Running a high‑throughput local inference server on affordable, legacy hardware while avoiding costly cloud API bills.

Outcome

A $2,000 inference server on ten‑year‑old hardware achieved production‑grade throughput, proving that legacy GPUs can serve large LLM workloads when tuned for their constraints.

Workflow

• 1. Procure legacy GPU and CPU components from surplus market
• 2. Assemble server with new case, PSU, and cooler
• 3. Install 128 GB DDR4 ECC RAM and 1 TB NVMe SSD
• 4. Install NVIDIA drivers and CUDA 11.0 (compatible with Pascal)
• 5. Build and benchmark LLMs (Gemma 4 26B‑A4B, 27B Q6_K) with quantization and DP4A kernels
• 6. Configure single‑stream and parallel‑slot inference pipelines
• 7. Deploy OpenAI‑compatible endpoint via Portainer API
• 8. Monitor performance, swap models, and log metrics
• 9. Iterate on flags, precision, and context size based on measured results

Evidence

The machine paid for itself before I finished tuning it.

Source

A 26B mixture-of-experts model decodes at 41 tokens/sec.

Source

Over one measured 40‑hour production window the stack processed 8,129 requests with a 0.17% failure rate and zero manual interventions.

Source