The $2,000 Inference Server: Standing Up Local AI on Ten-Year-Old Hardware

Description and use

A $2,000 inference server built on ten‑year‑old Tesla P40 GPUs handles thousands of daily agent requests with high throughput and low failure rates, proving legacy hardware can meet demanding AI workloads when properly configured and measured.

Problem

Running a high‑volume local inference server on inexpensive, legacy hardware while maintaining performance and reliability.

Outcome

The $2,000 build achieved high throughput (up to 41 tokens/sec on a 26B Mixture‑of‑Experts model) and low failure rates, proving that legacy hardware can support demanding inference workloads when properly configured and measured.

Workflow

• 1. Source used components from enterprise‑surplus market and new parts from Newegg.
• 2. Assemble server with Supermicro H11SSL‑i motherboard, AMD EPYC 7302P CPU, and two Tesla P40 GPUs.
• 3. Install operating system, Docker, and Portainer for container orchestration.
• 4. Build and deploy llama.cpp and other inference binaries with quantized models (e.g., Gemma 4 26B‑A4B Q8, 27B Q6_K).
• 5. Configure single‑GPU model residency, swap logic, and OpenAI‑compatible endpoint.
• 6. Benchmark performance: measure tokens/sec, context window, VRAM usage, and failure rates.
• 7. Iterate on flags and environment variables, testing one variable at a time to isolate effects.
• 8. Monitor production logs over 40 hours to validate stability and throughput.
• 9. Document findings and update configuration for future model swaps.

Evidence

Over one measured 40‑hour production window the stack processed 8,129 requests with a 0.17% failure rate and zero manual interventions.

Source

A 26B mixture‑of‑experts model (Gemma 4 26B‑A4B, Q8) decodes at 41 tokens/sec.

Source

The P40 is the heart of the build and the source of most of the pain.

Source