The $2,000 Inference Server: Standing Up Local AI on Ten-Year-Old Hardware

说明与用途

一台价值2000美元、使用十年前Tesla P40显卡的推理服务器，能够每天处理数千个代理请求，展示了在正确配置和测量的前提下，旧硬件也能满足高需求的AI工作负载。

问题

Running a high‑volume local inference server on inexpensive, legacy hardware while maintaining performance and reliability.

结果

The $2,000 build achieved high throughput (up to 41 tokens/sec on a 26B Mixture‑of‑Experts model) and low failure rates, proving that legacy hardware can support demanding inference workloads when properly configured and measured.

工作流

• 1. Source used components from enterprise‑surplus market and new parts from Newegg.
• 2. Assemble server with Supermicro H11SSL‑i motherboard, AMD EPYC 7302P CPU, and two Tesla P40 GPUs.
• 3. Install operating system, Docker, and Portainer for container orchestration.
• 4. Build and deploy llama.cpp and other inference binaries with quantized models (e.g., Gemma 4 26B‑A4B Q8, 27B Q6_K).
• 5. Configure single‑GPU model residency, swap logic, and OpenAI‑compatible endpoint.
• 6. Benchmark performance: measure tokens/sec, context window, VRAM usage, and failure rates.
• 7. Iterate on flags and environment variables, testing one variable at a time to isolate effects.
• 8. Monitor production logs over 40 hours to validate stability and throughput.
• 9. Document findings and update configuration for future model swaps.

证据

Over one measured 40‑hour production window the stack processed 8,129 requests with a 0.17% failure rate and zero manual interventions.

来源

A 26B mixture‑of‑experts model (Gemma 4 26B‑A4B, Q8) decodes at 41 tokens/sec.

来源

The P40 is the heart of the build and the source of most of the pain.

来源