The $2,000 Inference Server: Standing Up Local AI on Ten-Year-Old Hardware

说明与用途

使用2016年 NVIDIA Tesla P40 构建本地推理服务器，成本约2000美元，可每天处理数千请求，展示了在硬件上测量如何驱动性能调优。

问题

Running a high‑throughput local inference server on affordable, legacy hardware while avoiding costly cloud API bills.

结果

The $2,000 build delivers reliable, low‑cost inference for thousands of agent requests daily, achieving competitive throughput on legacy hardware and providing a practical benchmark for similar homelab deployments.

工作流

• 1. Source used components (CPU, GPU, memory, storage) from surplus market.
• 2. Assemble hardware in a new case with PSU and cooler.
• 3. Install OS, Docker, and Portainer for container orchestration.
• 4. Build and deploy llama.cpp binaries with quantized models (Q6_K, Q8).
• 5. Configure single‑GPU model loading via Portainer API, swapping models as needed.
• 6. Benchmark each model’s throughput and VRAM usage, recording results.
• 7. Tune flags (row‑split, quantization, speculative decoding) one at a time, measuring impact.
• 8. Deploy OpenAI‑compatible endpoint to route agent requests to the server.
• 9. Monitor production logs for failures and performance metrics.
• 10. Iterate on model selection and flag tuning based on measured data.

证据

Over one measured 40‑hour production window the stack processed 8,129 requests with a 0.17% failure rate and zero manual interventions.

来源

A 26B mixture‑of‑experts model (Gemma 4 26B‑A4B, Q8) decodes at 41 tokens/sec.

来源

The P40 is the heart of the build and the source of most of the pain.

来源