The $2,000 Inference Server: Standing Up Local AI on Ten-Year-Old Hardware

Description and use

Building a local inference server on a 2016 NVIDIA Tesla P40 for ~$2,000 enables thousands of daily requests with low cost and high reliability, showing how on‑hardware measurements drive performance tuning.

Problem

Running a high‑throughput local inference server on affordable, legacy hardware while avoiding costly cloud API bills.

Outcome

The $2,000 build delivers reliable, low‑cost inference for thousands of agent requests daily, achieving competitive throughput on legacy hardware and providing a practical benchmark for similar homelab deployments.

Workflow

• 1. Source used components (CPU, GPU, memory, storage) from surplus market.
• 2. Assemble hardware in a new case with PSU and cooler.
• 3. Install OS, Docker, and Portainer for container orchestration.
• 4. Build and deploy llama.cpp binaries with quantized models (Q6_K, Q8).
• 5. Configure single‑GPU model loading via Portainer API, swapping models as needed.
• 6. Benchmark each model’s throughput and VRAM usage, recording results.
• 7. Tune flags (row‑split, quantization, speculative decoding) one at a time, measuring impact.
• 8. Deploy OpenAI‑compatible endpoint to route agent requests to the server.
• 9. Monitor production logs for failures and performance metrics.
• 10. Iterate on model selection and flag tuning based on measured data.

Evidence

Over one measured 40‑hour production window the stack processed 8,129 requests with a 0.17% failure rate and zero manual interventions.

Source

A 26B mixture‑of‑experts model (Gemma 4 26B‑A4B, Q8) decodes at 41 tokens/sec.

Source

The P40 is the heart of the build and the source of most of the pain.

Source