The $2,000 Inference Server: Standing Up Local AI on Ten-Year-Old Hardware

Описание и применение

Сервер инференса за 2000 долларов, построенный на десятилетней старой графической карте Tesla P40, успешно обслуживает тысячи запросов в день, демонстрируя, что устаревшее оборудование может быть эффективным при правильной настройке и измерениях.

Задача

Running a high‑volume local inference server on inexpensive, legacy hardware while maintaining performance and reliability.

Результат

The $2,000 build achieved high throughput (up to 41 tokens/sec on a 26B Mixture‑of‑Experts model) and low failure rates, proving that legacy hardware can support demanding inference workloads when properly configured and measured.

Рабочий процесс

• 1. Source used components from enterprise‑surplus market and new parts from Newegg.
• 2. Assemble server with Supermicro H11SSL‑i motherboard, AMD EPYC 7302P CPU, and two Tesla P40 GPUs.
• 3. Install operating system, Docker, and Portainer for container orchestration.
• 4. Build and deploy llama.cpp and other inference binaries with quantized models (e.g., Gemma 4 26B‑A4B Q8, 27B Q6_K).
• 5. Configure single‑GPU model residency, swap logic, and OpenAI‑compatible endpoint.
• 6. Benchmark performance: measure tokens/sec, context window, VRAM usage, and failure rates.
• 7. Iterate on flags and environment variables, testing one variable at a time to isolate effects.
• 8. Monitor production logs over 40 hours to validate stability and throughput.
• 9. Document findings and update configuration for future model swaps.

Доказательства

Over one measured 40‑hour production window the stack processed 8,129 requests with a 0.17% failure rate and zero manual interventions.

Источник

A 26B mixture‑of‑experts model (Gemma 4 26B‑A4B, Q8) decodes at 41 tokens/sec.

Источник

The P40 is the heart of the build and the source of most of the pain.

Источник