Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

The $2,000 Inference Server: Standing Up Local AI on Ten-Year-Old Hardware

Описание и применение

Сборка локального сервера инференса на старом NVIDIA Tesla P40 за ~$2 000 позволяет обрабатывать тысячи запросов в день с низкой стоимостью и высокой надёжностью, демонстрируя, как измерения на собственном оборудовании помогают оптимизировать производительность.

Задача

Running a high‑throughput local inference server on affordable, legacy hardware while avoiding costly cloud API bills.

Результат

The $2,000 build delivers reliable, low‑cost inference for thousands of agent requests daily, achieving competitive throughput on legacy hardware and providing a practical benchmark for similar homelab deployments.

Рабочий процесс

  • 1. Source used components (CPU, GPU, memory, storage) from surplus market.
  • 2. Assemble hardware in a new case with PSU and cooler.
  • 3. Install OS, Docker, and Portainer for container orchestration.
  • 4. Build and deploy llama.cpp binaries with quantized models (Q6_K, Q8).
  • 5. Configure single‑GPU model loading via Portainer API, swapping models as needed.
  • 6. Benchmark each model’s throughput and VRAM usage, recording results.
  • 7. Tune flags (row‑split, quantization, speculative decoding) one at a time, measuring impact.
  • 8. Deploy OpenAI‑compatible endpoint to route agent requests to the server.
  • 9. Monitor production logs for failures and performance metrics.
  • 10. Iterate on model selection and flag tuning based on measured data.

Доказательства

Over one measured 40‑hour production window the stack processed 8,129 requests with a 0.17% failure rate and zero manual interventions.

Источник

A 26B mixture‑of‑experts model (Gemma 4 26B‑A4B, Q8) decodes at 41 tokens/sec.

Источник

The P40 is the heart of the build and the source of most of the pain.

Источник

  • Дата записи: 2026-09-15T14:45:23.499Z