What It Actually Takes to Run Your Own AI: One Consumer GPU, a Routed Model Mesh, and $100 a Month

Описание и применение

Система автономного ИИ на одном игровом GPU, маршрутизирующая запросы к дешевым моделям и использующая подписку $100/мес, обеспечивает ежедневные задачи с расходами менее $10/мес.

Задача

Many local‑vs‑cloud AI stories overestimate hardware needs and cost, leading to unrealistic expectations for solo operators.

Результат

The system delivers daily AI tasks—document classification, RAG, transcription, email triage, code drafting—on a single gaming GPU, with a routed mesh of cheap models and a modest $100/month subscription, achieving sub‑$10/month API costs and minimal power consumption.

Рабочий процесс

• 1. Deploy a bare‑metal host with an RTX 4070 Ti and Docker containers for other services.
• 2. Run Ollama locally with Llama 3.1 8B and other models that fit 12 GB VRAM.
• 3. Set up LiteLLM as a proxy to route requests to the cheapest capable model per job type.
• 4. Configure explicit failover chains for each job type.
• 5. Use OpenWebUI to provide a console that automatically switches to cheaper models when the flat Claude plan is exhausted.
• 6. Implement prompt caching and compressed context files to reduce token usage.
• 7. Schedule heavy jobs off‑peak and wake the GPU on demand to save power.
• 8. Iterate on hardening, optimizing, and reducing consumption.
• 9. Monitor performance and costs, and plan hardware upgrades (e.g., 24 GB RTX 3090) when needed.

Доказательства

The 32‑billion‑parameter model — the impressive one on every capabilities chart — runs at 3.9 tokens per second. Roughly one word every two seconds.

Источник

A flat subscription for interactive work. The heavy agent sessions run on a $100/month flat Claude plan.

Источник

The whole upgrade is a response to one measured number.

Источник