What It Actually Takes to Run Your Own AI: One Consumer GPU, a Routed Model Mesh, and $100 a Month

说明与用途

在单个游戏 GPU 上运行的自治 AI 系统，通过路由到低成本模型并使用 $100/月的 Claude 计划，实现每日任务，API 成本低于 $10/月。

问题

Many local‑vs‑cloud AI stories overestimate hardware needs and cost, leading to unrealistic expectations for solo operators.

结果

The system delivers daily AI tasks—document classification, RAG, transcription, email triage, code drafting—on a single gaming GPU, with a routed mesh of cheap models and a modest $100/month subscription, achieving sub‑$10/month API costs and minimal power consumption.

工作流

• 1. Deploy a bare‑metal host with an RTX 4070 Ti and Docker containers for other services.
• 2. Run Ollama locally with Llama 3.1 8B and other models that fit 12 GB VRAM.
• 3. Set up LiteLLM as a proxy to route requests to the cheapest capable model per job type.
• 4. Configure explicit failover chains for each job type.
• 5. Use OpenWebUI to provide a console that automatically switches to cheaper models when the flat Claude plan is exhausted.
• 6. Implement prompt caching and compressed context files to reduce token usage.
• 7. Schedule heavy jobs off‑peak and wake the GPU on demand to save power.
• 8. Iterate on hardening, optimizing, and reducing consumption.
• 9. Monitor performance and costs, and plan hardware upgrades (e.g., 24 GB RTX 3090) when needed.

证据

The 32‑billion‑parameter model — the impressive one on every capabilities chart — runs at 3.9 tokens per second. Roughly one word every two seconds.

来源

A flat subscription for interactive work. The heavy agent sessions run on a $100/month flat Claude plan.

来源

The whole upgrade is a response to one measured number.

来源