What It Actually Takes to Run Your Own AI: One Consumer GPU, a Routed Model Mesh, and $100 a Month

Description and use

A sovereign AI stack on a single gaming GPU routes requests to cheap models and uses a $100/month Claude plan, delivering daily tasks for under $10/month in API costs.

Problem

Many local‑vs‑cloud AI stories overestimate hardware needs and cost, leading to unrealistic expectations for solo operators.

Outcome

The system delivers daily AI tasks—document classification, RAG, transcription, email triage, code drafting—on a single gaming GPU, with a routed mesh of cheap models and a modest $100/month subscription, achieving sub‑$10/month API costs and minimal power consumption.

Workflow

• 1. Deploy a bare‑metal host with an RTX 4070 Ti and Docker containers for other services.
• 2. Run Ollama locally with Llama 3.1 8B and other models that fit 12 GB VRAM.
• 3. Set up LiteLLM as a proxy to route requests to the cheapest capable model per job type.
• 4. Configure explicit failover chains for each job type.
• 5. Use OpenWebUI to provide a console that automatically switches to cheaper models when the flat Claude plan is exhausted.
• 6. Implement prompt caching and compressed context files to reduce token usage.
• 7. Schedule heavy jobs off‑peak and wake the GPU on demand to save power.
• 8. Iterate on hardening, optimizing, and reducing consumption.
• 9. Monitor performance and costs, and plan hardware upgrades (e.g., 24 GB RTX 3090) when needed.

Evidence

The 32‑billion‑parameter model — the impressive one on every capabilities chart — runs at 3.9 tokens per second. Roughly one word every two seconds.

Source

A flat subscription for interactive work. The heavy agent sessions run on a $100/month flat Claude plan.

Source

The whole upgrade is a response to one measured number.

Source