How I cut my AI inference bill by 80% - moving from OpenAI to self-hosted models

说明与用途

通过将 GPT‑4o 推理迁移到 Hetzner GPU 服务器上运行 Qwen 2.5‑72B（使用 vLLM 和 LiteLLM），将每月 LLM 推理成本从 $490 降至 $195，节省 $295/月。

问题

High monthly cost for OpenAI GPT‑4o inference ($400‑$490/month) driven mainly by output token usage.

结果

Reduced total monthly cost from $490 to $195, an 80% savings of ~$295/month ($3,540/year).

工作流

• Measure current bill and identify cost drivers
• Pick highest‑volume task (e.g., RAG queries)
• Stand up vLLM locally with Qwen 2.5‑72B
• Add LiteLLM as a drop‑in OpenAI‑compatible gateway
• Move 10% of traffic to self‑hosted and compare outputs
• If quality is acceptable, move 100% of traffic
• Set up monitoring for vLLM crashes

证据

My OpenAI bill was $400/month. After moving to self-hosted models on a single Hetzner GPU box, it's $80/month and falling.

来源

Cost: $1.50/hour for an A100, mostly idle. About $300/month. Same quality as GPT-4o on most of my tasks. Breakthrough.

来源

Hardware: Hetzner GPU server, 1× A100 80GB on-demand. Effective cost ~$190/month (shared between workloads).

来源

Saves: ~$295/month = $3,540/year.

来源