I Built a Self-Hosted ChatGPT on My Laptop. Here’s What Broke.

说明与用途

OzAI 是在笔记本电脑上运行的本地 LLM 平台，使用 Ollama、LiteLLM 和 Open WebUI。通过别名、预算逻辑调整和 UI 重定向解决了性能、预算和准确性问题。

问题

Running an open‑weight LLM on a laptop required a platform that could serve the model safely, but several design choices caused performance, cost, and correctness issues.

结果

OzAI runs locally on a laptop, providing an OpenAI‑compatible API with per‑user keys and rate limits. Key operational issues were identified—slow reasoning, budget mis‑interpretation, UI background requests, flat throughput, and hallucinations—and mitigated with aliasing, budget logic changes, UI reconfiguration, and planned RAG.

工作流

• 1. Measure laptop inference speed for Qwen3 8B and Qwen2.5‑Coder 3B.
• 2. Run Ollama natively on macOS to use the GPU via Metal.
• 3. Deploy LiteLLM in Docker Compose as an OpenAI‑compatible gateway.
• 4. Configure model aliases (ozai, ozai‑think, ozai‑coder) and generate virtual keys via LiteLLM.
• 5. Test gateway limits and error handling with curl.
• 6. Identify performance and correctness issues: long reasoning, budget checks, UI background requests, throughput limits, and model hallucinations.
• 7. Apply fixes: disable reasoning for default alias, adjust budget logic, route UI background tasks to a smaller model, and plan RAG integration.
• 8. Measure throughput and TTFT under load.
• 9. Document lessons and plan next steps (observability, load testing, GPU scaling).

证据

Ollama runs the models. It runs natively on macOS, not in Docker (more on that below).

来源

The 8B model is less than half as fast as the 3B one.

来源

Budget has been exceeded! Current cost: 0.0, Max budget: 0.0

来源

After every answer, Open WebUI sends extra requests to the same model: one to generate a chat title, one for tags, one for follow‑up suggestions.

来源