I Built a Self-Hosted ChatGPT on My Laptop. Here’s What Broke.

Description and use

OzAI is a local LLM platform running on a laptop, using Ollama, LiteLLM, and Open WebUI. Performance, budget, and accuracy issues were identified and mitigated by aliasing, budget logic changes, and UI reconfiguration.

Problem

Running an open‑weight LLM on a laptop required a platform that could serve the model safely, but several design choices caused performance, cost, and correctness issues.

Outcome

OzAI runs locally on a laptop, providing an OpenAI‑compatible API with per‑user keys and rate limits. Key operational issues were identified—slow reasoning, budget mis‑interpretation, UI background requests, flat throughput, and hallucinations—and mitigated with aliasing, budget logic changes, UI reconfiguration, and planned RAG.

Workflow

• 1. Measure laptop inference speed for Qwen3 8B and Qwen2.5‑Coder 3B.
• 2. Run Ollama natively on macOS to use the GPU via Metal.
• 3. Deploy LiteLLM in Docker Compose as an OpenAI‑compatible gateway.
• 4. Configure model aliases (ozai, ozai‑think, ozai‑coder) and generate virtual keys via LiteLLM.
• 5. Test gateway limits and error handling with curl.
• 6. Identify performance and correctness issues: long reasoning, budget checks, UI background requests, throughput limits, and model hallucinations.
• 7. Apply fixes: disable reasoning for default alias, adjust budget logic, route UI background tasks to a smaller model, and plan RAG integration.
• 8. Measure throughput and TTFT under load.
• 9. Document lessons and plan next steps (observability, load testing, GPU scaling).

Evidence

Ollama runs the models. It runs natively on macOS, not in Docker (more on that below).

Source

The 8B model is less than half as fast as the 3B one.

Source

Budget has been exceeded! Current cost: 0.0, Max budget: 0.0

Source

After every answer, Open WebUI sends extra requests to the same model: one to generate a chat title, one for tags, one for follow‑up suggestions.

Source