Running Three Specialized Small Models for Self-Hosted AI

Description and use

The author stopped chasing a single perfect large model and instead runs three small, specialized models locally. This approach improved efficiency, privacy, and accuracy in specific tasks (chat, coding, math) while reducing time spent on model selection.

Problem

The user was wasting time chasing a single 'perfect' large model, experiencing 'model-hop weekends' and frustration with generalist models making confident errors in specialized tasks like coding and math.

Outcome

The user achieved a more efficient and private AI workflow by splitting tasks among three small, specialized models. This eliminated the need to chase large models, reduced hardware requirements, and improved accuracy in specific domains like coding and math.

Workflow

• Identify specific use cases: general chat, coding assistance, and mathematical reasoning.
• Select small, quantized models optimized for each specific task.
• Deploy the chat model in Open WebUI for daily general use and family access.
• Configure the coding model for tasks requiring careful import handling and minimal refactoring.
• Use the math model for numerical tasks where predictability is preferred over generalist confidence.
• Monitor resource usage (disk and power) which is minimal for small models.
• Evaluate success based on improvement over previous week's workflow rather than comparison to cloud frontier models.

Evidence

I run three. Not more. Each is quantized small enough to sit on hardware I already own, and each one is bad at the thing the other two are good at.

Source

The chat one is the workhorse... The coding one is picky... And the math one is the one I almost never admit to keeping.

Source

What I got back was my attention. No more model-hop weekends...

Source