Running Three Specialized Small Models for Self-Hosted AI

Описание и применение

Автор отказался от поиска одной идеальной большой модели в пользу трех небольших специализированных моделей, запущенных локально. Это решение повысило эффективность, приватность и точность в конкретных задачах (чат, код, математика), снизив затраты времени на настройку.

Задача

The user was wasting time chasing a single 'perfect' large model, experiencing 'model-hop weekends' and frustration with generalist models making confident errors in specialized tasks like coding and math.

Результат

The user achieved a more efficient and private AI workflow by splitting tasks among three small, specialized models. This eliminated the need to chase large models, reduced hardware requirements, and improved accuracy in specific domains like coding and math.

Рабочий процесс

• Identify specific use cases: general chat, coding assistance, and mathematical reasoning.
• Select small, quantized models optimized for each specific task.
• Deploy the chat model in Open WebUI for daily general use and family access.
• Configure the coding model for tasks requiring careful import handling and minimal refactoring.
• Use the math model for numerical tasks where predictability is preferred over generalist confidence.
• Monitor resource usage (disk and power) which is minimal for small models.
• Evaluate success based on improvement over previous week's workflow rather than comparison to cloud frontier models.

Доказательства

I run three. Not more. Each is quantized small enough to sit on hardware I already own, and each one is bad at the thing the other two are good at.

Источник

The chat one is the workhorse... The coding one is picky... And the math one is the one I almost never admit to keeping.

Источник

What I got back was my attention. No more model-hop weekends...

Источник