My RTX 3060 Can Run Many LLMs — Just Not at the Same Time

Описание и применение

На RTX 3060 с 12 ГБ видеопамяти можно запустить несколько больших моделей, но не одновременно. Решение – GGUF Switchboard, который автоматически переключает модели, управляя памятью GPU и предоставляя единый OpenAI‑совместимый API.

Задача

A 12 GB GPU can run multiple 7–12 B LLMs, but cannot keep all of them loaded simultaneously, making it hard to serve different specialized models (coding, reasoning, tool‑calling, embedding, reranking) on a single machine.

Результат

The scheduler allows a single 12 GB GPU to serve multiple large models on demand, reducing manual management and cold‑load latency to a few seconds per switch.

Рабочий процесс

• Receive request for a model
• Check if requested model is already loaded
• If loaded, serve request
• If not loaded, drain current model (stop new requests, wait for in‑flight to finish)
• Unload current model and free VRAM
• Load requested model (GGUF or SafeTensors via llama.cpp or vLLM)
• Serve request

Доказательства

On my 3060 a swap between 7–12B GGUF models costs a few seconds of cold-load latency, so the scheduler batches consecutive requests for the same model and only pays that cost on an actual switch.

Источник

GGUF Switchboard is an open-source model scheduler that provides one OpenAI-compatible endpoint while managing local model lifecycles underneath it.

Источник