My RTX 3060 Can Run Many LLMs — Just Not at the Same Time

说明与用途

在 RTX 3060 12 GB 显存上，可以运行多种大型模型，但无法同时加载。GGUF Switchboard 自动切换模型，管理 GPU 内存，并提供单一 OpenAI 兼容 API。

问题

A 12 GB GPU can run multiple 7–12 B LLMs, but cannot keep all of them loaded simultaneously, making it hard to serve different specialized models (coding, reasoning, tool‑calling, embedding, reranking) on a single machine.

结果

The scheduler allows a single 12 GB GPU to serve multiple large models on demand, reducing manual management and cold‑load latency to a few seconds per switch.

工作流

• Receive request for a model
• Check if requested model is already loaded
• If loaded, serve request
• If not loaded, drain current model (stop new requests, wait for in‑flight to finish)
• Unload current model and free VRAM
• Load requested model (GGUF or SafeTensors via llama.cpp or vLLM)
• Serve request

证据

On my 3060 a swap between 7–12B GGUF models costs a few seconds of cold-load latency, so the scheduler batches consecutive requests for the same model and only pays that cost on an actual switch.

来源

GGUF Switchboard is an open-source model scheduler that provides one OpenAI-compatible endpoint while managing local model lifecycles underneath it.

来源