My RTX 3060 Can Run Many LLMs — Just Not at the Same Time

Description and use

On an RTX 3060 with 12 GB VRAM, multiple large models can run but not all at once. GGUF Switchboard automatically switches models, manages GPU memory, and exposes a single OpenAI‑compatible API.

Problem

A 12 GB GPU can run multiple 7–12 B LLMs, but cannot keep all of them loaded simultaneously, making it hard to serve different specialized models (coding, reasoning, tool‑calling, embedding, reranking) on a single machine.

Outcome

The scheduler allows a single 12 GB GPU to serve multiple large models on demand, reducing manual management and cold‑load latency to a few seconds per switch.

Workflow

• Receive request for a model
• Check if requested model is already loaded
• If loaded, serve request
• If not loaded, drain current model (stop new requests, wait for in‑flight to finish)
• Unload current model and free VRAM
• Load requested model (GGUF or SafeTensors via llama.cpp or vLLM)
• Serve request

Evidence

On my 3060 a swap between 7–12B GGUF models costs a few seconds of cold-load latency, so the scheduler batches consecutive requests for the same model and only pays that cost on an actual switch.

Source

GGUF Switchboard is an open-source model scheduler that provides one OpenAI-compatible endpoint while managing local model lifecycles underneath it.

Source