I tried to fire my LLM API. Open models passed one task and failed the other.

Description and use

Open‑Switch evaluates whether an open-weight model can replace a closed LLM API locally, offering statistically‑grounded verdicts and LoRA cost analysis.

Problem

Determine if an open-weight model can safely replace a closed LLM API for a specific business workflow.

Outcome

Open‑Switch confirmed Qwen 2.5 7B + LoRA as a safe switch for support triage (90.0% [80, 98]) and identified Gemma 2 2B as unsafe for banking intent (41.2% [27.5, 54.9]).

Workflow

• Clone the Open‑Switch repo
• Install dependencies
• Run Open‑Switch with desired models and task file
• Review the Safe Switch or No Safe Switch verdict and LoRA break‑even analysis

Evidence

Open‑Switch answers one question: can an open model replace your closed LLM API on YOUR data?

Source

It runs your workflow through open-weight models, scores them with bootstrap 95% CIs, and gives a verdict: Safe Switch or No Safe Switch.

Source

The tool says "no" when the answer is no. That's the point.

Source

Support triage – Qwen 2.5 7B + LoRA – 90.0% [80, 98] – ✅ Safe switch, 85% cheaper

Source

Banking intent – Gemma 2 2B – 41.2% [27.5, 54.9] – ❌ Not safe

Source