A 3B model that beats a 7B: failure-driven orchestration on uncontaminated knowledge

Description and use

A 3 B model with a sovereign QA stack achieved 52 % accuracy on post‑cutoff knowledge, surpassing a 7 B zero‑shot model, showing the power of systematic failure diagnosis and deterministic gating.

Problem

Retrieval benchmarks often measure memorized knowledge, making it hard to evaluate true retrieval effectiveness on new facts.

Outcome

The 3B sovereign stack achieved 52.0% accuracy on post-cutoff-150, beating the 7B zero-shot model’s 50.0% and demonstrating that systematic failure diagnosis and calibrated gates can unlock retrieval value for small models.

Workflow

• Build a local Wikipedia index of 7.2M pages and 11.4M redirects (49 GB).
• Train a Qwen2.5-3B-Instruct reader with LoRA using SFT on noisy evidence and DPO.
• Create an orchestrator with deterministic arms: anchor extraction, lead fetch, gated joint reading, bridge hop, full-text search retry, tables arm, semantic lens, live full-text search.
• Apply gate discipline: promote a challenger only if it beats the champion on the same 150 questions with Wilson CIs; otherwise rollback.
• Iteratively add and evaluate arms: fullpool, FTS-retry, entropy-gated routing, SLOTS, FR→EN translation, Wikipedia tables arm, semantic lens.
• Publish rollbacks and promotions with per-question outputs.
• Compare the 3B orchestrated model to a 7B zero-shot reader on the same chain and questions.

Evidence

The 3B + sovereign chain (final, Oct 1) 52.0% [44.1–59.8]

Source

7B zero-shot, same chain 50.0% [42.1–57.9]

Source

The 3B naked (no retrieval) 0.0% [0–2.5]

Source

The 7B naked 0.0% (by construction — see below)

Source

The benchmark is public on Kaggle (post-cutoff-150).

Source