A 3B model that beats a 7B: failure-driven orchestration on uncontaminated knowledge

说明与用途

使用主权QA堆栈的3B模型在新知识上实现了52%的准确率，超过了7B零样本模型，展示了系统化错误诊断和确定性门控的有效性。

问题

Retrieval benchmarks often measure memorized knowledge, making it hard to evaluate true retrieval effectiveness on new facts.

结果

The 3B sovereign stack achieved 52.0% accuracy on post-cutoff-150, beating the 7B zero-shot model’s 50.0% and demonstrating that systematic failure diagnosis and calibrated gates can unlock retrieval value for small models.

工作流

• Build a local Wikipedia index of 7.2M pages and 11.4M redirects (49 GB).
• Train a Qwen2.5-3B-Instruct reader with LoRA using SFT on noisy evidence and DPO.
• Create an orchestrator with deterministic arms: anchor extraction, lead fetch, gated joint reading, bridge hop, full-text search retry, tables arm, semantic lens, live full-text search.
• Apply gate discipline: promote a challenger only if it beats the champion on the same 150 questions with Wilson CIs; otherwise rollback.
• Iteratively add and evaluate arms: fullpool, FTS-retry, entropy-gated routing, SLOTS, FR→EN translation, Wikipedia tables arm, semantic lens.
• Publish rollbacks and promotions with per-question outputs.
• Compare the 3B orchestrated model to a 7B zero-shot reader on the same chain and questions.

证据

The 3B + sovereign chain (final, Oct 1) 52.0% [44.1–59.8]

来源

7B zero-shot, same chain 50.0% [42.1–57.9]

来源

The 3B naked (no retrieval) 0.0% [0–2.5]

来源

The 7B naked 0.0% (by construction — see below)

来源

The benchmark is public on Kaggle (post-cutoff-150).

来源