A 3B model that beats a 7B: failure-driven orchestration on uncontaminated knowledge

Описание и применение

Модель 3 Б с суверенным стеком QA достигла 52 % точности на новых фактах, превзойдя 7 Б без обучения, демонстрируя эффективность систематического анализа ошибок и детерминированных шлюзов.

Задача

Retrieval benchmarks often measure memorized knowledge, making it hard to evaluate true retrieval effectiveness on new facts.

Результат

The 3B sovereign stack achieved 52.0% accuracy on post-cutoff-150, beating the 7B zero-shot model’s 50.0% and demonstrating that systematic failure diagnosis and calibrated gates can unlock retrieval value for small models.

Рабочий процесс

• Build a local Wikipedia index of 7.2M pages and 11.4M redirects (49 GB).
• Train a Qwen2.5-3B-Instruct reader with LoRA using SFT on noisy evidence and DPO.
• Create an orchestrator with deterministic arms: anchor extraction, lead fetch, gated joint reading, bridge hop, full-text search retry, tables arm, semantic lens, live full-text search.
• Apply gate discipline: promote a challenger only if it beats the champion on the same 150 questions with Wilson CIs; otherwise rollback.
• Iteratively add and evaluate arms: fullpool, FTS-retry, entropy-gated routing, SLOTS, FR→EN translation, Wikipedia tables arm, semantic lens.
• Publish rollbacks and promotions with per-question outputs.
• Compare the 3B orchestrated model to a 7B zero-shot reader on the same chain and questions.

Доказательства

The 3B + sovereign chain (final, Oct 1) 52.0% [44.1–59.8]

Источник

7B zero-shot, same chain 50.0% [42.1–57.9]

Источник

The 3B naked (no retrieval) 0.0% [0–2.5]

Источник

The 7B naked 0.0% (by construction — see below)

Источник

The benchmark is public on Kaggle (post-cutoff-150).

Источник