AstaBrief 8B: How AllenAI Trained a Small Open Model to Generate Cited Scientific Reports

说明与用途

AstaBrief 8B 是一款开源模型，能在51秒内生成带引用的科学报告，采用SFT+DPO训练并通过过滤低引用密度样本提升质量。

问题

Generate fully cited scientific reports quickly while ensuring high citation quality.

结果

AstaBrief 8B achieves 87/200 on SQABench‑CS2, outperforms SFT‑only and base Qwen3‑8B, and wins 55%/72% in pairwise LLM comparisons against the Claude‑powered pipeline. Human study shows 2/3 researchers prefer it for citation accuracy.

工作流

• Collect research question and retrieved literature excerpts via ScholarQA
• Filter queries for quality, relevance, and privacy to build a 90k pool
• Generate full‑report targets with Claude 3.5/3.7 Sonnet, o3, o4‑mini, GPT‑4.1
• Filter targets for quality, reducing to 47k usable examples
• Fine‑tune Qwen3‑8B with supervised fine‑tuning (SFT) on the 47k examples
• Create preference pairs from a separate query subset, compare reports from ScholarQA pipeline vs. o3, o4‑mini, DeepSeek‑V3, DeepSeek‑R1
• Have GPT‑4.1 and DeepSeek‑R1 judges select winners; keep only pairs where both agree (~6k examples)
• Fine‑tune with Direct Preference Optimization (DPO) on the 6k pairs using 8×H100 GPUs
• Deploy the resulting model as Fast mode in Asta
• Users generate reports in a single forward pass

证据

AstaBrief replaces that pipeline for users who want speed. Given the same research question and retrieved excerpts, it writes the full report in one pass, averaging 51.1 seconds — roughly 3.5 times faster.

来源

The strongest quality gains came from filtering out reports with low citation density. Reports with large stretches of unsupported text — even if they were otherwise well‑written — degraded the model's ability to produce grounded output.

来源

Ai2 reports that the judges agreed with human preferences 95% of the time.

来源

On the ScholarQA-CS2 test set, AstaBrief 8B averaged 87 across tracked metrics, compared with 83.7 for the SFT-only checkpoint and 77.3 for base Qwen3-8B.

来源