AstaBrief 8B: How AllenAI Trained a Small Open Model to Generate Cited Scientific Reports

Description and use

AstaBrief 8B is an open‑weights model that produces cited scientific reports in 51 s, using SFT followed by DPO and a key citation‑density filter.

Problem

Generate fully cited scientific reports quickly while ensuring high citation quality.

Outcome

AstaBrief 8B achieves 87/200 on SQABench‑CS2, outperforms SFT‑only and base Qwen3‑8B, and wins 55%/72% in pairwise LLM comparisons against the Claude‑powered pipeline. Human study shows 2/3 researchers prefer it for citation accuracy.

Workflow

• Collect research question and retrieved literature excerpts via ScholarQA
• Filter queries for quality, relevance, and privacy to build a 90k pool
• Generate full‑report targets with Claude 3.5/3.7 Sonnet, o3, o4‑mini, GPT‑4.1
• Filter targets for quality, reducing to 47k usable examples
• Fine‑tune Qwen3‑8B with supervised fine‑tuning (SFT) on the 47k examples
• Create preference pairs from a separate query subset, compare reports from ScholarQA pipeline vs. o3, o4‑mini, DeepSeek‑V3, DeepSeek‑R1
• Have GPT‑4.1 and DeepSeek‑R1 judges select winners; keep only pairs where both agree (~6k examples)
• Fine‑tune with Direct Preference Optimization (DPO) on the 6k pairs using 8×H100 GPUs
• Deploy the resulting model as Fast mode in Asta
• Users generate reports in a single forward pass

Evidence

AstaBrief replaces that pipeline for users who want speed. Given the same research question and retrieved excerpts, it writes the full report in one pass, averaging 51.1 seconds — roughly 3.5 times faster.

Source

The strongest quality gains came from filtering out reports with low citation density. Reports with large stretches of unsupported text — even if they were otherwise well‑written — degraded the model's ability to produce grounded output.

Source

Ai2 reports that the judges agreed with human preferences 95% of the time.

Source

On the ScholarQA-CS2 test set, AstaBrief 8B averaged 87 across tracked metrics, compared with 83.7 for the SFT-only checkpoint and 77.3 for base Qwen3-8B.

Source