A 26M-parameter tagger beat our fine-tuned small LLMs at grammar correction, and runs in ~4 ms on a phone

说明与用途

26 M参数标记器在手机端语法纠错中优于小型LLM：F0.5 0.799，错误修正率2.4%，无保护片段丢失，约4 ms推理，模型大小26 MB。

问题

Deliver on-device grammar correction for a phone keyboard with strict memory, no network, and high precision requirements.

结果

The 26 M tagger outperformed small LLMs in accuracy, precision, and speed, while fitting within phone memory constraints and operating fully offline.

工作流

• 1. Collect 4,356 held‑out evaluation cases across formal, casual, coedit, preserve, and golden sets.
• 2. Train a 26 M‑parameter encoder‑only transformer with 8 layers, 384‑dim hidden, 6 heads, 15,000 edit labels, using a 16k‑byte BPE tokenizer.
• 3. Pretrain with masked LM on FineWeb‑Edu, then fine‑tune on 12.02 M synthetic and real correction pairs.
• 4. Apply inference‑time guards: KEEP bias, protected‑span enforcement, plausibility guard.
• 5. Evaluate on the held‑out suite, compare against LoRA‑fine‑tuned LLM baselines.
• 6. Identify and fix training‑bug where longer training increased false positives.
• 7. Deploy Core ML (iOS) and ONNX (Android) models, ensuring ~4 ms inference and <30 MB size.
• 8. Release weights on Hugging Face under Apache‑2.0.
• 9. Publish results and takeaways.

证据

The 26M tagger (v1.5) wins on accuracy (F0.5 0.799 vs 0.663) and changes correct text 2.4% of the time vs 20.1%.

来源

Protected spans. Inside a detected name, number, email or URL, only $KEEP is allowed. Result: 0.0% lost protected spans across the suite, for every tagger version.

来源

On the 2,000 held‑out real‑style messages, v1.5 changes correct text 1.9% of the time (v1.2: 2.8%).

来源

v1.3 (new error types) and v1.4 (a longer‑pretrained encoder) both lost to v1.2. On correct text, false positives went from 3.1% to 7.4% and 9.1%.

来源

Always train with a fixed step count, never a time budget.

来源