A 26M-parameter tagger beat our fine-tuned small LLMs at grammar correction, and runs in ~4 ms on a phone

Description and use

A 26 M‑parameter tagger outperforms small LLMs for on‑device grammar correction: F0.5 0.799, 2.4 % false positives, 0 % lost protected spans, ~4 ms inference, 26 MB model.

Problem

Deliver on-device grammar correction for a phone keyboard with strict memory, no network, and high precision requirements.

Outcome

The 26 M tagger outperformed small LLMs in accuracy, precision, and speed, while fitting within phone memory constraints and operating fully offline.

Workflow

• 1. Collect 4,356 held‑out evaluation cases across formal, casual, coedit, preserve, and golden sets.
• 2. Train a 26 M‑parameter encoder‑only transformer with 8 layers, 384‑dim hidden, 6 heads, 15,000 edit labels, using a 16k‑byte BPE tokenizer.
• 3. Pretrain with masked LM on FineWeb‑Edu, then fine‑tune on 12.02 M synthetic and real correction pairs.
• 4. Apply inference‑time guards: KEEP bias, protected‑span enforcement, plausibility guard.
• 5. Evaluate on the held‑out suite, compare against LoRA‑fine‑tuned LLM baselines.
• 6. Identify and fix training‑bug where longer training increased false positives.
• 7. Deploy Core ML (iOS) and ONNX (Android) models, ensuring ~4 ms inference and <30 MB size.
• 8. Release weights on Hugging Face under Apache‑2.0.
• 9. Publish results and takeaways.

Evidence

The 26M tagger (v1.5) wins on accuracy (F0.5 0.799 vs 0.663) and changes correct text 2.4% of the time vs 20.1%.

Source

Protected spans. Inside a detected name, number, email or URL, only $KEEP is allowed. Result: 0.0% lost protected spans across the suite, for every tagger version.

Source

On the 2,000 held‑out real‑style messages, v1.5 changes correct text 1.9% of the time (v1.2: 2.8%).

Source

v1.3 (new error types) and v1.4 (a longer‑pretrained encoder) both lost to v1.2. On correct text, false positives went from 3.1% to 7.4% and 9.1%.

Source

Always train with a fixed step count, never a time budget.

Source