Rebuilding a brain-to-text decoder: from 50% to 23.5% word errors
Description and use
Improved a speech BCI decoder from 50 % to 23.5 % word error rate using architectural tweaks, better decoding, and a stronger language model.
Problem
Improve word error rate of a speech BCI decoder trained on motor cortex activity from 50% to 23.5% using offline reanalysis of published neural data.
Outcome
The decoder’s word error rate was reduced from 50.0% to 23.5% (23.0% on a held‑out half) by architectural changes, improved decoding, and a better language model.
Workflow
- 1. Use only area 6v features (threshold crossings and spike‑band power) and z‑score per recording block.
- 2. Shrink the GRU to 3 layers of 512 units and group sentences by length to reduce padding.
- 3. Decode words directly from CTC output with a lexicon and 3‑gram LM, then let a large language model (Claude Sonnet 4.6) pick the best of the top 10 candidates.
- 4. Replace the training‑sentence LM with a Tatoeba English 3‑gram (≈2 M sentences) and restrict the lexicon to the LM’s 6,569 words.
- 5. Evaluate on a split test set (even sentences for dev, odd for eval) and report the held‑out error rate.
Evidence
The model trained on CPUs for about five hours, and the word error rate on 880 test sentences went from 50.0% to 23.5% over four changes
I shrank the GRU from 5 layers and 1,024 units to 3 layers and 512 units (37.8 million parameters) and grouped sentences by length to cut padding.
I decoded words directly from the CTC output with a lexicon and a 3-gram, using torchaudio's flashlight decoder, and gave the LLM the 10 best sequences to choose from by number
With the full CMU dictionary (about 130,000 words) the error rate jumped to around 90%. My LM gave <unk> about 7% of the probability mass, so every rare dictionary word scored as well as an unknown word and won often.
With it, 297 of 880 sentences came out exactly right, and the oracle (best of the 10 candidates, chosen with the answer) is 19.4%
- Record date: 2026-10-03T07:41:00.601Z