My speech-flaw detector flagged 41 false alarms a minute. One line of math fixed it.

Description and use

Podium’s speech‑flaw detector initially flagged 41 false alarms per minute on a clean recording from a different speaker. By separating style from flaws and adding a voiced‑gap feature, false alarms dropped to 3 per minute and F1 improved to 0.51.

Problem

A speech‑flaw detector flagged 41 false alarms per minute on a clean recording from a different speaker.

Outcome

False alarms on clean different‑speaker recordings reduced from 41/min to 3/min; F1 on held‑out set reached 0.51 with mean IoU 0.81.

Workflow

• 1. Force‑align both recordings to the same transcript using torchaudio’s MMS_FA.
• 2. Compute speaker‑normalised features (pitch, loudness, articulation rate, pause, voiced‑gap, high‑frequency energy).
• 3. Build a labelled dataset by injecting synthetic flaws into recordings and preserving exact timestamps.
• 4. Score each word with two robust z‑scores: deviation from reference (z_ref) and deviation from the user’s own style (z_self).
• 5. Define flaw as the soft AND of z_ref and z_self.
• 6. Add a voiced‑gap feature to distinguish stutters from pauses.
• 7. Tune thresholds on JFK, freeze, and test on Reagan and an unseen voice.

Evidence

Version 1 scored each word by how far it departed from the reference, as a robust z‑score (median/MAD) after removing your overall offset.

Source

False alarms on clean different‑speaker recordings went from 41/min to 3/min.

Source

I built Podium, a tool that compares your reading of a speech with a great delivery of the same text (JFK, Reagan) and tells you exactly where you drift and why.

Source

I also had the same texts read by open Piper TTS voices, with flaws injected into those too. That's the "different speaker" test.

Source