How I benchmarked on-device speech-to-text on Apple silicon (Whisper vs Parakeet)

Description and use

The author benchmarked Whisper and Parakeet on a Mac, finding Parakeet faster (0.10 s vs 0.68 s) with similar WER (2.50 % vs 2.89 %).

Problem

Choosing a default on-device dictation engine for a Mac app required objective measurements of accuracy and latency for Whisper large-v3-turbo and Parakeet TDT 0.6B v3.

Outcome

Parakeet achieved 0.10 s average latency and 2.50 % WER; Whisper had 0.68 s latency and 2.89 % WER. Latency difference dominates push‑to‑talk experience; accuracy difference is within normalization noise.

Workflow

• Select 50‑utterance stratified sample from LibriSpeech test‑clean
• Decode FLAC files to 16 kHz mono Float32
• Prepare WhisperKitEngine and ParakeetEngine once
• Time transcribe(samples:prompt:) for each clip
• Normalize reference and hypothesis (lowercase, strip punctuation except apostrophes, collapse whitespace)
• Compute WER via Levenshtein alignment
• Record per‑clip latency and aggregate statistics
• Commit results.json and audio files to repo

Evidence

I used LibriSpeech test‑clean (CC BY 4.0), downloaded from OpenSLR and checked against its published MD5.

Source

The harness is a ~240‑line Swift command‑line target, Sources/BenchmarkCLI/main.swift.

Source

Parakeet TDT 0.6B v3 WER 2.50% (25 / 1,002), Avg latency 0.10 s, Real‑time factor 0.014.

Source

Neither engine failed on any clip.

Source