How I benchmarked on-device speech-to-text on Apple silicon (Whisper vs Parakeet)

Описание и применение

Автор сравнил два движка распознавания речи для Mac‑приложения Plainsay, используя 50‑пробный набор LibriSpeech. Parakeet показал более быструю латентность (0,10 с против 0,68 с) и схожую точность (2,50 % против 2,89 %).

Задача

Choosing a default on-device dictation engine for a Mac app required objective measurements of accuracy and latency for Whisper large-v3-turbo and Parakeet TDT 0.6B v3.

Результат

Parakeet achieved 0.10 s average latency and 2.50 % WER; Whisper had 0.68 s latency and 2.89 % WER. Latency difference dominates push‑to‑talk experience; accuracy difference is within normalization noise.

Рабочий процесс

• Select 50‑utterance stratified sample from LibriSpeech test‑clean
• Decode FLAC files to 16 kHz mono Float32
• Prepare WhisperKitEngine and ParakeetEngine once
• Time transcribe(samples:prompt:) for each clip
• Normalize reference and hypothesis (lowercase, strip punctuation except apostrophes, collapse whitespace)
• Compute WER via Levenshtein alignment
• Record per‑clip latency and aggregate statistics
• Commit results.json and audio files to repo

Доказательства

I used LibriSpeech test‑clean (CC BY 4.0), downloaded from OpenSLR and checked against its published MD5.

Источник

The harness is a ~240‑line Swift command‑line target, Sources/BenchmarkCLI/main.swift.

Источник

Parakeet TDT 0.6B v3 WER 2.50% (25 / 1,002), Avg latency 0.10 s, Real‑time factor 0.014.

Источник

Neither engine failed on any clip.

Источник