How I built hands‑free voice mode for my AI assistant: ffmpeg VAD, barge‑in, zero npm packages

Description and use

A lightweight, dependency‑free voice interface that leverages ffmpeg’s silencedetect for VAD, Whisper for STT, and a custom WebSocket handshake for TTS, enabling hands‑free interaction and barge‑in on any machine with ffmpeg.

Problem

Implement a hands‑free voice interface for an AI assistant without relying on heavy speech SDKs or npm dependencies.

Outcome

A lightweight, dependency‑free voice interface that leverages ffmpeg’s silencedetect for VAD, Whisper for STT, and a custom WebSocket handshake for TTS, enabling hands‑free interaction and barge‑in on any machine with ffmpeg.

Workflow

• 1. Start a single ffmpeg process to record mic audio and stream silence detection via stderr using the silencedetect filter.
• 2. Parse ffmpeg stderr lines with createSilenceParser to emit silence_start and silence_end events.
• 3. Feed events into createTurnDetector to determine speech and end states for a 4‑state turn machine.
• 4. When speech is detected, send the recorded PCM audio to Groq’s Whisper API (whisper-large-v3-turbo) for STT.
• 5. For TTS, hand‑craft a WebSocket handshake with Node’s tls module to Microsoft Edge’s free endpoint, or use Orpheus TTS if a GROQ_API_KEY is present.
• 6. Play the synthesized WAV via ffplay while keeping the mic open for barge‑in detection.
• 7. If speech starts >400 ms after playback begins, kill ffplay, increment playbackGeneration to avoid race, and transcribe the new utterance as the next turn.
• 8. Handle Bluetooth headsets by routing playback to the Hands‑Free endpoint when the mic is Bluetooth to avoid silent barge‑in.
• 9. Summarize long tool outputs into a short spoken note and provide a link for full details, with optional full read via VOICE_FULL_READ.
• 10. Expose tunable VAD parameters (VOICE_NOISE_DB, VOICE_SILENCE_MS, VOICE_BARGE_DB, VOICE_MAX_UTTERANCE_MS) for customization.
• 11. Run 30 unit tests on parsers and turn detector without touching a microphone.

Evidence

A voice interface sounds like it needs a speech SDK, a streaming framework, and a small pile of native dependencies. Mine is one 1,100-line JavaScript module, zero npm dependencies, and it runs on binaries you probably already have: ffmpeg and ffplay.

Source

The trick I like most: I never wrote a VAD (voice activity detection) library. ffmpeg ships a silencedetect audio filter, so a single ffmpeg process records the mic and streams speech boundaries on its stderr.

Source

STT goes to Groq's Whisper API (whisper-large-v3-turbo). One fetch call with multipart audio. It needs a GROQ_API_KEY, and that's the only key in the whole voice path.

Source

TTS defaults to Microsoft Edge's free endpoint — and this is the part I find most satisfying. Instead of adding a WebSocket library, I did the handshake over Node's built-in node:tls, generating the Sec-MS-GEC token the endpoint expects and speaking the raw WebSocket protocol by hand.

Source

If speech starts more than ~400ms after playback began, that's a barge: kill the player with SIGKILL, bump a playbackGeneration counter so any audio that's still being written to disk can't play over the new utterance.

Source

Bluetooth headsets drop their high-quality A2DP audio endpoint the moment the mic opens — so if playback was routed through A2DP, barge‑in replies came out silent. The fix lives in src/channels/audio-device.mjs.

Source