I Cut Voice AI Latency From 4.2s to 780ms with Deepgram + ElevenLabs
Description and use
<p>The first version of my voice agent took 4.2 seconds to answer a question. Not "felt slow." Measured: 4,247ms median from the last syllable a human spoke to the first syllable that came back. Voice AI latency is the one metric where users don't need your dashboard to notice the regression. They just say "hello?" into the silence and then start repeating themselves, which produces a second transcript, which the bot also answers.</p> <p>I spent about three weeks taking that number apart. Here's the honest waterfall, what each fix actually bought in milliseconds, and the three things that brok
Problem
<p>The first version of my voice agent took 4.2 seconds to answer a question. Not "felt slow." Measured: 4,247ms median from the last syllable a human spoke to the first syllable that came back. Voice AI latency is the one metric where users don't need your dashboard to notice the regression. They just say "hello?" into the silence and then start repeating themselves, which produces a second transcript, which the bot also answers.</p> <p>I spent about three weeks taking that number apart. Here's the honest waterfall, what each fix actually bought in milliseconds, and the three things that brok
Outcome
<p>The first version of my voice agent took 4.2 seconds to answer a question. Not "felt slow." Measured: 4,247ms median from the last syllable a human spoke to the first syllable that came back. Voice AI latency is the one metric where users don't need your dashboard to notice the regression. They just say "hello?" into the silence and then start repeating themselves, which produces a second transcript, which the bot also answers.</p> <p>I spent about three weeks taking that number apart. Here's the honest waterfall, what each fix actually bought in milliseconds, and the three things that brok
Evidence
I Cut Voice AI Latency From 4.2s to 780ms with Deepgram + ElevenLabs
- Record date: 2026-09-11T05:30:21.865Z