Build a Real‑Time Voice AI Agent with LiveKit, Pulse STT, and Lightning TTS

Description and use

Build a low‑latency voice agent using LiveKit, Pulse STT, and Lightning TTS, where STT, LLM, and TTS operate concurrently to reduce response time.

Problem

Real‑time voice agents suffer from high latency when each stage (STT, LLM, TTS) waits for the previous to finish, making the user experience sluggish.

Outcome

Implemented a low‑latency voice agent where STT, LLM, and TTS operate concurrently, reducing end‑to‑end response time and improving user experience.

Workflow

• 1. Set up LiveKit (cloud or self‑hosted) and obtain credentials.
• 2. Create a Smallest AI account and generate an API key.
• 3. Install Python 3.11, create a virtual environment, and install livekit‑plugins‑smallestai, livekit‑plugins‑openai, livekit‑plugins‑silero, python‑dotenv.
• 4. Store SMALLEST_API_KEY, LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET, and OPENAI_API_KEY in environment variables.
• 5. Write a VoiceAgent class extending livekit.agents.Agent, configuring instructions, prewarm VAD, and entrypoint to load credentials.
• 6. In entrypoint, create an AgentSession with VAD, STT (smallestai.STT), LLM (openai.LLM), and TTS (smallestai.TTS).
• 7. Run the agent via livekit CLI or programmatically, ensuring LiveKit handles media plumbing and streaming.
• 8. Test latency by measuring time from user speech to audible response, adjusting models or parameters as needed.

Evidence

LiveKit handles the real‑time media and agent plumbing. Your STT and TTS choices still determine how quickly speech becomes usable text and how quickly generated text becomes audible speech.

Source

The pipeline is concurrent, not sequential

Source