Build a Real‑Time Voice AI Agent with LiveKit, Pulse STT, and Lightning TTS

Описание и применение

Создание голосового агента с низкой задержкой, использующего LiveKit, Pulse STT и Lightning TTS, где STT, LLM и TTS работают параллельно, уменьшая время отклика.

Задача

Real‑time voice agents suffer from high latency when each stage (STT, LLM, TTS) waits for the previous to finish, making the user experience sluggish.

Результат

Implemented a low‑latency voice agent where STT, LLM, and TTS operate concurrently, reducing end‑to‑end response time and improving user experience.

Рабочий процесс

• 1. Set up LiveKit (cloud or self‑hosted) and obtain credentials.
• 2. Create a Smallest AI account and generate an API key.
• 3. Install Python 3.11, create a virtual environment, and install livekit‑plugins‑smallestai, livekit‑plugins‑openai, livekit‑plugins‑silero, python‑dotenv.
• 4. Store SMALLEST_API_KEY, LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET, and OPENAI_API_KEY in environment variables.
• 5. Write a VoiceAgent class extending livekit.agents.Agent, configuring instructions, prewarm VAD, and entrypoint to load credentials.
• 6. In entrypoint, create an AgentSession with VAD, STT (smallestai.STT), LLM (openai.LLM), and TTS (smallestai.TTS).
• 7. Run the agent via livekit CLI or programmatically, ensuring LiveKit handles media plumbing and streaming.
• 8. Test latency by measuring time from user speech to audible response, adjusting models or parameters as needed.

Доказательства

LiveKit handles the real‑time media and agent plumbing. Your STT and TTS choices still determine how quickly speech becomes usable text and how quickly generated text becomes audible speech.

Источник

The pipeline is concurrent, not sequential

Источник