We Tried to Make a Voice Agent Control a Real Device. Here’s What Broke

Описание и применение

Разделение голосового взаимодействия и управления устройством позволило агенту быстро реагировать на прерывания, точно отслеживать завершение задач и обрабатывать вмешательство человека.

Задача

Integrating real‑time voice interaction with long‑running device control tasks caused interference, delayed responses, and unreliable task outcomes.

Результат

The redesigned architecture allowed the agent to remain responsive to interruptions, accurately report task outcomes, and handle human handoffs without confusing the user.

Рабочий процесс

• 1. User speaks a command.
• 2. Foreground agent captures audio and starts a short conversational response.
• 3. Backend agent receives task, performs screen inspection, tool calls, and device actions.
• 4. Backend sends task updates to a notification queue.
• 5. Queue debounces events (500 ms) before forwarding to foreground agent.
• 6. Foreground agent delivers combined update, allowing user to interrupt or modify the task.
• 7. Task status (Created, Queued, Running, Completed, Failed) and result are stored separately.
• 8. If human action is required, backend signals pending user action and pauses execution until resolved.

Доказательства

The solution was to split the responsibilities: The foreground Realtime Agent handles listening, short conversational responses, and interruption. The backend Agent handles screen understanding, tool calls, and device actions.

Источник

We added a notification queue with a short aggregation window. When several task results arrive close together, the runtime waits 500 milliseconds, extending the window if another result arrives, and then delivers the combined update.

Источник