We Tried to Make a Voice Agent Control a Real Device. Here’s What Broke

说明与用途

将语音交互与设备控制分离，使代理能够快速响应中断、准确跟踪任务完成并处理人工干预。

问题

Integrating real‑time voice interaction with long‑running device control tasks caused interference, delayed responses, and unreliable task outcomes.

结果

The redesigned architecture allowed the agent to remain responsive to interruptions, accurately report task outcomes, and handle human handoffs without confusing the user.

工作流

• 1. User speaks a command.
• 2. Foreground agent captures audio and starts a short conversational response.
• 3. Backend agent receives task, performs screen inspection, tool calls, and device actions.
• 4. Backend sends task updates to a notification queue.
• 5. Queue debounces events (500 ms) before forwarding to foreground agent.
• 6. Foreground agent delivers combined update, allowing user to interrupt or modify the task.
• 7. Task status (Created, Queued, Running, Completed, Failed) and result are stored separately.
• 8. If human action is required, backend signals pending user action and pauses execution until resolved.

证据

The solution was to split the responsibilities: The foreground Realtime Agent handles listening, short conversational responses, and interruption. The backend Agent handles screen understanding, tool calls, and device actions.

来源

We added a notification queue with a short aggregation window. When several task results arrive close together, the runtime waits 500 milliseconds, extending the window if another result arrives, and then delivers the combined update.

来源