We Tried to Make a Voice Agent Control a Real Device. Here’s What Broke

Description and use

Separating voice interaction from device control enabled the agent to respond quickly to interruptions, accurately track task completion, and handle human handoffs.

Problem

Integrating real‑time voice interaction with long‑running device control tasks caused interference, delayed responses, and unreliable task outcomes.

Outcome

The redesigned architecture allowed the agent to remain responsive to interruptions, accurately report task outcomes, and handle human handoffs without confusing the user.

Workflow

• 1. User speaks a command.
• 2. Foreground agent captures audio and starts a short conversational response.
• 3. Backend agent receives task, performs screen inspection, tool calls, and device actions.
• 4. Backend sends task updates to a notification queue.
• 5. Queue debounces events (500 ms) before forwarding to foreground agent.
• 6. Foreground agent delivers combined update, allowing user to interrupt or modify the task.
• 7. Task status (Created, Queued, Running, Completed, Failed) and result are stored separately.
• 8. If human action is required, backend signals pending user action and pauses execution until resolved.

Evidence

The solution was to split the responsibilities: The foreground Realtime Agent handles listening, short conversational responses, and interruption. The backend Agent handles screen understanding, tool calls, and device actions.

Source

We added a notification queue with a short aggregation window. When several task results arrive close together, the runtime waits 500 milliseconds, extending the window if another result arrives, and then delivers the combined update.

Source