We Kept Rebuilding Aiden for Six Months: From a HID Demo to a Physical AI Agent

Description and use

Aiden evolved from a simple demo to a robust physical AI agent with a layered architecture, single resource ownership, explicit action semantics, and unified configuration, enabling reliable control across devices.

Problem

Building a robust physical AI agent that can reliably control phones and computers via hardware interfaces, handling failures like USB reconnects, stale screenshots, context limits, interrupted voice, OS differences, and occasional human intervention.

Outcome

Aiden evolved from a fragile demo to a production‑ready physical AI agent with robust hardware abstraction, explicit action semantics, unified configuration, and comprehensive logging, enabling reliable control across diverse devices and failure scenarios.

Workflow

• 1. Capture target display via HDMI bridge to CSI and /dev/video0 using frame_service.
• 2. Send screenshot to Go Agent, which forwards to configured multimodal model endpoint.
• 3. Receive structured tool calls from model.
• 4. Translate tool calls into HID reports via /dev/hidg0‑2 and send to target device.
• 5. For voice, capture audio via audio_service, perform VAD/STT, send to LLM, receive TTS, and play back.
• 6. After each action, capture post‑action screenshot and markers for model reasoning.
• 7. Use configuration control plane to manage model, STT, TTS providers and credentials.
• 8. Log all actions, states, and observations for audit and debugging.

Evidence

The current system has several distinct layers: Hardware abstraction, Hardware services, Cross‑language transport, Go Agent, USB gadget layer, Firmware integration, Evaluation.

Source

We moved that responsibility into frame_service. frame_service is the single owner of the video device.

Source

The configuration control plane includes JSON Merge Patch updates, atomic TOML writes, and preservation of comments and credentials.

Source