We Kept Rebuilding Aiden for Six Months: From a HID Demo to a Physical AI Agent

说明与用途

Aiden 从简单演示发展为稳健的物理 AI 代理，采用分层架构、单一资源所有权、明确动作语义和统一配置，实现在多设备上的可靠控制。

问题

Building a robust physical AI agent that can reliably control phones and computers via hardware interfaces, handling failures like USB reconnects, stale screenshots, context limits, interrupted voice, OS differences, and occasional human intervention.

结果

Aiden evolved from a fragile demo to a production‑ready physical AI agent with robust hardware abstraction, explicit action semantics, unified configuration, and comprehensive logging, enabling reliable control across diverse devices and failure scenarios.

工作流

• 1. Capture target display via HDMI bridge to CSI and /dev/video0 using frame_service.
• 2. Send screenshot to Go Agent, which forwards to configured multimodal model endpoint.
• 3. Receive structured tool calls from model.
• 4. Translate tool calls into HID reports via /dev/hidg0‑2 and send to target device.
• 5. For voice, capture audio via audio_service, perform VAD/STT, send to LLM, receive TTS, and play back.
• 6. After each action, capture post‑action screenshot and markers for model reasoning.
• 7. Use configuration control plane to manage model, STT, TTS providers and credentials.
• 8. Log all actions, states, and observations for audit and debugging.

证据

The current system has several distinct layers: Hardware abstraction, Hardware services, Cross‑language transport, Go Agent, USB gadget layer, Firmware integration, Evaluation.

来源

We moved that responsibility into frame_service. frame_service is the single owner of the video device.

来源

The configuration control plane includes JSON Merge Patch updates, atomic TOML writes, and preservation of comments and credentials.

来源