We Kept Rebuilding Aiden for Six Months: From a HID Demo to a Physical AI Agent

Описание и применение

Aiden прошёл путь от простого демо к надёжному физическому ИИ‑агенту, реализовав единую архитектуру с управлением ресурсами, явными действиями и централизованной конфигурацией, что позволило стабильно управлять устройствами через USB‑HID.

Задача

Building a robust physical AI agent that can reliably control phones and computers via hardware interfaces, handling failures like USB reconnects, stale screenshots, context limits, interrupted voice, OS differences, and occasional human intervention.

Результат

Aiden evolved from a fragile demo to a production‑ready physical AI agent with robust hardware abstraction, explicit action semantics, unified configuration, and comprehensive logging, enabling reliable control across diverse devices and failure scenarios.

Рабочий процесс

• 1. Capture target display via HDMI bridge to CSI and /dev/video0 using frame_service.
• 2. Send screenshot to Go Agent, which forwards to configured multimodal model endpoint.
• 3. Receive structured tool calls from model.
• 4. Translate tool calls into HID reports via /dev/hidg0‑2 and send to target device.
• 5. For voice, capture audio via audio_service, perform VAD/STT, send to LLM, receive TTS, and play back.
• 6. After each action, capture post‑action screenshot and markers for model reasoning.
• 7. Use configuration control plane to manage model, STT, TTS providers and credentials.
• 8. Log all actions, states, and observations for audit and debugging.

Доказательства

The current system has several distinct layers: Hardware abstraction, Hardware services, Cross‑language transport, Go Agent, USB gadget layer, Firmware integration, Evaluation.

Источник

We moved that responsibility into frame_service. frame_service is the single owner of the video device.

Источник

The configuration control plane includes JSON Merge Patch updates, atomic TOML writes, and preservation of comments and credentials.

Источник