Mobile UI Control for AI Agents

Описание и применение

AI-агенты теперь могут видеть и управлять интерфейсом мобильного телефона, используя локальный шлюз OpenClaw.

Задача

Enabling AI agents to interact with mobile device interfaces by providing visual perception and touch control.

Результат

AI agents can now directly interact with mobile apps, enabling automated UI testing and task execution on real devices.

Рабочий процесс

• 1. Install the OpenClaw Android gateway on the target phone.
• 2. Run the gateway locally to expose a control interface.
• 3. Connect the AI agent to the gateway endpoint.
• 4. Agent captures screen snapshots and sends tap commands to the phone.
• 5. Phone executes the gestures and updates the UI accordingly.

Доказательства

this project gives AI Agents UI control, it lets it see, tap and interact on your phone

Источник

also i ran the gateway locally on my phone

Источник