Mobile UI Control for AI Agents

Description and use

AI agents can now view and control a mobile phone’s UI via a locally run OpenClaw gateway.

Problem

Enabling AI agents to interact with mobile device interfaces by providing visual perception and touch control.

Outcome

AI agents can now directly interact with mobile apps, enabling automated UI testing and task execution on real devices.

Workflow

• 1. Install the OpenClaw Android gateway on the target phone.
• 2. Run the gateway locally to expose a control interface.
• 3. Connect the AI agent to the gateway endpoint.
• 4. Agent captures screen snapshots and sends tap commands to the phone.
• 5. Phone executes the gestures and updates the UI accordingly.

Evidence

this project gives AI Agents UI control, it lets it see, tap and interact on your phone

Source

also i ran the gateway locally on my phone

Source