I gave Claude Code hands: it drives my real Mac and Android phone

Description and use

Agents can now control real Mac and Android UIs, executing goal‑based tasks with deterministic replay and local secret handling.

Problem

AI agents often stop at text and require manual execution of code snippets, limiting their ability to interact with real user interfaces.

Outcome

Agents can now perform multi‑step UI tasks on real devices, achieving deterministic repeatability, self‑healing, and local secret handling.

Workflow

• 1. Install Auten MCP server via npm: claude mcp add auten -- npx -y @autenai/mcp
• 2. Configure the MCP client to use Auten.
• 3. Provide the agent with a goal (e.g., "open System Settings and turn on Night Shift").
• 4. Agent reads the accessibility tree, identifies UI elements, and performs clicks and selections.
• 5. Agent repeats the task on Android (e.g., "turn on battery saver").
• 6. For repeatable tasks, record a skill once; Auten replays it deterministically without AI calls.
• 7. If UI changes, Auten self‑heals by repairing the failed step.
• 8. Secrets are entered locally, never exposed to the agent.

Evidence

Auten ships as an MCP server on npm, so any MCP client can use it:

Source

You do not script coordinates or write selectors. You give the agent a goal, and it reads the accessibility tree to find what it needs.

Source

The agent listed the on‑screen elements, clicked into Displays, found the Night Shift button by its label, opened the schedule dropdown, and picked the option.

Source

Record a task once and Auten saves it as a skill with parameters. Next time Auten replays it deterministically — seconds, no model tokens.

Source

Secrets stay local. Passwords and keys are typed on your machine through a local login path; the agent never sees the value.

Source