I gave Claude Code hands: it drives my real Mac and Android phone

Описание и применение

Агенты теперь могут управлять реальными интерфейсами Mac и Android, выполняя задачи по целям, а не по скриптам, с детерминированным воспроизведением и локальной обработкой секретов.

Задача

AI agents often stop at text and require manual execution of code snippets, limiting their ability to interact with real user interfaces.

Результат

Agents can now perform multi‑step UI tasks on real devices, achieving deterministic repeatability, self‑healing, and local secret handling.

Рабочий процесс

• 1. Install Auten MCP server via npm: claude mcp add auten -- npx -y @autenai/mcp
• 2. Configure the MCP client to use Auten.
• 3. Provide the agent with a goal (e.g., "open System Settings and turn on Night Shift").
• 4. Agent reads the accessibility tree, identifies UI elements, and performs clicks and selections.
• 5. Agent repeats the task on Android (e.g., "turn on battery saver").
• 6. For repeatable tasks, record a skill once; Auten replays it deterministically without AI calls.
• 7. If UI changes, Auten self‑heals by repairing the failed step.
• 8. Secrets are entered locally, never exposed to the agent.

Доказательства

Auten ships as an MCP server on npm, so any MCP client can use it:

Источник

You do not script coordinates or write selectors. You give the agent a goal, and it reads the accessibility tree to find what it needs.

Источник

The agent listed the on‑screen elements, clicked into Displays, found the Night Shift button by its label, opened the schedule dropdown, and picked the option.

Источник

Record a task once and Auten saves it as a skill with parameters. Next time Auten replays it deterministically — seconds, no model tokens.

Источник

Secrets stay local. Passwords and keys are typed on your machine through a local login path; the agent never sees the value.

Источник