I gave Claude Code hands: it drives my real Mac and Android phone

说明与用途

代理现在可以控制真实的 Mac 和 Android UI，执行基于目标的任务，具有确定性重放和本地秘密处理。

问题

AI agents often stop at text and require manual execution of code snippets, limiting their ability to interact with real user interfaces.

结果

Agents can now perform multi‑step UI tasks on real devices, achieving deterministic repeatability, self‑healing, and local secret handling.

工作流

• 1. Install Auten MCP server via npm: claude mcp add auten -- npx -y @autenai/mcp
• 2. Configure the MCP client to use Auten.
• 3. Provide the agent with a goal (e.g., "open System Settings and turn on Night Shift").
• 4. Agent reads the accessibility tree, identifies UI elements, and performs clicks and selections.
• 5. Agent repeats the task on Android (e.g., "turn on battery saver").
• 6. For repeatable tasks, record a skill once; Auten replays it deterministically without AI calls.
• 7. If UI changes, Auten self‑heals by repairing the failed step.
• 8. Secrets are entered locally, never exposed to the agent.

证据

Auten ships as an MCP server on npm, so any MCP client can use it:

来源

You do not script coordinates or write selectors. You give the agent a goal, and it reads the accessibility tree to find what it needs.

来源

The agent listed the on‑screen elements, clicked into Displays, found the Night Shift button by its label, opened the schedule dropdown, and picked the option.

来源

Record a task once and Auten saves it as a skill with parameters. Next time Auten replays it deterministically — seconds, no model tokens.

来源

Secrets stay local. Passwords and keys are typed on your machine through a local login path; the agent never sees the value.

来源