An iPhone agent in Safari, with an ESP32 as its Bluetooth hands

说明与用途

iPhone Agent Harness 通过 Safari 控制 iPhone，使用 ESP32 作为 Bluetooth HID 桥接，并利用视觉模型分析屏幕。

问题

Enable a phone to be controlled by a vision model via a nearby ESP32 without a native iOS app.

结果

The harness demonstrates automated setting of an alarm in the iPhone Clock app, proving the concept of web‑based phone automation via Bluetooth HID.

工作流

• 1. Flash ESP32 with provided firmware via browser installer.
• 2. Connect ESP32 to Wi‑Fi and pair over Bluetooth; enable AssistiveTouch on iPhone.
• 3. Open iPhone Agent Harness in Safari and configure vision model access.
• 4. Share the iPhone screen with the harness and start a task.
• 5. Vision model analyzes screen, plans actions, and harness sends WebRTC commands to ESP32.
• 6. ESP32 emits Bluetooth mouse/keyboard events to iPhone, performing the task.
• 7. Observe the result on the iPhone screen.

证据

We built iPhone Agent Harness to explore a simple interaction: describe a phone task in Safari, let a vision model inspect the screen, and use a nearby ESP32 to supply the taps and keyboard input.

来源

The current demo shows the agent setting an alarm in the iPhone Clock app.

来源

The ESP32 is an input bridge. Safari runs the conversation, screen capture and model interaction.

来源

AssistiveTouch lets the phone respond to pointer input.

来源

The current requirement is iOS / iPadOS 27 or later. The phone and bridge need to share a Wi‑Fi network.

来源