An iPhone agent in Safari, with an ESP32 as its Bluetooth hands

Description and use

iPhone Agent Harness enables controlling an iPhone via Safari, using an ESP32 as a Bluetooth HID bridge and a vision model to analyze the screen.

Problem

Enable a phone to be controlled by a vision model via a nearby ESP32 without a native iOS app.

Outcome

The harness demonstrates automated setting of an alarm in the iPhone Clock app, proving the concept of web‑based phone automation via Bluetooth HID.

Workflow

• 1. Flash ESP32 with provided firmware via browser installer.
• 2. Connect ESP32 to Wi‑Fi and pair over Bluetooth; enable AssistiveTouch on iPhone.
• 3. Open iPhone Agent Harness in Safari and configure vision model access.
• 4. Share the iPhone screen with the harness and start a task.
• 5. Vision model analyzes screen, plans actions, and harness sends WebRTC commands to ESP32.
• 6. ESP32 emits Bluetooth mouse/keyboard events to iPhone, performing the task.
• 7. Observe the result on the iPhone screen.

Evidence

We built iPhone Agent Harness to explore a simple interaction: describe a phone task in Safari, let a vision model inspect the screen, and use a nearby ESP32 to supply the taps and keyboard input.

Source

The current demo shows the agent setting an alarm in the iPhone Clock app.

Source

The ESP32 is an input bridge. Safari runs the conversation, screen capture and model interaction.

Source

AssistiveTouch lets the phone respond to pointer input.

Source

The current requirement is iOS / iPadOS 27 or later. The phone and bridge need to share a Wi‑Fi network.

Source