Giving an AI Agent a Real Phone: What ADB Automation Actually Looks Like

Description and use

ADB automation of an Android device is harder than expected: the

Problem

Driving an Android device from an AI agent over ADB is harder than driving a desktop, because the interface is easy but state management is hard: secure screens return zero-byte screenshots, UI trees lie about what is tappable, and non-ASCII text cannot be typed via input text.

Outcome

With the read-act-read rule and secure-screen detection, an AI agent can drive any Android app a human can drive by hand, including opening apps, logging in once, and performing repetitive tasks without manual intervention.

Workflow

• Use adb shell uiautomator dump to get the view hierarchy as XML with bounds for each node.
• Use adb shell input tap/swipe/text to inject events.
• Use adb exec-out screencap -p to capture a PNG screenshot.
• Check byte length and PNG header; if zero bytes, treat as FLAG_SECURE screen and switch strategy.
• For Compose/Flutter apps where coordinates resolve to the same element, use adb shell input keyevent to move focus between fields.
• Verify by reading the tree back after every action rather than assuming the tap landed.
• For non-ASCII text, use clipboard: adb am broadcast -a clipper.set -e text "..." then adb shell input keyevent 279 to paste.
• Never act on a cached view of the screen: read, act, read again.

Evidence

adb shell uiautomator dump returns the view hierarchy as XML with bounds for each node.

Source

adb shell input tap/swipe/text injects events.

Source

adb exec-out screencap -p gives you a PNG.

Source

FLAG_SECURE screens return a zero-byte screenshot. If your loop treats an empty image as 'screen unchanged', it will tap happily for ten minutes.

Source

The UI tree lies about what is tappable. The fix is not bigger coordinates: use the keyboard (adb shell input keyevent) to move focus between fields, and verify by reading the tree back after every action.

Source

gettext does not exist in the shell. Clipboard is the only sane way to type anything non-trivial: adb am broadcast -a clipper.set -e text "..." (or a clipboard helper), then input keyevent 279 to paste.

Source

Never act on a cached view of the screen. Read, act, read again. It is slower per step and faster per task, because you stop discovering five steps later that the third tap went to the wrong window.

Source