Giving an AI Agent a Real Phone: What ADB Automation Actually Looks Like

Описание и применение

Движение по ADB к Android-устройству оказалось сложнее, чем ожидалось: интерфейс прост, а управление состоянием — нет. Три проблемы: зашироченные экраны (zero-байтовые скэдео), деревцы UI от Jetpack Compose/Flutter (один узел на весь экран), отсутствие gettext в shell. Решение — читать дерево, действовать, читать снова; для не-ASCII текста использовать剪clipboard.Agent может открыть приложение, авторизороваться и выполнять повторяющиеся задачи.

Задача

Driving an Android device from an AI agent over ADB is harder than driving a desktop, because the interface is easy but state management is hard: secure screens return zero-byte screenshots, UI trees lie about what is tappable, and non-ASCII text cannot be typed via input text.

Результат

With the read-act-read rule and secure-screen detection, an AI agent can drive any Android app a human can drive by hand, including opening apps, logging in once, and performing repetitive tasks without manual intervention.

Рабочий процесс

• Use adb shell uiautomator dump to get the view hierarchy as XML with bounds for each node.
• Use adb shell input tap/swipe/text to inject events.
• Use adb exec-out screencap -p to capture a PNG screenshot.
• Check byte length and PNG header; if zero bytes, treat as FLAG_SECURE screen and switch strategy.
• For Compose/Flutter apps where coordinates resolve to the same element, use adb shell input keyevent to move focus between fields.
• Verify by reading the tree back after every action rather than assuming the tap landed.
• For non-ASCII text, use clipboard: adb am broadcast -a clipper.set -e text "..." then adb shell input keyevent 279 to paste.
• Never act on a cached view of the screen: read, act, read again.

Доказательства

adb shell uiautomator dump returns the view hierarchy as XML with bounds for each node.

Источник

adb shell input tap/swipe/text injects events.

Источник

adb exec-out screencap -p gives you a PNG.

Источник

FLAG_SECURE screens return a zero-byte screenshot. If your loop treats an empty image as 'screen unchanged', it will tap happily for ten minutes.

Источник

The UI tree lies about what is tappable. The fix is not bigger coordinates: use the keyboard (adb shell input keyevent) to move focus between fields, and verify by reading the tree back after every action.

Источник

gettext does not exist in the shell. Clipboard is the only sane way to type anything non-trivial: adb am broadcast -a clipper.set -e text "..." (or a clipboard helper), then input keyevent 279 to paste.

Источник

Never act on a cached view of the screen. Read, act, read again. It is slower per step and faster per task, because you stop discovering five steps later that the third tap went to the wrong window.

Источник