Mobilerun Achieves 100% on AndroidWorld, Beating Google ARTEMIS

Description and use

Mobilerun achieved a perfect 100% score on the AndroidWorld benchmark, surpassing Google ARTEMIS, by using mobile-harness to provide tools, app cards, and shared memory for mobile automation.

Problem

Achieving a perfect score on AndroidWorld benchmark for mobile AI agents.

Outcome

Mobilerun achieved a perfect 100% official score on AndroidWorld, outperforming Google ARTEMIS (99%+).

Workflow

• Run AndroidWorld benchmark with mobile-harness
• Use mobile tools to interact with device
• Leverage app cards for structured app knowledge
• Utilize shared memory for recurring patterns
• Verify outcomes via accessibility info and screenshots
• Record interactions, screenshots, tool ops, judgments, timing

Evidence

Mobilerun just achieved a 100% official score on AndroidWorld, Google Research's benchmark for evaluating AI agents on Android.

Source

For our latest evaluation, we ran AndroidWorld using mobile-harness, our set of mobile tools and instructions for running mobile tasks through coding agents.

Source

The evaluation used: Tasks:116, Full scores:116, Official score:100%, Recorded interactions:1,419, Device:Pixel 6 emulator, OS:Android 13 / API 33, Model:gpt-6-astra / low.

Source

mobilerun / mobile-harness: 100%, Google ARTEMIS: 99%+

Source