Mobilerun Achieves 100% on AndroidWorld, Beating Google ARTEMIS

Описание и применение

Mobilerun достиг 100 % на бенчмарке AndroidWorld, превзойдя Google ARTEMIS, благодаря мобильному harness, который обеспечивает инструменты, карточки приложений и совместную память для улучшения взаимодействия с мобильными интерфейсами.

Задача

Achieving a perfect score on AndroidWorld benchmark for mobile AI agents.

Результат

Mobilerun achieved a perfect 100% official score on AndroidWorld, outperforming Google ARTEMIS (99%+).

Рабочий процесс

• Run AndroidWorld benchmark with mobile-harness
• Use mobile tools to interact with device
• Leverage app cards for structured app knowledge
• Utilize shared memory for recurring patterns
• Verify outcomes via accessibility info and screenshots
• Record interactions, screenshots, tool ops, judgments, timing

Доказательства

Mobilerun just achieved a 100% official score on AndroidWorld, Google Research's benchmark for evaluating AI agents on Android.

Источник

For our latest evaluation, we ran AndroidWorld using mobile-harness, our set of mobile tools and instructions for running mobile tasks through coding agents.

Источник

The evaluation used: Tasks:116, Full scores:116, Official score:100%, Recorded interactions:1,419, Device:Pixel 6 emulator, OS:Android 13 / API 33, Model:gpt-6-astra / low.

Источник

mobilerun / mobile-harness: 100%, Google ARTEMIS: 99%+

Источник