Running the Same GUI Agent on Different Hardware: What Changes?

Описание и применение

Статья описывает, как оценивать производительность и надёжность визуального GUI‑агента на разных устройствах, подчёркивая важность метрик уровня задачи, а не только пропускной способности токенов.

Задача

Determining how hardware form factors affect the performance and reliability of a vision‑driven GUI agent.

Результат

The article establishes that task‑level evaluation is essential for comparing hardware form factors, and that metrics such as task pass rate, per‑step latency, and sustained performance provide a more accurate picture than token throughput alone.

Рабочий процесс

• 1. Select the same model, runtime, and quantization configuration across devices.
• 2. Run the identical task set from the same initial UI state on each device.
• 3. Define task success criteria before testing.
• 4. Repeat tasks and record both averages and failure cases.
• 5. Measure per‑step latency and total task time.
• 6. Record memory pressure, power state, and sustained performance.
• 7. Separate model failures from UI execution and recovery failures.

Доказательства

A GUI agent may process a screenshot at every step, along with the task instruction and the history of previous actions.

Источник

In one existing test, we ran Mano-CUA-Thinking-4B on a MacBook Pro M5 with 16GB of memory and evaluated it on 100 real macOS GUI tasks.

Источник

Task pass rate: 56%

Источник

Average inference time per step: 7.9 seconds

Источник

Average number of steps per task: 11.5

Источник

If we compare the same agent across a desktop, a laptop, and a dedicated device, we should keep the variables as consistent as possible: ...

Источник