Running the Same GUI Agent on Different Hardware: What Changes?

说明与用途

文章说明了如何在不同设备上评估视觉驱动的 GUI 代理的性能和可靠性，强调任务级指标而非仅仅是令牌吞吐量。

问题

Determining how hardware form factors affect the performance and reliability of a vision‑driven GUI agent.

结果

The article establishes that task‑level evaluation is essential for comparing hardware form factors, and that metrics such as task pass rate, per‑step latency, and sustained performance provide a more accurate picture than token throughput alone.

工作流

• 1. Select the same model, runtime, and quantization configuration across devices.
• 2. Run the identical task set from the same initial UI state on each device.
• 3. Define task success criteria before testing.
• 4. Repeat tasks and record both averages and failure cases.
• 5. Measure per‑step latency and total task time.
• 6. Record memory pressure, power state, and sustained performance.
• 7. Separate model failures from UI execution and recovery failures.

证据

A GUI agent may process a screenshot at every step, along with the task instruction and the history of previous actions.

来源

In one existing test, we ran Mano-CUA-Thinking-4B on a MacBook Pro M5 with 16GB of memory and evaluated it on 100 real macOS GUI tasks.

来源

Task pass rate: 56%

来源

Average inference time per step: 7.9 seconds

来源

Average number of steps per task: 11.5

来源

If we compare the same agent across a desktop, a laptop, and a dedicated device, we should keep the variables as consistent as possible: ...

来源