Give Your AI Agent Eyes and Hands on a Remote Desktop with rdp-cli

说明与用途

本指南演示如何使用 rdp-cli 让 AI 代理通过 RDP/VNC 控制远程桌面，实现基于观察-执行-验证的 GUI 自动化。

问题

AI agents need to interact with graphical desktop applications, which current APIs do not support.

结果

Agents can now control remote desktops by following an observe‑act‑verify loop, enabling automation of tasks that require GUI interaction.

工作流

• Install rdp-cli on Linux x86_64
• Connect to a Windows RDP session using username and password
• Capture a screenshot of the current desktop
• Use the screenshot to locate UI elements and decide actions
• Execute click and type commands with rdp-cli, providing observations
• Capture a final screenshot to verify the action
• Optionally watch or take over the session with a native viewer

证据

rdp-cli provides a simple command-line workflow for letting an AI agent control RDP and VNC desktops—and verify every action through screenshots.

来源

The command returns a session identifier. Save it as SESSION for the next steps.

来源

An AI agent can inspect before.png, locate a button or input field, and decide what to do next.

来源

The agent can inspect the final screenshot instead of assuming the command succeeded.

来源