Give Your AI Agent Eyes and Hands on a Remote Desktop with rdp-cli

Description and use

This guide shows how to use rdp-cli to give an AI agent control over remote desktops via RDP/VNC, enabling GUI automation with an observe‑act‑verify loop.

Problem

AI agents need to interact with graphical desktop applications, which current APIs do not support.

Outcome

Agents can now control remote desktops by following an observe‑act‑verify loop, enabling automation of tasks that require GUI interaction.

Workflow

• Install rdp-cli on Linux x86_64
• Connect to a Windows RDP session using username and password
• Capture a screenshot of the current desktop
• Use the screenshot to locate UI elements and decide actions
• Execute click and type commands with rdp-cli, providing observations
• Capture a final screenshot to verify the action
• Optionally watch or take over the session with a native viewer

Evidence

rdp-cli provides a simple command-line workflow for letting an AI agent control RDP and VNC desktops—and verify every action through screenshots.

Source

The command returns a session identifier. Save it as SESSION for the next steps.

Source

An AI agent can inspect before.png, locate a button or input field, and decide what to do next.

Source

The agent can inspect the final screenshot instead of assuming the command succeeded.

Source