How I built Browy, an AI agent that lives in your browser

Description and use

Browy is an AI agent that runs inside your browser, leveraging your signed‑in session to automate web tasks while allowing users to enable or disable potentially unsafe tools.

Problem

Develop an AI agent that can interact with web pages in the user’s existing browser session, leveraging the user’s authentication and data without launching a separate browser instance.

Outcome

Browy enables authenticated web automation within the user’s existing browser, eliminating the need for a separate headless browser and preserving login state. It achieves this through a native messaging host, Copilot SDK integration, and a comprehensive set of browser tools, while allowing users to enable or disable potentially risky tools.

Workflow

• 1. Install Browy extension from Chrome Web Store.
• 2. Run a one‑line PowerShell/terminal command to register the native host.
• 3. Open the side panel or DevTools panel to chat with the agent.
• 4. Agent sends a snapshot of the active tab (title, URL, viewport, scroll, selected text, DOM snapshot).
• 5. Agent calls a tool (e.g., click_index, fill_form) via the Copilot SDK.
• 6. Host forwards DevTools Protocol commands to the browser via chrome.debugger.
• 7. Browser executes actions and returns results to the host.
• 8. Host reports changes (urlChanged, domChanged, modalAppeared, elementStateChanged) back to the agent.
• 9. Agent updates chat history and repeats for subsequent messages.

Evidence

Browy is an AI agent that lives in your browser, and I mean that literally.

Source

When you send Browy its first message, Chrome puts up a bar saying that Browy started debugging this browser.

Source

The host is a Node program, and it's where the agent lives. It wraps GitHub's Copilot SDK, which starts the third program, the Copilot CLI, in headless mode and talks JSON-RPC to it over another pair of pipes.

Source

Browy has 54 browser tools, from click_index and fill_form to get_network_requests and replay_request. Most have a switch in Browy's settings.

Source

Browy sidesteps that by staying in your browser. Ask about the pull request in front of you, and it reads the pull request in front of you, in the tab where you're already signed in.

Source