Building AetherDraw: An Agent‑Native Infinite Whiteboard with WebMCP

Description and use

AetherDraw gives AI agents direct access to a visual whiteboard’s internal scene graph via WebMCP, enabling millisecond‑level operations with deterministic layout and routing.

Problem

AI agents cannot interact with visual whiteboards because existing methods rely on pixel‑based vision or fragile DOM automation, leading to slow, token‑heavy, and error‑prone interactions.

Outcome

AetherDraw enables AI agents to interact with visual whiteboards in milliseconds, supporting true bi‑directional co‑creation and deterministic layout, improving efficiency and reliability over vision or DOM automation.

Workflow

• 1. Agent calls get_canvas_state to retrieve scene AST.
• 2. Agent uses find_elements or get_selected_elements to locate user‑drawn shapes.
• 3. Agent calls create_diagram or connect_elements to add or link elements using deterministic graph algorithms.
• 4. Agent applies apply_auto_layout or apply_theme for visual refinement.
• 5. Agent exports or inspects canvas via export_canvas or live telemetry.

Evidence

Instead of forcing AI agents to squint at canvas pixels from the outside, the web application can expose its internal scene AST, geometric bounds, and layout algorithms directly to the browser runtime via document.modelContext.

Source

AetherDraw is an open‑source, agent‑native infinite whiteboard. It takes the battle‑tested Excalidraw canvas engine and extends it with a suite of 11 schema‑typed tools organized into four functional layers.

Source