Does the Web Page Your AI Agent Just Wrote Actually Work?

Description and use

Assay automatically tests AI‑generated web pages, detecting real defects without manual interaction.

Problem

AI‑generated web pages often contain functional bugs that are invisible in the source code and require manual interaction to detect.

Outcome

Across 225 AI‑generated pages, assay found 15 of 20 real defects and raised no false alarms on 205 working pages. Median run time was 14 s, 220 pages finished within a minute.

Workflow

• Point assay at a generated page
• Assay opens the page in a real browser
• It waits until the set of controls stabilizes
• Derives a test plan from the controls
• Drives every control (clicks, inputs, drags) twice
• Records before/after state (pixels, text, styles)
• Compares states to identify failures
• Reports results and optional screenshots

Evidence

across 225 pages it found 15 of the 20 real defects and raised 0 false alarms on the 205 that work

Source

the whole time i was thinking: the model wrote each of these in seconds, and the only way anyone finds out whether it works is a person, sitting here, clicking

Source