PageTwins: I Planted One Defect on 100 Landing Pages. Which AI Graders Noticed?

Description and use

Comparing AI models on detecting defects in landing pages using screenshots and text revealed that strong vision-enabled models almost always detect defects even without a reference, while weaker models suffer from position bias and perform worse with images.

Problem

Determine how well AI grading models detect subtle defects in landing pages when given only text versus screenshots.

Outcome

Strong vision models (Claude Sonnet 5.5, GPT-6.1 Sol, Gemini 3.7 Flash, Gemma 4 31B) achieved near-perfect defect detection with or without reference screenshots. Weak models (Gemini 3.5 Flash-Lite, Claude Haiku 4.5) performed poorly, especially when given screenshots, and exhibited position bias. Text-only models performed similarly to vision models on defects visible in both views, but failed on purely visual defects.

Workflow

• 1. Build PageTwins dataset with 10 originals and 100 twins each having one defect.
• 2. Render each twin in headless Chromium and keep only if defect measurement changed.
• 3. Create four benchmark tasks using kaggle-benchmarks SDK.
• 4. Run ten models (four vision-capable, six text-only) on all tasks via Kaggle CLI.
• 5. Collect and analyze scores for defect detection and control handling.
• 6. Compare performance between screenshot and text inputs.
• 7. Interpret results to assess model strengths and weaknesses.

Evidence

I built PageTwins: 10 original landing pages and 100 "evil twins". Each twin is identical to its original except for one planted defect.

Source

Claude Sonnet 5.5 caught 98% of defects judging a page alone and 100% with the original next to it; GPT-6.1 Sol 98% and 96%; Gemini 3.7 Flash 96% and 98%; Gemma 4 31B 93% and 95%.

Source

Restricted to the seven defects that are visible in both views, and pooled over the seven models that ran both, the screenshot and the text view tie: 82.9% vs 82.2% caught on the single-page task.

Source