PageTwins: I Planted One Defect on 100 Landing Pages. Which AI Graders Noticed?

说明与用途

使用截图和文本比较 AI 模型检测登录页面缺陷，结果显示强大的视觉模型几乎总能检测到缺陷，即使没有参考页面，而弱模型则受到位置偏差影响，图像输入效果更差。

问题

Determine how well AI grading models detect subtle defects in landing pages when given only text versus screenshots.

结果

Strong vision models (Claude Sonnet 5.5, GPT-6.1 Sol, Gemini 3.7 Flash, Gemma 4 31B) achieved near-perfect defect detection with or without reference screenshots. Weak models (Gemini 3.5 Flash-Lite, Claude Haiku 4.5) performed poorly, especially when given screenshots, and exhibited position bias. Text-only models performed similarly to vision models on defects visible in both views, but failed on purely visual defects.

工作流

• 1. Build PageTwins dataset with 10 originals and 100 twins each having one defect.
• 2. Render each twin in headless Chromium and keep only if defect measurement changed.
• 3. Create four benchmark tasks using kaggle-benchmarks SDK.
• 4. Run ten models (four vision-capable, six text-only) on all tasks via Kaggle CLI.
• 5. Collect and analyze scores for defect detection and control handling.
• 6. Compare performance between screenshot and text inputs.
• 7. Interpret results to assess model strengths and weaknesses.

证据

I built PageTwins: 10 original landing pages and 100 "evil twins". Each twin is identical to its original except for one planted defect.

来源

Claude Sonnet 5.5 caught 98% of defects judging a page alone and 100% with the original next to it; GPT-6.1 Sol 98% and 96%; Gemini 3.7 Flash 96% and 98%; Gemma 4 31B 93% and 95%.

来源

Restricted to the seven defects that are visible in both views, and pooled over the seven models that ran both, the screenshot and the text view tie: 82.9% vs 82.2% caught on the single-page task.

来源