PageTwins: I Planted One Defect on 100 Landing Pages. Which AI Graders Noticed?

Описание и применение

Сравнение моделей ИИ по обнаружению дефектов на посадочных страницах с использованием снимков экрана и текста показало, что сильные модели с визуальной поддержкой почти всегда выявляют дефекты, даже без ссылки, тогда как слабые модели страдают от смещения и хуже работают с изображениями.

Задача

Determine how well AI grading models detect subtle defects in landing pages when given only text versus screenshots.

Результат

Strong vision models (Claude Sonnet 5.5, GPT-6.1 Sol, Gemini 3.7 Flash, Gemma 4 31B) achieved near-perfect defect detection with or without reference screenshots. Weak models (Gemini 3.5 Flash-Lite, Claude Haiku 4.5) performed poorly, especially when given screenshots, and exhibited position bias. Text-only models performed similarly to vision models on defects visible in both views, but failed on purely visual defects.

Рабочий процесс

• 1. Build PageTwins dataset with 10 originals and 100 twins each having one defect.
• 2. Render each twin in headless Chromium and keep only if defect measurement changed.
• 3. Create four benchmark tasks using kaggle-benchmarks SDK.
• 4. Run ten models (four vision-capable, six text-only) on all tasks via Kaggle CLI.
• 5. Collect and analyze scores for defect detection and control handling.
• 6. Compare performance between screenshot and text inputs.
• 7. Interpret results to assess model strengths and weaknesses.

Доказательства

I built PageTwins: 10 original landing pages and 100 "evil twins". Each twin is identical to its original except for one planted defect.

Источник

Claude Sonnet 5.5 caught 98% of defects judging a page alone and 100% with the original next to it; GPT-6.1 Sol 98% and 96%; Gemini 3.7 Flash 96% and 98%; Gemma 4 31B 93% and 95%.

Источник

Restricted to the seven defects that are visible in both views, and pooled over the seven models that ran both, the screenshot and the text view tie: 82.9% vs 82.2% caught on the single-page task.

Источник