My website audit crawler named Google as the company that built the site

说明与用途

网站审核爬虫因 reCAPTCHA 徽标链接错误识别 Google 为网站构建者，并错误计数供应商页面。通过添加基础设施主机黑名单、准确页面计数逻辑和门控检查，工具现在能正确检测外部供应商并避免错误声明。

问题

The crawler incorrectly identified Google as the builder of a small‑business website due to reCAPTCHA badge links, and misreported vendor page counts, leading to false statements in audit briefs.

结果

The updated crawler accurately identifies external vendors, avoids misattributing Google as a builder, correctly reports vendor page counts, and enforces strict gate checks to prevent false positives in audit briefs.

工作流

• Crawl up to 60 pages of the target site
• Extract external hosts from src, href, form actions
• Filter out known infrastructure hosts and already identified vendors
• Count unique vendor hosts and record page occurrences
• Generate markdown brief with vendor information
• Run gate script to block disallowed patterns and enforce write permissions
• Save brief only on successful gate pass

证据

for every link on every page, if it points off‑site and the path contains privacy, terms or cookies, count that host as a candidate builder

The fix had two parts. First, a short list of known infrastructure hosts that can never be the builder

When nothing is left, the field says not detected

The reviewer tried replacing len(pages) with total_pages, which turns "1 of 8" into "8 of 8" for every vendor