My website audit crawler named Google as the company that built the site

Описание и применение

Краулер веб‑аудита ошибочно определил Google как создателя сайта из-за ссылок reCAPTCHA, а также неверно подсчитал страницы с поставщиками. После внедрения черного списка инфраструктурных хостов, правил подсчёта и проверки на этапе выхода, инструмент теперь корректно выявляет внешних поставщиков и не выдаёт ложных заявлений.

Задача

The crawler incorrectly identified Google as the builder of a small‑business website due to reCAPTCHA badge links, and misreported vendor page counts, leading to false statements in audit briefs.

Результат

The updated crawler accurately identifies external vendors, avoids misattributing Google as a builder, correctly reports vendor page counts, and enforces strict gate checks to prevent false positives in audit briefs.

Рабочий процесс

• Crawl up to 60 pages of the target site
• Extract external hosts from src, href, form actions
• Filter out known infrastructure hosts and already identified vendors
• Count unique vendor hosts and record page occurrences
• Generate markdown brief with vendor information
• Run gate script to block disallowed patterns and enforce write permissions
• Save brief only on successful gate pass

Доказательства

for every link on every page, if it points off‑site and the path contains privacy, terms or cookies, count that host as a candidate builder

The fix had two parts. First, a short list of known infrastructure hosts that can never be the builder

When nothing is left, the field says not detected

The reviewer tried replacing len(pages) with total_pages, which turns "1 of 8" into "8 of 8" for every vendor