My website audit crawler named Google as the company that built the site

Description and use

The web audit crawler mistakenly identified Google as the site builder due to reCAPTCHA badge links and miscounted vendor pages. After adding infrastructure host blacklists, accurate page‑count logic, and gate checks, the tool now correctly detects external vendors and avoids false statements.

Problem

The crawler incorrectly identified Google as the builder of a small‑business website due to reCAPTCHA badge links, and misreported vendor page counts, leading to false statements in audit briefs.

Outcome

The updated crawler accurately identifies external vendors, avoids misattributing Google as a builder, correctly reports vendor page counts, and enforces strict gate checks to prevent false positives in audit briefs.

Workflow

• Crawl up to 60 pages of the target site
• Extract external hosts from src, href, form actions
• Filter out known infrastructure hosts and already identified vendors
• Count unique vendor hosts and record page occurrences
• Generate markdown brief with vendor information
• Run gate script to block disallowed patterns and enforce write permissions
• Save brief only on successful gate pass

Evidence

for every link on every page, if it points off‑site and the path contains privacy, terms or cookies, count that host as a candidate builder

The fix had two parts. First, a short list of known infrastructure hosts that can never be the builder

When nothing is left, the field says not detected

The reviewer tried replacing len(pages) with total_pages, which turns "1 of 8" into "8 of 8" for every vendor