I measured whether datacenter IPs are actually blocked. Only 4% of the top 1,000 sites refuse them.

Description and use

Only 4 % of the 772 sites from the Tranco list block datacenter IPs; 24 of 25 sites still respond to datacenter requests. Popularity does not predict blocking, and residential IPs are not strictly better.

Problem

Scrapers often assume datacenter IPs are blocked by major sites, but no quantitative evidence existed.

Outcome

Only 4% of the 772 sites that serve a homepage block datacenter IPs; 24 of 25 sites still respond to datacenter requests. Popularity does not predict blocking, and residential IPs are not strictly better.

Workflow

• 1. Retrieve Tranco top 1,000 list daily.
• 2. Resolve each hostname to IPs, filtering out names without an apex address.
• 3. For each host, send a GET / request twice at the same moment: once from a datacenter IP, once from a residential exit, following up to three redirects.
• 4. Compare status and headers only; ignore body.
• 5. Re-ask any refusal twice from datacenter and up to three times from residential to confirm reproducibility.
• 6. Publish only cases where the two responses disagree.
• 7. Aggregate results and publish JSON/CSV via public API.

Evidence

228 of the top 1,000 names have no address at their apex at all.

Source

Four percent. Twenty-four sites in twenty-five still answer a datacenter IP.

Source

Ask twice, at the same moment.

Source