I measured whether datacenter IPs are actually blocked. Only 4% of the top 1,000 sites refuse them.

Описание и применение

Только 4 % из 772 сайтов из списка Tranco блокируют IP‑адреса дата‑центров; 24 из 25 сайтов всё ещё отвечают на запросы с дата‑центров. Популярность сайта не предсказывает блокировку, а резидентные IP не всегда лучше.

Задача

Scrapers often assume datacenter IPs are blocked by major sites, but no quantitative evidence existed.

Результат

Only 4% of the 772 sites that serve a homepage block datacenter IPs; 24 of 25 sites still respond to datacenter requests. Popularity does not predict blocking, and residential IPs are not strictly better.

Рабочий процесс

• 1. Retrieve Tranco top 1,000 list daily.
• 2. Resolve each hostname to IPs, filtering out names without an apex address.
• 3. For each host, send a GET / request twice at the same moment: once from a datacenter IP, once from a residential exit, following up to three redirects.
• 4. Compare status and headers only; ignore body.
• 5. Re-ask any refusal twice from datacenter and up to three times from residential to confirm reproducibility.
• 6. Publish only cases where the two responses disagree.
• 7. Aggregate results and publish JSON/CSV via public API.

Доказательства

228 of the top 1,000 names have no address at their apex at all.

Источник

Four percent. Twenty-four sites in twenty-five still answer a datacenter IP.

Источник

Ask twice, at the same moment.

Источник