Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

Your robots.txt says yes. Your CDN says no. (We tested the top 1,000 sites)

Description and use

<p>Most "do AI crawlers block" articles only read robots.txt. That file is a request, not a lock. The server, the CDN and the bot-management layer decide what a crawler actually gets, and nobody was measuring that part. So on 11 October 2026 we did.</p> <p><strong>The one-line result:</strong> of the 549 top-1,000 sites that answered a normal browser, 78 (14.2%) refused at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot while their robots.txt allowed it. Leave out the 14 sites whose only refusal was a 429 rate limit, and it is still 64 sites, or 11.7%.</p> <p>That is the gap bet

Problem

<p>Most "do AI crawlers block" articles only read robots.txt. That file is a request, not a lock. The server, the CDN and the bot-management layer decide what a crawler actually gets, and nobody was measuring that part. So on 11 October 2026 we did.</p> <p><strong>The one-line result:</strong> of the 549 top-1,000 sites that answered a normal browser, 78 (14.2%) refused at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot while their robots.txt allowed it. Leave out the 14 sites whose only refusal was a 429 rate limit, and it is still 64 sites, or 11.7%.</p> <p>That is the gap bet

Outcome

<p>Most "do AI crawlers block" articles only read robots.txt. That file is a request, not a lock. The server, the CDN and the bot-management layer decide what a crawler actually gets, and nobody was measuring that part. So on 11 October 2026 we did.</p> <p><strong>The one-line result:</strong> of the 549 top-1,000 sites that answered a normal browser, 78 (14.2%) refused at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot while their robots.txt allowed it. Leave out the 14 sites whose only refusal was a 429 rate limit, and it is still 64 sites, or 11.7%.</p> <p>That is the gap bet

Evidence

Your robots.txt says yes. Your CDN says no. (We tested the top 1,000 sites)

Source

  • Record date: 2026-10-15T22:41:01.612Z