Your robots.txt says yes. Your CDN says no. (We tested the top 1,000 sites)
Description and use
<p>Most "do AI crawlers block" articles only read robots.txt. That file is a request, not a lock. The server, the CDN and the bot-management layer decide what a crawler actually gets, and nobody was measuring that part. So on 11 October 2026 we did.</p> <p><strong>The one-line result:</strong> of the 549 top-1,000 sites that answered a normal browser, 78 (14.2%) refused at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot while their robots.txt allowed it. Leave out the 14 sites whose only refusal was a 429 rate limit, and it is still 64 sites, or 11.7%.</p> <p>That is the gap bet
Problem
<p>Most "do AI crawlers block" articles only read robots.txt. That file is a request, not a lock. The server, the CDN and the bot-management layer decide what a crawler actually gets, and nobody was measuring that part. So on 11 October 2026 we did.</p> <p><strong>The one-line result:</strong> of the 549 top-1,000 sites that answered a normal browser, 78 (14.2%) refused at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot while their robots.txt allowed it. Leave out the 14 sites whose only refusal was a 429 rate limit, and it is still 64 sites, or 11.7%.</p> <p>That is the gap bet
Outcome
<p>Most "do AI crawlers block" articles only read robots.txt. That file is a request, not a lock. The server, the CDN and the bot-management layer decide what a crawler actually gets, and nobody was measuring that part. So on 11 October 2026 we did.</p> <p><strong>The one-line result:</strong> of the 549 top-1,000 sites that answered a normal browser, 78 (14.2%) refused at least one of GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot while their robots.txt allowed it. Leave out the 14 sites whose only refusal was a 429 rate limit, and it is still 64 sites, or 11.7%.</p> <p>That is the gap bet
Evidence
Your robots.txt says yes. Your CDN says no. (We tested the top 1,000 sites)
- Record date: 2026-10-15T22:41:01.612Z