Can AI crawlers read your site? I checked mine

Description and use

<p><em>First published on <a href="https://yappatyih.com/writing/can-ai-crawlers-read-your-site/" rel="noopener noreferrer">yappatyih.com</a>.</em></p> <h2> TL;DR </h2> <p>An AI crawler access check is a test of whether the bots behind ChatGPT, Claude, Perplexity and the others can fetch your pages, by requesting a page with each bot's name and reading the status code. On 30 September 2026 I ran one on this site: eight crawler names, eight answers of 200. Nothing was keeping the machines out. What the same audit found instead was a line in my own robots.txt, <code>Disallow: /og/</code>, that b

Problem

<p><em>First published on <a href="https://yappatyih.com/writing/can-ai-crawlers-read-your-site/" rel="noopener noreferrer">yappatyih.com</a>.</em></p> <h2> TL;DR </h2> <p>An AI crawler access check is a test of whether the bots behind ChatGPT, Claude, Perplexity and the others can fetch your pages, by requesting a page with each bot's name and reading the status code. On 30 September 2026 I ran one on this site: eight crawler names, eight answers of 200. Nothing was keeping the machines out. What the same audit found instead was a line in my own robots.txt, <code>Disallow: /og/</code>, that b

Outcome

<p><em>First published on <a href="https://yappatyih.com/writing/can-ai-crawlers-read-your-site/" rel="noopener noreferrer">yappatyih.com</a>.</em></p> <h2> TL;DR </h2> <p>An AI crawler access check is a test of whether the bots behind ChatGPT, Claude, Perplexity and the others can fetch your pages, by requesting a page with each bot's name and reading the status code. On 30 September 2026 I ran one on this site: eight crawler names, eight answers of 200. Nothing was keeping the machines out. What the same audit found instead was a line in my own robots.txt, <code>Disallow: /og/</code>, that b

Evidence

Can AI crawlers read your site? I checked mine

Source