Which blogging platforms let AI search bots read your posts? (robots.txt and raw HTML, checked 31 Oct 2026)

Description and use

<p><strong>Short answer:</strong> in my check, dev.to, Substack and Gumroad allow GPTBot, OAI-SearchBot, PerplexityBot and Google-Extended in robots.txt. Medium disallows GPTBot but allows the others. Indie Hackers disallows all four for the whole site. On dev.to the full post text is also in the raw HTML, so a crawler that does not run JavaScript still gets the article.</p> <p>Small sample: one day, one machine, plain <code>curl</code> with a browser user agent, no bot user agents spoofed. Part of the run was interrupted, so anything I could not finish is left out.</p> <h2> robots.txt, per ho

Problem

<p><strong>Short answer:</strong> in my check, dev.to, Substack and Gumroad allow GPTBot, OAI-SearchBot, PerplexityBot and Google-Extended in robots.txt. Medium disallows GPTBot but allows the others. Indie Hackers disallows all four for the whole site. On dev.to the full post text is also in the raw HTML, so a crawler that does not run JavaScript still gets the article.</p> <p>Small sample: one day, one machine, plain <code>curl</code> with a browser user agent, no bot user agents spoofed. Part of the run was interrupted, so anything I could not finish is left out.</p> <h2> robots.txt, per ho

Outcome

<p><strong>Short answer:</strong> in my check, dev.to, Substack and Gumroad allow GPTBot, OAI-SearchBot, PerplexityBot and Google-Extended in robots.txt. Medium disallows GPTBot but allows the others. Indie Hackers disallows all four for the whole site. On dev.to the full post text is also in the raw HTML, so a crawler that does not run JavaScript still gets the article.</p> <p>Small sample: one day, one machine, plain <code>curl</code> with a browser user agent, no bot user agents spoofed. Part of the run was interrupted, so anything I could not finish is left out.</p> <h2> robots.txt, per ho

Evidence

Which blogging platforms let AI search bots read your posts? (robots.txt and raw HTML, checked 31 Oct 2026)

Source