Which blogging platforms let AI search bots read your posts? (robots.txt and raw HTML, checked 31 Oct 2026)

Описание и применение

<p><strong>Short answer:</strong> in my check, dev.to, Substack and Gumroad allow GPTBot, OAI-SearchBot, PerplexityBot and Google-Extended in robots.txt. Medium disallows GPTBot but allows the others. Indie Hackers disallows all four for the whole site. On dev.to the full post text is also in the raw HTML, so a crawler that does not run JavaScript still gets the article.</p> <p>Small sample: one day, one machine, plain <code>curl</code> with a browser user agent, no bot user agents spoofed. Part of the run was interrupted, so anything I could not finish is left out.</p> <h2> robots.txt, per ho

Задача

<p><strong>Short answer:</strong> in my check, dev.to, Substack and Gumroad allow GPTBot, OAI-SearchBot, PerplexityBot and Google-Extended in robots.txt. Medium disallows GPTBot but allows the others. Indie Hackers disallows all four for the whole site. On dev.to the full post text is also in the raw HTML, so a crawler that does not run JavaScript still gets the article.</p> <p>Small sample: one day, one machine, plain <code>curl</code> with a browser user agent, no bot user agents spoofed. Part of the run was interrupted, so anything I could not finish is left out.</p> <h2> robots.txt, per ho

Результат

<p><strong>Short answer:</strong> in my check, dev.to, Substack and Gumroad allow GPTBot, OAI-SearchBot, PerplexityBot and Google-Extended in robots.txt. Medium disallows GPTBot but allows the others. Indie Hackers disallows all four for the whole site. On dev.to the full post text is also in the raw HTML, so a crawler that does not run JavaScript still gets the article.</p> <p>Small sample: one day, one machine, plain <code>curl</code> with a browser user agent, no bot user agents spoofed. Part of the run was interrupted, so anything I could not finish is left out.</p> <h2> robots.txt, per ho

Доказательства

Which blogging platforms let AI search bots read your posts? (robots.txt and raw HTML, checked 31 Oct 2026)

Источник