AI training: one robots.txt line to say no

Description and use

<p><em>First published in <a href="https://getpack.fr/en/blog/robots-txt-signal-commun-entrainement-ia/?utm_source=devto&amp;utm_medium=social" rel="noopener noreferrer">GetPack Magazine</a>, sourced guides to use AI well as a student.</em></p> <p>Until last week, telling an AI company not to train on your site meant one line per crawler: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, CCBot for Common Crawl, and another one every time a newcomer showed up. On October 30, 2026, four companies announced on the same day that they would all read the same token. Two lines a

Problem

<p><em>First published in <a href="https://getpack.fr/en/blog/robots-txt-signal-commun-entrainement-ia/?utm_source=devto&amp;utm_medium=social" rel="noopener noreferrer">GetPack Magazine</a>, sourced guides to use AI well as a student.</em></p> <p>Until last week, telling an AI company not to train on your site meant one line per crawler: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, CCBot for Common Crawl, and another one every time a newcomer showed up. On October 30, 2026, four companies announced on the same day that they would all read the same token. Two lines a

Outcome

<p><em>First published in <a href="https://getpack.fr/en/blog/robots-txt-signal-commun-entrainement-ia/?utm_source=devto&amp;utm_medium=social" rel="noopener noreferrer">GetPack Magazine</a>, sourced guides to use AI well as a student.</em></p> <p>Until last week, telling an AI company not to train on your site meant one line per crawler: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, CCBot for Common Crawl, and another one every time a newcomer showed up. On October 30, 2026, four companies announced on the same day that they would all read the same token. Two lines a

Evidence

AI training: one robots.txt line to say no

Source