Which Malaysian websites block AI crawlers? I checked 100

Описание и применение

<p><em>First published on <a href="https://yappatyih.com/writing/malaysian-websites-block-ai-crawlers/" rel="noopener noreferrer">yappatyih.com</a>.</em></p> <h2> TL;DR </h2> <p>An AI crawler block is a rule in a website's robots.txt file that tells one named bot, such as OpenAI's GPTBot, not to fetch the site's pages. On 20 October 2026 I read the robots.txt of 100 major Malaysian websites in ten sectors, from banks and telcos to news, government, shopping and universities. 13 of them block at least one AI crawler. None blocks them all, and none blocks Google's search crawler. News sites bloc

Задача

<p><em>First published on <a href="https://yappatyih.com/writing/malaysian-websites-block-ai-crawlers/" rel="noopener noreferrer">yappatyih.com</a>.</em></p> <h2> TL;DR </h2> <p>An AI crawler block is a rule in a website's robots.txt file that tells one named bot, such as OpenAI's GPTBot, not to fetch the site's pages. On 20 October 2026 I read the robots.txt of 100 major Malaysian websites in ten sectors, from banks and telcos to news, government, shopping and universities. 13 of them block at least one AI crawler. None blocks them all, and none blocks Google's search crawler. News sites bloc

Результат

<p><em>First published on <a href="https://yappatyih.com/writing/malaysian-websites-block-ai-crawlers/" rel="noopener noreferrer">yappatyih.com</a>.</em></p> <h2> TL;DR </h2> <p>An AI crawler block is a rule in a website's robots.txt file that tells one named bot, such as OpenAI's GPTBot, not to fetch the site's pages. On 20 October 2026 I read the robots.txt of 100 major Malaysian websites in ten sectors, from banks and telcos to news, government, shopping and universities. 13 of them block at least one AI crawler. None blocks them all, and none blocks Google's search crawler. News sites bloc

Доказательства

Which Malaysian websites block AI crawlers? I checked 100

Источник