Which Malaysian websites block AI crawlers? I checked 100

Description and use

<p><em>First published on <a href="https://yappatyih.com/writing/malaysian-websites-block-ai-crawlers/" rel="noopener noreferrer">yappatyih.com</a>.</em></p> <h2> TL;DR </h2> <p>An AI crawler block is a rule in a website's robots.txt file that tells one named bot, such as OpenAI's GPTBot, not to fetch the site's pages. On 20 October 2026 I read the robots.txt of 100 major Malaysian websites in ten sectors, from banks and telcos to news, government, shopping and universities. 13 of them block at least one AI crawler. None blocks them all, and none blocks Google's search crawler. News sites bloc

Problem

<p><em>First published on <a href="https://yappatyih.com/writing/malaysian-websites-block-ai-crawlers/" rel="noopener noreferrer">yappatyih.com</a>.</em></p> <h2> TL;DR </h2> <p>An AI crawler block is a rule in a website's robots.txt file that tells one named bot, such as OpenAI's GPTBot, not to fetch the site's pages. On 20 October 2026 I read the robots.txt of 100 major Malaysian websites in ten sectors, from banks and telcos to news, government, shopping and universities. 13 of them block at least one AI crawler. None blocks them all, and none blocks Google's search crawler. News sites bloc

Outcome

<p><em>First published on <a href="https://yappatyih.com/writing/malaysian-websites-block-ai-crawlers/" rel="noopener noreferrer">yappatyih.com</a>.</em></p> <h2> TL;DR </h2> <p>An AI crawler block is a rule in a website's robots.txt file that tells one named bot, such as OpenAI's GPTBot, not to fetch the site's pages. On 20 October 2026 I read the robots.txt of 100 major Malaysian websites in ten sectors, from banks and telcos to news, government, shopping and universities. 13 of them block at least one AI crawler. None blocks them all, and none blocks Google's search crawler. News sites bloc

Evidence

Which Malaysian websites block AI crawlers? I checked 100

Source