We read the robots.txt of 2,347 MCP hosts: Python 3.12's parser gets 252 endpoints wrong

Description and use

<p>Two crawlers that log MCP servers compared notes on 1 November, and one server didn't add up. <a href="http://www.apier.no" rel="noopener noreferrer">www.apier.no</a> had changed its tool definitions twice in a day. Our log, HISTOR, recorded both changes. ScraperWatch, which honours robots.txt, skipped the server entirely. Its robots.txt looked clear enough (an excerpt; the group has twelve more Allow lines):<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>User-agent: * Allow: /api/mcp$ Disallow: /api/ </code></pre> </div> <p>The site even explains

Problem

<p>Two crawlers that log MCP servers compared notes on 1 November, and one server didn't add up. <a href="http://www.apier.no" rel="noopener noreferrer">www.apier.no</a> had changed its tool definitions twice in a day. Our log, HISTOR, recorded both changes. ScraperWatch, which honours robots.txt, skipped the server entirely. Its robots.txt looked clear enough (an excerpt; the group has twelve more Allow lines):<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>User-agent: * Allow: /api/mcp$ Disallow: /api/ </code></pre> </div> <p>The site even explains

Outcome

<p>Two crawlers that log MCP servers compared notes on 1 November, and one server didn't add up. <a href="http://www.apier.no" rel="noopener noreferrer">www.apier.no</a> had changed its tool definitions twice in a day. Our log, HISTOR, recorded both changes. ScraperWatch, which honours robots.txt, skipped the server entirely. Its robots.txt looked clear enough (an excerpt; the group has twelve more Allow lines):<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>User-agent: * Allow: /api/mcp$ Disallow: /api/ </code></pre> </div> <p>The site even explains

Evidence

We read the robots.txt of 2,347 MCP hosts: Python 3.12's parser gets 252 endpoints wrong

Source