Can LLMs read robots.txt? Easily. Can they write one? Only one of nine got every policy right.

说明与用途

LLM 在读取 robots.txt 时几乎无误，但在生成文件时常出现错误，导致允许不受欢迎的爬虫或过度阻止合法流量。

问题

Web crawlers must respect robots.txt rules, but many LLMs struggle to read or write correct policies, risking unintended access or over-blocking.

结果

Claude Sonnet 5.5 read all rules correctly and wrote all 40 policies without leaks. Other models achieved 92–100% reading accuracy but only 14–28% of policies were fully correct, with many probes leaking or over‑blocking.

工作流

• 1. Define reading and writing tasks per RFC 9309.
• 2. Generate 240 single‑rule test files and 240 real‑site robots.txt files.
• 3. Create 40 plain‑English policies and probe each with all crawler×URL pairs.
• 4. Run each model once per task via Kaggle’s Model Proxy.
• 5. Verify results with a reference parser (Protego) and probe checker.
• 6. Aggregate scores: read‑rules, read‑real sites, write‑policies fully right, write‑probes right, and leaks.

证据

Claude Sonnet 5.5 achieved 1.000 on read‑rules, 1.000 on read‑real sites, and 40/40 on write‑policies fully right.

来源

Python’s own urllib.robotparser gets only 70.4% of the rules cases and 73.3% of the real‑site cases right.

来源

The best writer, Claude Sonnet 5.5, passed all 40 policies with no leaks and no over‑blocks.

来源