Can LLMs read robots.txt? Easily. Can they write one? Only one of nine got every policy right.

Description and use

LLMs read robots.txt with high accuracy, yet many fail to write correct policies, leading to permissive or overly restrictive files.

Problem

Web crawlers must respect robots.txt rules, but many LLMs struggle to read or write correct policies, risking unintended access or over-blocking.

Outcome

Claude Sonnet 5.5 read all rules correctly and wrote all 40 policies without leaks. Other models achieved 92–100% reading accuracy but only 14–28% of policies were fully correct, with many probes leaking or over‑blocking.

Workflow

• 1. Define reading and writing tasks per RFC 9309.
• 2. Generate 240 single‑rule test files and 240 real‑site robots.txt files.
• 3. Create 40 plain‑English policies and probe each with all crawler×URL pairs.
• 4. Run each model once per task via Kaggle’s Model Proxy.
• 5. Verify results with a reference parser (Protego) and probe checker.
• 6. Aggregate scores: read‑rules, read‑real sites, write‑policies fully right, write‑probes right, and leaks.

Evidence

Claude Sonnet 5.5 achieved 1.000 on read‑rules, 1.000 on read‑real sites, and 40/40 on write‑policies fully right.

Source

Python’s own urllib.robotparser gets only 70.4% of the rules cases and 73.3% of the real‑site cases right.

Source

The best writer, Claude Sonnet 5.5, passed all 40 policies with no leaks and no over‑blocks.

Source