Can LLMs read robots.txt? Easily. Can they write one? Only one of nine got every policy right.

Описание и применение

LLM могут читать robots.txt почти без ошибок, но при генерации файлов часто допускают ошибки, позволяющие нежелательным ботам доступ или блокируя нужный трафик.

Задача

Web crawlers must respect robots.txt rules, but many LLMs struggle to read or write correct policies, risking unintended access or over-blocking.

Результат

Claude Sonnet 5.5 read all rules correctly and wrote all 40 policies without leaks. Other models achieved 92–100% reading accuracy but only 14–28% of policies were fully correct, with many probes leaking or over‑blocking.

Рабочий процесс

• 1. Define reading and writing tasks per RFC 9309.
• 2. Generate 240 single‑rule test files and 240 real‑site robots.txt files.
• 3. Create 40 plain‑English policies and probe each with all crawler×URL pairs.
• 4. Run each model once per task via Kaggle’s Model Proxy.
• 5. Verify results with a reference parser (Protego) and probe checker.
• 6. Aggregate scores: read‑rules, read‑real sites, write‑policies fully right, write‑probes right, and leaks.

Доказательства

Claude Sonnet 5.5 achieved 1.000 on read‑rules, 1.000 on read‑real sites, and 40/40 on write‑policies fully right.

Источник

Python’s own urllib.robotparser gets only 70.4% of the rules cases and 73.3% of the real‑site cases right.

Источник

The best writer, Claude Sonnet 5.5, passed all 40 policies with no leaks and no over‑blocks.

Источник