Your site's robots.txt says AI crawlers are welcome. Your CDN might disagree.
说明与用途
AgentReady 检查 AI 爬虫是否被 robots.txt 允许但被 CDN 阻止,给出可访问性评分并提供可操作的修复建议。
问题
Site owners often allow AI crawlers in robots.txt but their CDN blocks them, leading to inconsistent access and missed content indexing.
结果
AgentReady identifies mismatches between robots.txt permissions and CDN enforcement, scores crawler accessibility, and provides prioritized fixes, helping sites ensure AI crawlers can access content as intended.
工作流
- 1. Provide a target URL to AgentReady.
- 2. AgentReady makes six real HTTP requests: one as a normal browser and five using the published user‑agents of ClaudeBot, GPTBot, OAI-SearchBot, PerplexityBot, and Google-Extended.
- 3. It fetches and parses robots.txt, llms.txt, llms-full.txt, sitemap.xml, and security.txt from the site.
- 4. It compares status codes, content lengths, and presence of text without JavaScript execution for each crawler versus a browser request.
- 5. It evaluates JSON‑LD, price markup, token ratios, and semantic structure within
<main>tags. - 6. It calculates a 0‑100 score and lists fixes ordered by recoverable points.
- 7. The result is displayed as a badge that can be embedded on the site.
- 8. Optionally, run AgentReady continuously via the Agent Web Index for ongoing monitoring.
证据
AgentReady: you give it a URL, it makes six real HTTP requests…
nytimes.com came back a D (58) — 4 out of 5 AI crawlers got a 403 straight from the edge, and robots.txt disallows everything.
stripe.com scored A+ (93).
- 记录日期: 2026-09-20T18:00:33.079Z