Your site's robots.txt says AI crawlers are welcome. Your CDN might disagree.

Description and use

AgentReady checks whether AI crawlers are allowed by robots.txt but blocked by the CDN, scores accessibility, and provides actionable fixes.

Problem

Site owners often allow AI crawlers in robots.txt but their CDN blocks them, leading to inconsistent access and missed content indexing.

Outcome

AgentReady identifies mismatches between robots.txt permissions and CDN enforcement, scores crawler accessibility, and provides prioritized fixes, helping sites ensure AI crawlers can access content as intended.

Workflow

• 1. Provide a target URL to AgentReady.
• 2. AgentReady makes six real HTTP requests: one as a normal browser and five using the published user‑agents of ClaudeBot, GPTBot, OAI-SearchBot, PerplexityBot, and Google-Extended.
• 3. It fetches and parses robots.txt, llms.txt, llms-full.txt, sitemap.xml, and security.txt from the site.
• 4. It compares status codes, content lengths, and presence of text without JavaScript execution for each crawler versus a browser request.
• 5. It evaluates JSON‑LD, price markup, token ratios, and semantic structure within <main> tags.
• 6. It calculates a 0‑100 score and lists fixes ordered by recoverable points.
• 7. The result is displayed as a badge that can be embedded on the site.
• 8. Optionally, run AgentReady continuously via the Agent Web Index for ongoing monitoring.

Evidence

AgentReady: you give it a URL, it makes six real HTTP requests…

Source

nytimes.com came back a D (58) — 4 out of 5 AI crawlers got a 403 straight from the edge, and robots.txt disallows everything.

Source

stripe.com scored A+ (93).

Source