Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

Your site's robots.txt says AI crawlers are welcome. Your CDN might disagree.

Описание и применение

Инструмент AgentReady проверяет, разрешают ли robots.txt AI‑краулеры, но блокирует ли CDN, и выдаёт оценку доступности с рекомендациями.

Задача

Site owners often allow AI crawlers in robots.txt but their CDN blocks them, leading to inconsistent access and missed content indexing.

Результат

AgentReady identifies mismatches between robots.txt permissions and CDN enforcement, scores crawler accessibility, and provides prioritized fixes, helping sites ensure AI crawlers can access content as intended.

Рабочий процесс

  • 1. Provide a target URL to AgentReady.
  • 2. AgentReady makes six real HTTP requests: one as a normal browser and five using the published user‑agents of ClaudeBot, GPTBot, OAI-SearchBot, PerplexityBot, and Google-Extended.
  • 3. It fetches and parses robots.txt, llms.txt, llms-full.txt, sitemap.xml, and security.txt from the site.
  • 4. It compares status codes, content lengths, and presence of text without JavaScript execution for each crawler versus a browser request.
  • 5. It evaluates JSON‑LD, price markup, token ratios, and semantic structure within <main> tags.
  • 6. It calculates a 0‑100 score and lists fixes ordered by recoverable points.
  • 7. The result is displayed as a badge that can be embedded on the site.
  • 8. Optionally, run AgentReady continuously via the Agent Web Index for ongoing monitoring.

Доказательства

AgentReady: you give it a URL, it makes six real HTTP requests…

Источник

nytimes.com came back a D (58) — 4 out of 5 AI crawlers got a 403 straight from the edge, and robots.txt disallows everything.

Источник

stripe.com scored A+ (93).

Источник

  • Дата записи: 2026-09-20T18:00:33.079Z