Your robots.txt might not say what you think it says

Описание и применение

Инструмент позволяет быстро проверить, соответствует ли robots.txt, обслуживаемый сайтом, тому, что хранится в репозитории, и выявить изменения, внесённые CDN, например Cloudflare.

Задача

Site owners often assume the robots.txt file they host in their repo is the same file served to crawlers, but edge services like Cloudflare can rewrite it on the fly, leading to discrepancies that go unnoticed.

Результат

The tool quickly confirms whether the live robots.txt matches the repository version and flags any CDN‑injected changes, enabling developers to catch misconfigurations before they affect crawlers.

Рабочий процесс

• 1. Install the tool via npx: npx github:janibert1/robots-check yourdomain.com
• 2. The tool fetches https://yourdomain.com/robots.txt and displays its contents.
• 3. It highlights any Cloudflare‑injected rules if present.
• 4. Use the --ci flag in CI pipelines to exit with code 1 when the site is disallowed for all crawlers.
• 5. Use --fail-on-cloudflare-managed to fail the pipeline if any Cloudflare rewrite is detected.
• 6. Optionally add the GitHub Action janibert1/robots-check@v1 to a workflow for automated checks.

Доказательства

Zero dependencies, just fetches the live file and tells you what's there, including whether anything looks CDN-injected.

Источник

There's also a --ci flag for build pipelines (exits 1 if the whole site is accidentally disallowed) and a stricter --fail-on-cloudflare-managed flag if you want CI to fail on any Cloudflare-injected rewrite specifically, plus a matching GitHub Action (janibert1/robots-check@v1).

Источник