Your robots.txt might not say what you think it says

Description and use

A lightweight CLI checks the live robots.txt served to crawlers, detects CDN‑injected rewrites, and integrates with CI pipelines to enforce correct crawl permissions.

Problem

Site owners often assume the robots.txt file they host in their repo is the same file served to crawlers, but edge services like Cloudflare can rewrite it on the fly, leading to discrepancies that go unnoticed.

Outcome

The tool quickly confirms whether the live robots.txt matches the repository version and flags any CDN‑injected changes, enabling developers to catch misconfigurations before they affect crawlers.

Workflow

• 1. Install the tool via npx: npx github:janibert1/robots-check yourdomain.com
• 2. The tool fetches https://yourdomain.com/robots.txt and displays its contents.
• 3. It highlights any Cloudflare‑injected rules if present.
• 4. Use the --ci flag in CI pipelines to exit with code 1 when the site is disallowed for all crawlers.
• 5. Use --fail-on-cloudflare-managed to fail the pipeline if any Cloudflare rewrite is detected.
• 6. Optionally add the GitHub Action janibert1/robots-check@v1 to a workflow for automated checks.

Evidence

Zero dependencies, just fetches the live file and tells you what's there, including whether anything looks CDN-injected.

Source

There's also a --ci flag for build pipelines (exits 1 if the whole site is accidentally disallowed) and a stricter --fail-on-cloudflare-managed flag if you want CI to fail on any Cloudflare-injected rewrite specifically, plus a matching GitHub Action (janibert1/robots-check@v1).

Source