Your robots.txt might not say what you think it says

说明与用途

轻量级 CLI 工具可检查网站实际提供的 robots.txt，检测 CDN 注入的规则，并可集成到 CI 流水线中，确保爬虫权限正确。

问题

Site owners often assume the robots.txt file they host in their repo is the same file served to crawlers, but edge services like Cloudflare can rewrite it on the fly, leading to discrepancies that go unnoticed.

结果

The tool quickly confirms whether the live robots.txt matches the repository version and flags any CDN‑injected changes, enabling developers to catch misconfigurations before they affect crawlers.

工作流

• 1. Install the tool via npx: npx github:janibert1/robots-check yourdomain.com
• 2. The tool fetches https://yourdomain.com/robots.txt and displays its contents.
• 3. It highlights any Cloudflare‑injected rules if present.
• 4. Use the --ci flag in CI pipelines to exit with code 1 when the site is disallowed for all crawlers.
• 5. Use --fail-on-cloudflare-managed to fail the pipeline if any Cloudflare rewrite is detected.
• 6. Optionally add the GitHub Action janibert1/robots-check@v1 to a workflow for automated checks.

证据

Zero dependencies, just fetches the live file and tells you what's there, including whether anything looks CDN-injected.

来源

There's also a --ci flag for build pipelines (exits 1 if the whole site is accidentally disallowed) and a stricter --fail-on-cloudflare-managed flag if you want CI to fail on any Cloudflare-injected rewrite specifically, plus a matching GitHub Action (janibert1/robots-check@v1).

来源