I built CrawlGuard: A free tool to block aggressive AI crawlers and protect server bandwidth

说明与用途

CrawlGuard 是一款免费工具，可生成阻止激进 AI 爬虫的规则，支持多种导出格式，并提供交互式测试器。

问题

Aggressive AI web crawlers such as Bytespider, GPTBot, and ClaudeBot were scraping servers without respecting crawl delays or referral traffic, causing bandwidth strain.

结果

CrawlGuard provides a 1‑click solution to generate optimal blocking rules for aggressive AI crawlers, supports multiple export formats, and includes a live tester, all free and client‑side.

工作流

• Identify desired protection preset (e.g., Block All AI Training).
• Click the preset to generate rules.
• Export rules in desired format (robots.txt, .htaccess, Nginx .conf, Cloudflare WAF).
• Apply exported rules to server or reverse proxy.
• Use the live tester to verify URL paths are correctly allowed or blocked.

证据

Manually looking up user‑agents and writing complex rules for both robots.txt and reverse proxies was getting tedious, so I built CrawlGuard.

来源

Generate optimal rules (e.g., "Block All AI Training", "E-Commerce Protection", or "Balanced Modern Web").

来源

Spits out valid RFC 9309 robots.txt, Apache .htaccess, Nginx .conf, and Cloudflare WAF rules for bots that ignore robots directives.

来源

Includes an interactive simulator at the bottom to test if your URL paths are allowed or blocked.

来源