I built CrawlGuard: A free tool to block aggressive AI crawlers and protect server bandwidth

Description and use

CrawlGuard is a free tool that generates rules to block aggressive AI crawlers, exports them in multiple formats, and offers an interactive tester.

Problem

Aggressive AI web crawlers such as Bytespider, GPTBot, and ClaudeBot were scraping servers without respecting crawl delays or referral traffic, causing bandwidth strain.

Outcome

CrawlGuard provides a 1‑click solution to generate optimal blocking rules for aggressive AI crawlers, supports multiple export formats, and includes a live tester, all free and client‑side.

Workflow

• Identify desired protection preset (e.g., Block All AI Training).
• Click the preset to generate rules.
• Export rules in desired format (robots.txt, .htaccess, Nginx .conf, Cloudflare WAF).
• Apply exported rules to server or reverse proxy.
• Use the live tester to verify URL paths are correctly allowed or blocked.

Evidence

Manually looking up user‑agents and writing complex rules for both robots.txt and reverse proxies was getting tedious, so I built CrawlGuard.

Source

Generate optimal rules (e.g., "Block All AI Training", "E-Commerce Protection", or "Balanced Modern Web").

Source

Spits out valid RFC 9309 robots.txt, Apache .htaccess, Nginx .conf, and Cloudflare WAF rules for bots that ignore robots directives.

Source

Includes an interactive simulator at the bottom to test if your URL paths are allowed or blocked.

Source