I built CrawlGuard: A free tool to block aggressive AI crawlers and protect server bandwidth

Описание и применение

CrawlGuard — бесплатный инструмент, генерирующий правила для блокировки агрессивных AI‑скрейперов, экспортирующий их в различные форматы и предоставляющий интерактивный тестер.

Задача

Aggressive AI web crawlers such as Bytespider, GPTBot, and ClaudeBot were scraping servers without respecting crawl delays or referral traffic, causing bandwidth strain.

Результат

CrawlGuard provides a 1‑click solution to generate optimal blocking rules for aggressive AI crawlers, supports multiple export formats, and includes a live tester, all free and client‑side.

Рабочий процесс

• Identify desired protection preset (e.g., Block All AI Training).
• Click the preset to generate rules.
• Export rules in desired format (robots.txt, .htaccess, Nginx .conf, Cloudflare WAF).
• Apply exported rules to server or reverse proxy.
• Use the live tester to verify URL paths are correctly allowed or blocked.

Доказательства

Manually looking up user‑agents and writing complex rules for both robots.txt and reverse proxies was getting tedious, so I built CrawlGuard.

Источник

Generate optimal rules (e.g., "Block All AI Training", "E-Commerce Protection", or "Balanced Modern Web").

Источник

Spits out valid RFC 9309 robots.txt, Apache .htaccess, Nginx .conf, and Cloudflare WAF rules for bots that ignore robots directives.

Источник

Includes an interactive simulator at the bottom to test if your URL paths are allowed or blocked.

Источник