Cloudflare starts blocking agents by default on September 15

Описание и применение

Новый режим Cloudflare блокирует автоматические запросы (агенты и обучение) по умолчанию, что приводит к тихим сбоям в скриптах и AI‑агентах. Необходимо тестировать ответы до и после 15 сентября, выявлять сигнатуры блокировки и добавлять защиту в код.

Задача

Scrapers and AI agents that fetch web pages will be blocked by Cloudflare’s new default setting, causing silent failures and revenue loss for developers who rely on automated data extraction.

Результат

The new Cloudflare setting forces AI agents and scrapers to detect and handle blocked responses, preventing silent failures and ensuring compliance with the new policy.

Рабочий процесс

• 1. Identify a Cloudflare‑protected site that shows ads.
• 2. Perform three fetches today: (a) plain client, (b) client with a custom user‑agent string, (c) client via your agent framework.
• 3. Record status code, response headers, and first 500 bytes of the body for each fetch.
• 4. Repeat steps 2–3 on September 15 and 16.
• 5. Compare responses for changes such as cf‑mitigated header, 403/402 status, or challenge content.
• 6. Implement a guard in your agent to detect these signatures and refuse to summarize blocked pages.

Доказательства

Every bot that hits a site behind them now gets sorted into one of three buckets. Search stays allowed. Agent and training get blocked by default on any page that shows ads.

Источник

The model calls the fetch tool, the tool gets a challenge page or a 403, and the model summarizes the challenge page like it is content.

Источник