Cloudflare starts blocking agents by default on September 15

Description and use

Cloudflare’s new default blocks agent traffic, causing silent failures in scrapers and AI agents. Test responses before and after September 15, detect block signatures, and add guard logic to your code.

Problem

Scrapers and AI agents that fetch web pages will be blocked by Cloudflare’s new default setting, causing silent failures and revenue loss for developers who rely on automated data extraction.

Outcome

The new Cloudflare setting forces AI agents and scrapers to detect and handle blocked responses, preventing silent failures and ensuring compliance with the new policy.

Workflow

• 1. Identify a Cloudflare‑protected site that shows ads.
• 2. Perform three fetches today: (a) plain client, (b) client with a custom user‑agent string, (c) client via your agent framework.
• 3. Record status code, response headers, and first 500 bytes of the body for each fetch.
• 4. Repeat steps 2–3 on September 15 and 16.
• 5. Compare responses for changes such as cf‑mitigated header, 403/402 status, or challenge content.
• 6. Implement a guard in your agent to detect these signatures and refuse to summarize blocked pages.

Evidence

Every bot that hits a site behind them now gets sorted into one of three buckets. Search stays allowed. Agent and training get blocked by default on any page that shows ads.

Source

The model calls the fetch tool, the tool gets a challenge page or a 403, and the model summarizes the challenge page like it is content.

Source