Why your scraper gets a 403 on Cloudflare Turnstile (and the token-injection fix)

说明与用途

在 Cloudflare Turnstile 触发 403 的情况下，抓取脚本可以通过住宅 IP（使用浏览器或解码 API）获取有效令牌，并将其注入请求，从而绕过阻止。

问题

Scrapers hit 403 errors or endless Turnstile challenges when accessing sites protected by Cloudflare Turnstile, despite working HTML requests.

结果

Scrapers can reliably bypass Cloudflare Turnstile 403s by generating a valid token from a residential IP and injecting it into their requests, reducing reliance on full browser sessions and improving throughput.

工作流

• 1. Inspect target page to locate the Turnstile sitekey (data-sitekey attribute or JS render call).
• 2. Obtain a Turnstile token: either run a real browser (Playwright/Selenium) through a residential proxy and read the cf-turnstile-response value, or call a solving API with the target URL and sitekey.
• 3. If using a solving API, send a POST to the API endpoint with API key, task_type, url, and sitekey; receive the token in the response.
• 4. Attach the token to the scraper’s HTTP request payload (e.g., as cf-turnstile-response) and submit before the token expires (~300 s).
• 5. Handle common failure cases: ensure token is fresh, single-use, correct sitekey, and matching solving IP if required.
• 6. If token alone fails, consider full browser session with proper fingerprint and session cookies.
• 7. Monitor for changes in Turnstile challenge variants and update browser or API usage accordingly.

证据

Cloudflare Turnstile scores the browser, and separately Cloudflare scores the connection. A 403 usually means you failed one of these: Fingerprint… IP reputation.

来源

The widget produces a token — cf-turnstile-response — and on most deployments that token is validated server‑side against the sitekey and the hostname, not against the IP that produced it.

来源

Call a solving API. You hand it the URL and sitekey; it runs the browser farm on residential IPs and hands back a token.

来源