The "empty response" bug that quietly breaks a third of production scrapers (and a 20-line fix)

说明与用途

"空响应"错误导致爬虫在收到仅包含 SPA Shell 的 200 响应后存储空行。解决方案是使用检测器检查可见文本、Shell 标记和预期字段，必要时才升级到无头浏览器。

问题

Scrapers often receive HTTP 200 responses that contain only a client‑side rendered shell, resulting in empty database rows without errors.

结果

The dual‑engine fetcher keeps ~70% of requests on the fast HTTP path and only escalates to a headless browser when necessary, preventing silent empty responses and improving scraper reliability.

工作流

• 1. Send HTTP GET with a standard User‑Agent.
• 2. If status is 200, run looks_unhydrated on the response text.
• 3. looks_unhydrated checks for shell markers, visible‑text length below a threshold, and missing expected fields.
• 4. If unhydrated, call fetch_with_browser (e.g., Playwright) to hydrate the page.
• 5. Store the hydrated HTML or log the decision engine used.

证据

Your job returns HTTP 200, your code saves a row, and the row is empty. No exception. No 403. Just silence — and a database full of nothing.

来源

You don't need a browser to detect this. You need to check three cheap things: Visible-text ratio, Known app-shell markers, Expected-field absence.

来源

That's the whole idea: use the cheap engine until it tells you it can't work, then escape.

来源