The "empty response" bug that quietly breaks a third of production scrapers (and a 20-line fix)

Description and use

The "empty response" bug causes scrapers to store empty rows after receiving a 200 with only a SPA shell. The fix uses a detector that checks visible text, shell markers, and expected fields, escalating to a headless browser only when needed.

Problem

Scrapers often receive HTTP 200 responses that contain only a client‑side rendered shell, resulting in empty database rows without errors.

Outcome

The dual‑engine fetcher keeps ~70% of requests on the fast HTTP path and only escalates to a headless browser when necessary, preventing silent empty responses and improving scraper reliability.

Workflow

• 1. Send HTTP GET with a standard User‑Agent.
• 2. If status is 200, run looks_unhydrated on the response text.
• 3. looks_unhydrated checks for shell markers, visible‑text length below a threshold, and missing expected fields.
• 4. If unhydrated, call fetch_with_browser (e.g., Playwright) to hydrate the page.
• 5. Store the hydrated HTML or log the decision engine used.

Evidence

Your job returns HTTP 200, your code saves a row, and the row is empty. No exception. No 403. Just silence — and a database full of nothing.

Source

You don't need a browser to detect this. You need to check three cheap things: Visible-text ratio, Known app-shell markers, Expected-field absence.

Source

That's the whole idea: use the cheap engine until it tells you it can't work, then escape.

Source