The "empty response" bug that quietly breaks a third of production scrapers (and a 20-line fix)

Описание и применение

Баг «пустого ответа» приводит к тому, что скрейперы получают HTTP‑200 с минимальной оболочкой SPA, создавая пустые записи в базе. Решение — детектор, проверяющий видимый текст, маркеры оболочки и наличие ожидаемых полей, после чего при необходимости переключается на браузер Playwright.

Задача

Scrapers often receive HTTP 200 responses that contain only a client‑side rendered shell, resulting in empty database rows without errors.

Результат

The dual‑engine fetcher keeps ~70% of requests on the fast HTTP path and only escalates to a headless browser when necessary, preventing silent empty responses and improving scraper reliability.

Рабочий процесс

• 1. Send HTTP GET with a standard User‑Agent.
• 2. If status is 200, run looks_unhydrated on the response text.
• 3. looks_unhydrated checks for shell markers, visible‑text length below a threshold, and missing expected fields.
• 4. If unhydrated, call fetch_with_browser (e.g., Playwright) to hydrate the page.
• 5. Store the hydrated HTML or log the decision engine used.

Доказательства

Your job returns HTTP 200, your code saves a row, and the row is empty. No exception. No 403. Just silence — and a database full of nothing.

Источник

You don't need a browser to detect this. You need to check three cheap things: Visible-text ratio, Known app-shell markers, Expected-field absence.

Источник

That's the whole idea: use the cheap engine until it tells you it can't work, then escape.

Источник