Stop Scraping DOMs When JSON APIs Are Hiding in Plain Sight: Benchmarks with Async Python & Playwright

Описание и применение

Переключение от парсинга DOM в headless‑браузере к прямому запросу JSON‑API с помощью async httpx повышает пропускную способность более чем в 50‑раз и снижает потребление памяти почти до нуля, позволяя выполнять задачи в серверless‑контейнерах за доли цента.

Задача

Engineering teams spend hundreds of dollars monthly running headless Chromium clusters to scrape product prices, real estate listings, or B2B directory profiles, despite the server already providing clean JSON APIs.

Результат

Direct async JSON API scraping improves throughput by 51.2×, cuts memory usage by 94 %, and enables sub‑cent cost execution in lightweight serverless containers.

Рабочий процесс

• Inspect target domain for Schema.org JSON‑LD or sitemaps (Tier 1).
• If present, extract entities with a BeautifulSoup parser.
• If not, look for SSR hydration payloads such as __NEXT_DATA__ or __NUXT__ (Tier 2).
• If still missing, run a 30‑line Playwright XHR sniffer to discover JSON endpoints (Tier 3).
• Once endpoints are identified, use async httpx to fetch paginated JSON directly (Tier 4).

Доказательства

Every week I audit broken data pipelines where an engineering team is spending hundreds of dollars a month running containerized headless Chromium clusters just to extract product prices, real estate listings, or B2B directory profiles.

Источник

Switching from DOM-rendered Chromium scraping to direct async httpx JSON pagination improves throughput by 51.2x while cutting memory usage by 94%.

Источник

The 4‑Tier Extraction Waterfall I use across commercial scrapers and client pipelines, backed by benchmarks across 1,000 paginated catalog requests and complete Python 3.11 implementations.

Источник