How We Built Scraping AI: Turning 500+ Enterprise Projects Into a Self-Serve API

Описание и применение

PigData превратил ручную службу скрапинга в масштабируемый API, используя Django, Celery и LLM, что позволило обрабатывать тысячи задач одновременно.

Задача

Managed enterprise scraping required building a new scraper from scratch for each client, causing lack of scale, high costs, long sales cycles, and siloed expertise.

Результат

PigData transformed from a manual, per‑client scraping service to a high‑scale, self‑serve API, reducing engineering effort, enabling 10,000 concurrent jobs, and offering 200 free tokens for trial.

Рабочий процесс

• Define base URL and user instructions in InputState
• URL Finder discovers link graph
• Crawler fetches HTML via httpx or headless browser
• AI Ranker ranks pages using BM25 + vector embeddings
• LLM Extractor applies JSON schema via GPT‑4o / Gemini
• Export structured JSON/CSV to S3/MinIO

Доказательства

Every project started from scratch…we were rebuilding identical parsing logic…

Источник

We codified this workflow into a single state machine backed by our central InputState model

Источник

With dozens of Celery workers processing URLs concurrently…multiple workers attempted to update InputState status simultaneously

Источник