How We Built Scraping AI: Turning 500+ Enterprise Projects Into a Self-Serve API

Description and use

PigData turned manual scraping into a scalable self‑serve API with Django, Celery, and LLMs, enabling thousands of concurrent jobs.

Problem

Managed enterprise scraping required building a new scraper from scratch for each client, causing lack of scale, high costs, long sales cycles, and siloed expertise.

Outcome

PigData transformed from a manual, per‑client scraping service to a high‑scale, self‑serve API, reducing engineering effort, enabling 10,000 concurrent jobs, and offering 200 free tokens for trial.

Workflow

• Define base URL and user instructions in InputState
• URL Finder discovers link graph
• Crawler fetches HTML via httpx or headless browser
• AI Ranker ranks pages using BM25 + vector embeddings
• LLM Extractor applies JSON schema via GPT‑4o / Gemini
• Export structured JSON/CSV to S3/MinIO

Evidence

Every project started from scratch…we were rebuilding identical parsing logic…

Source

We codified this workflow into a single state machine backed by our central InputState model

Source

With dozens of Celery workers processing URLs concurrently…multiple workers attempted to update InputState status simultaneously

Source