How We Built Scraping AI: Turning 500+ Enterprise Projects Into a Self-Serve API

说明与用途

PigData 将手动抓取转变为可扩展的自助 API，使用 Django、Celery 和 LLM，支持数千并发任务。

问题

Managed enterprise scraping required building a new scraper from scratch for each client, causing lack of scale, high costs, long sales cycles, and siloed expertise.

结果

PigData transformed from a manual, per‑client scraping service to a high‑scale, self‑serve API, reducing engineering effort, enabling 10,000 concurrent jobs, and offering 200 free tokens for trial.

工作流

• Define base URL and user instructions in InputState
• URL Finder discovers link graph
• Crawler fetches HTML via httpx or headless browser
• AI Ranker ranks pages using BM25 + vector embeddings
• LLM Extractor applies JSON schema via GPT‑4o / Gemini
• Export structured JSON/CSV to S3/MinIO

证据

Every project started from scratch…we were rebuilding identical parsing logic…

来源

We codified this workflow into a single state machine backed by our central InputState model

来源

With dozens of Celery workers processing URLs concurrently…multiple workers attempted to update InputState status simultaneously

来源