Eleven scrapers in three days: what broke and what a row actually costs

Description and use

The experiment building 11 pay‑per‑row scrapers on Apify uncovered billing bugs, data loss, and high per‑row costs for browser‑based sources. Fixes included reordering charge‑push, updating seen‑set logic, handling API quirks and terms of service, and measuring real per‑row costs.

Problem

Building pay‑per‑event scrapers on Apify Store quickly revealed billing bugs, data loss, and high per‑row costs, especially for browser‑based sources.

Outcome

The experiment produced 11 scrapers, identified critical billing and data‑loss bugs, quantified per‑row costs (up to $0.0046 for browser‑based sources), and highlighted the importance of respecting API terms and distribution strategies.

Workflow

• 1. Fetch detail page with retry logic.
• 2. Charge event after successful fetch.
• 3. Push data to output.
• 4. Iterate over list items, push and charge each, then mark as delivered only after push.
• 5. Close source after all items processed.
• 6. Handle API quirks per source (e.g., Substack search, Meetup location, Workday POST, Himalayas pagination, keyword regex).
• 7. Respect terms of service by excluding disallowed sources or adding attribution.
• 8. Monitor per‑row cost metrics and adjust pricing.
• 9. Publish scrapers to Apify Store and track visibility and user acquisition.

Evidence

Pay per event means you call Actor.charge({ eventName: 'result' }) when you hand the buyer something.

Source

The buyer is charged, then the page loads. In a test run the browser died between those two lines, and the run billed for a detail it never delivered.

Source

All at 256 MB, no proxy unless stated.

Source

Substack returns nothing to Node fetch.

Source