Eleven scrapers in three days: what broke and what a row actually costs

Описание и применение

Эксперимент по созданию 11 платных скриптов на Apify выявил ошибки биллинга, потерю данных и высокие расходы на браузерные источники. Были исправлены порядок вызовов charge‑push, обновлена логика seen‑set, учтены особенности API и условия использования, а также измерены реальные затраты на строку.

Задача

Building pay‑per‑event scrapers on Apify Store quickly revealed billing bugs, data loss, and high per‑row costs, especially for browser‑based sources.

Результат

The experiment produced 11 scrapers, identified critical billing and data‑loss bugs, quantified per‑row costs (up to $0.0046 for browser‑based sources), and highlighted the importance of respecting API terms and distribution strategies.

Рабочий процесс

• 1. Fetch detail page with retry logic.
• 2. Charge event after successful fetch.
• 3. Push data to output.
• 4. Iterate over list items, push and charge each, then mark as delivered only after push.
• 5. Close source after all items processed.
• 6. Handle API quirks per source (e.g., Substack search, Meetup location, Workday POST, Himalayas pagination, keyword regex).
• 7. Respect terms of service by excluding disallowed sources or adding attribution.
• 8. Monitor per‑row cost metrics and adjust pricing.
• 9. Publish scrapers to Apify Store and track visibility and user acquisition.

Доказательства

Pay per event means you call Actor.charge({ eventName: 'result' }) when you hand the buyer something.

Источник

The buyer is charged, then the page loads. In a test run the browser died between those two lines, and the run billed for a detail it never delivered.

Источник

All at 256 MB, no proxy unless stated.

Источник

Substack returns nothing to Node fetch.

Источник