Eleven scrapers in three days: what broke and what a row actually costs

说明与用途

在 Apify 上构建 11 个按行计费的抓取器实验中，发现了计费错误、数据丢失以及浏览器源的高成本问题。通过重新排序 charge‑push、更新 seen‑set 逻辑、处理 API 细节和服务条款，并测量实际每行成本来解决这些问题。

问题

Building pay‑per‑event scrapers on Apify Store quickly revealed billing bugs, data loss, and high per‑row costs, especially for browser‑based sources.

结果

The experiment produced 11 scrapers, identified critical billing and data‑loss bugs, quantified per‑row costs (up to $0.0046 for browser‑based sources), and highlighted the importance of respecting API terms and distribution strategies.

工作流

• 1. Fetch detail page with retry logic.
• 2. Charge event after successful fetch.
• 3. Push data to output.
• 4. Iterate over list items, push and charge each, then mark as delivered only after push.
• 5. Close source after all items processed.
• 6. Handle API quirks per source (e.g., Substack search, Meetup location, Workday POST, Himalayas pagination, keyword regex).
• 7. Respect terms of service by excluding disallowed sources or adding attribution.
• 8. Monitor per‑row cost metrics and adjust pricing.
• 9. Publish scrapers to Apify Store and track visibility and user acquisition.

证据

Pay per event means you call Actor.charge({ eventName: 'result' }) when you hand the buyer something.

来源

The buyer is charged, then the page loads. In a test run the browser died between those two lines, and the run billed for a detail it never delivered.

来源

All at 256 MB, no proxy unless stated.

来源

Substack returns nothing to Node fetch.

来源