Username Reconnaissance with Sherlock Scraper and Python

说明与用途

使用 Apify Sherlock Scraper 进行大规模用户名侦察的自动化工作流程，包括输入验证、异步执行、预算限制和结果处理。

问题

Running large-scale username reconnaissance on Apify’s Sherlock Scraper can quickly exceed compute and proxy budgets, leading to HTTP 408 timeouts and unexpected charges.

结果

By automating input validation, async execution, and budget limits, users can perform large-scale username checks without hitting timeouts or incurring runaway costs.

工作流

• 1. Prepare a JSON payload with usernames and optional wildcards.
• 2. Validate payload size and wildcard usage to avoid excessive requests.
• 3. Use the apify-client library or curl to POST to the run endpoint asynchronously, receiving a run ID immediately.
• 4. Poll the run status endpoint or set up a webhook to detect completion.
• 5. Retrieve results from the default dataset and process the username–links pairs.
• 6. Optionally set the maxTotalChargeUsd query parameter to cap spending.
• 7. Monitor run logs for any 408 errors and adjust input or plan tier accordingly.

证据

The synchronous run endpoint returns an HTTP 408 error because it enforces a hard cap of 300 seconds on the request duration.

来源

Pass the maxTotalChargeUsd query parameter directly to the Apify run endpoint.

来源