Username Reconnaissance with Sherlock Scraper and Python

Описание и применение

Автоматизированный рабочий процесс для массового поиска имен пользователей с помощью Apify Sherlock Scraper, включая валидацию входных данных, асинхронный запуск, ограничение бюджета и обработку результатов.

Задача

Running large-scale username reconnaissance on Apify’s Sherlock Scraper can quickly exceed compute and proxy budgets, leading to HTTP 408 timeouts and unexpected charges.

Результат

By automating input validation, async execution, and budget limits, users can perform large-scale username checks without hitting timeouts or incurring runaway costs.

Рабочий процесс

• 1. Prepare a JSON payload with usernames and optional wildcards.
• 2. Validate payload size and wildcard usage to avoid excessive requests.
• 3. Use the apify-client library or curl to POST to the run endpoint asynchronously, receiving a run ID immediately.
• 4. Poll the run status endpoint or set up a webhook to detect completion.
• 5. Retrieve results from the default dataset and process the username–links pairs.
• 6. Optionally set the maxTotalChargeUsd query parameter to cap spending.
• 7. Monitor run logs for any 408 errors and adjust input or plan tier accordingly.

Доказательства

The synchronous run endpoint returns an HTTP 408 error because it enforces a hard cap of 300 seconds on the request duration.

Источник

Pass the maxTotalChargeUsd query parameter directly to the Apify run endpoint.

Источник