Why Preventing Duplicate SEMrush Scraper Output Needs Named Storage

Описание и применение

Для предотвращения повторной обработки данных из SEMrush Scraper необходимо хранить список уже обработанных доменов в постоянном хранилище, например, в именованном Key‑Value Store Apify. Это позволяет последующим запускам пропускать уже обработанные домены, экономя ресурсы и обеспечивая корректность downstream‑процессов.

Задача

Duplicate processing of SEMrush Free Website Stats Scraper output causes wasted resources and inaccurate downstream data.

Результат

By persisting processed domain identifiers, the pipeline avoids duplicate scraping and processing, saving API calls and ensuring downstream systems receive only new data.

Рабочий процесс

• 1. Load processed domains from a named KV store.
• 2. Filter the input domain list against the loaded set.
• 3. Run the SEMrush scraper actor with the filtered list.
• 4. After the run, fetch the dataset items from the actor’s default dataset.
• 5. Process each item and collect newly processed domains.
• 6. Save the updated set of processed domains back to the KV store.

Доказательства

Maintain a list of all domains for which you have successfully retrieved and processed SEMrush stats, storing this information in persistent storage.

Источник

Named Key-Value Stores, unlike unnamed storages which expire, are persistent and provide a reliable place to store your processed domain identifiers and their scrapedAt timestamps.

Источник