Refresh website Markdown without losing unchanged pages

Описание и применение

Обновление Markdown‑документации без потери неизменных страниц: сравниваем хэши, сохраняем прежний контент, используя Apify и SQLite.

Задача

When refreshing a documentation crawl, unchanged pages return markdown: null, causing the importer to overwrite or delete existing content.

Результат

The workflow preserves unchanged pages across refreshes, writes zero bytes for unchanged content, and keeps the corpus intact while incurring the same Actor charge for each run.

Рабочий процесс

• 1. Run the first crawl with Apify’s Website Markdown Crawler and export to JSONL.
• 2. Import the JSONL into a local SQLite database (docs.sqlite3).
• 3. Run the second crawl, supplying the SQLite database as previousHashes to the runner.
• 4. The runner compares hashes; unchanged pages return markdown: null but the importer keeps existing content.
• 5. Verify that content_written is 0 for unchanged pages and that the corpus files (corpus.jsonl, hashes.json) match the first run.
• 6. Repeat the process to keep the corpus up‑to‑date without re‑importing unchanged pages.

Доказательства

The local JSONL corpus and hash file matched byte for byte after the repeat. Zero Markdown bytes describes those five response fields; the repeat still transfers metadata and still fetches the source pages.

Источник

The runner starts one Actor run at 512 MB with a 300-second timeout and a $0.02 maximum Actor charge.

Источник