Refresh website Markdown without losing unchanged pages

说明与用途

使用 Apify 和 SQLite，刷新 Markdown 文档而不丢失未更改的页面：比较哈希值，保留现有内容。

问题

When refreshing a documentation crawl, unchanged pages return markdown: null, causing the importer to overwrite or delete existing content.

结果

The workflow preserves unchanged pages across refreshes, writes zero bytes for unchanged content, and keeps the corpus intact while incurring the same Actor charge for each run.

工作流

• 1. Run the first crawl with Apify’s Website Markdown Crawler and export to JSONL.
• 2. Import the JSONL into a local SQLite database (docs.sqlite3).
• 3. Run the second crawl, supplying the SQLite database as previousHashes to the runner.
• 4. The runner compares hashes; unchanged pages return markdown: null but the importer keeps existing content.
• 5. Verify that content_written is 0 for unchanged pages and that the corpus files (corpus.jsonl, hashes.json) match the first run.
• 6. Repeat the process to keep the corpus up‑to‑date without re‑importing unchanged pages.

证据

The local JSONL corpus and hash file matched byte for byte after the repeat. Zero Markdown bytes describes those five response fields; the repeat still transfers metadata and still fetches the source pages.

来源

The runner starts one Actor run at 512 MB with a 300-second timeout and a $0.02 maximum Actor charge.

来源