Migrating a Crawl Pipeline from Firecrawl: Keep the Document Contract Stable

说明与用途

将爬虫管道从 Firecrawl 迁移到 Nstdata，同时保持稳定的内部文档契约，规范化提供者响应，持久化作业状态，并应用接受策略。

问题

Changing crawl providers requires a workflow migration that preserves the internal document contract and avoids breaking downstream RAG pipelines.

结果

The migration stabilizes the document contract, enables resumable crawls, and provides a clear acceptance policy, improving reliability and comparability across providers.

工作流

• Define an internal Document contract
• Normalize provider responses at the adapter boundary
• Persist durable crawl job state
• Implement application‑level acceptance checks
• Run the same corpus through both providers and compare metrics
• Roll out with a feature flag and rollback plan

证据

Changing crawl providers is a workflow migration, not a base-URL edit.

来源

The code is illustrative and has not been run against either paid API. Verify current API contracts before production use.

来源

Define an internal document contract

来源

Persist job state so the migration can resume

来源

Treat acceptance as a separate status

来源