Migrating a Crawl Pipeline from Firecrawl: Keep the Document Contract Stable

Описание и применение

Перенос пайплайна обхода с Firecrawl на Nstdata с сохранением стабильного внутреннего контракта документа, нормализацией ответов, сохранением состояния задачи и применением политики принятия.

Задача

Changing crawl providers requires a workflow migration that preserves the internal document contract and avoids breaking downstream RAG pipelines.

Результат

The migration stabilizes the document contract, enables resumable crawls, and provides a clear acceptance policy, improving reliability and comparability across providers.

Рабочий процесс

• Define an internal Document contract
• Normalize provider responses at the adapter boundary
• Persist durable crawl job state
• Implement application‑level acceptance checks
• Run the same corpus through both providers and compare metrics
• Roll out with a feature flag and rollback plan

Доказательства

Changing crawl providers is a workflow migration, not a base-URL edit.

Источник

The code is illustrative and has not been run against either paid API. Verify current API contracts before production use.

Источник

Define an internal document contract

Источник

Persist job state so the migration can resume

Источник

Treat acceptance as a separate status

Источник