How I built a RAG ingestion pipeline for ecommerce chatbots

说明与用途

为电商聊天机器人构建了 RAG 采集管道，使用 Cloudflare R2、Queue、AI Search 和 Durable Object。采用稳定的文档标识和幂等完成，避免旧内容和重复计费。

问题

Ensuring that updated store documents are correctly ingested, indexed, and available for retrieval without leaving stale or conflicting versions during reindexing.

结果

The pipeline separates upload, background processing, and job state, uses stable document identities, and ensures idempotent completion, reducing stale content and improving reliability.

工作流

• Upload document to R2
• Create manifest event
• Trigger Cloudflare Queue
• Worker claims job via backend
• Delete existing document in AI Search
• Upload new document to AI Search
• Durable Object tracks job state and progress
• Settle job and notify dashboard

证据

The ingestion flow looks like this: Document upload to R2 | Manifest creation event | Cloudflare Queue | Worker claims the job | Replace document in AI Search | Durable Object tracks indexing | Settle job and notify dashboard

来源

In ProxyAI, the document name is an identity, not just a display filename. An edited document can use the same identity, and a synced WordPress page can use its permalink.

来源

I upload the document whole to Cloudflare AI Search and let it handle chunking and embedding.

来源