How I built a RAG ingestion pipeline for ecommerce chatbots

Description and use

A RAG ingestion pipeline for ecommerce chatbots was built using Cloudflare R2, Queue, AI Search, and Durable Objects. Stable document identities and idempotent completion prevent stale content and double billing.

Problem

Ensuring that updated store documents are correctly ingested, indexed, and available for retrieval without leaving stale or conflicting versions during reindexing.

Outcome

The pipeline separates upload, background processing, and job state, uses stable document identities, and ensures idempotent completion, reducing stale content and improving reliability.

Workflow

• Upload document to R2
• Create manifest event
• Trigger Cloudflare Queue
• Worker claims job via backend
• Delete existing document in AI Search
• Upload new document to AI Search
• Durable Object tracks job state and progress
• Settle job and notify dashboard

Evidence

The ingestion flow looks like this: Document upload to R2 | Manifest creation event | Cloudflare Queue | Worker claims the job | Replace document in AI Search | Durable Object tracks indexing | Settle job and notify dashboard

Source

In ProxyAI, the document name is an identity, not just a display filename. An edited document can use the same identity, and a synced WordPress page can use its permalink.

Source

I upload the document whole to Cloudflare AI Search and let it handle chunking and embedding.

Source