Why Production RAG Pipelines Need More Than a Vector Database on AWS

Description and use

In production RAG pipelines, synchronous ingestion causes timeouts and duplicates. The solution is an asynchronous pipeline using S3, SQS, Lambda, and OpenSearch with deterministic IDs, DLQ, and DynamoDB status tracking.

Problem

In production RAG pipelines, synchronous ingestion causes timeouts, duplicate processing, and unreliable scaling when embedding APIs are slow or rate‑limited.

Outcome

The asynchronous pipeline decouples upload from processing, eliminates timeouts, ensures idempotent indexing, and provides retry and failure handling, at the cost of eventual consistency and added status‑tracking complexity.

Workflow

• 1. Client uploads file to S3 via API; API returns 200 OK immediately.
• 2. S3 event triggers SQS message (or via SQS directly).
• 3. Lambda worker pulls message, chunks document, generates deterministic chunk IDs using S3 key, version, and index.
• 4. Worker calls OpenSearch bulk API; duplicate IDs overwrite existing chunks.
• 5. Worker deletes orphaned chunks if chunk count decreases.
• 6. Worker handles per‑item bulk errors by retrying failed chunks.
• 7. Worker updates DynamoDB status table to COMPLETED or FAILED.
• 8. Frontend polls status endpoint to display progress and eventual searchability.

Evidence

The embedding API would time out after 28 seconds on page 38, returning a 504 Gateway Timeout.

Source

I chose SQS because I wanted an explicit buffering boundary between uploads and workers.

Source

The worker needed to be idempotent. The simplest way to achieve this in a vector database is to use a deterministic document ID.

Source