Why Production RAG Pipelines Need More Than a Vector Database on AWS

说明与用途

在生产 RAG 管道中，同步摄取导致超时和重复。解决方案是使用 S3、SQS、Lambda 和 OpenSearch 的异步管道，采用确定性 ID、DLQ 和 DynamoDB 状态跟踪。

问题

In production RAG pipelines, synchronous ingestion causes timeouts, duplicate processing, and unreliable scaling when embedding APIs are slow or rate‑limited.

结果

The asynchronous pipeline decouples upload from processing, eliminates timeouts, ensures idempotent indexing, and provides retry and failure handling, at the cost of eventual consistency and added status‑tracking complexity.

工作流

• 1. Client uploads file to S3 via API; API returns 200 OK immediately.
• 2. S3 event triggers SQS message (or via SQS directly).
• 3. Lambda worker pulls message, chunks document, generates deterministic chunk IDs using S3 key, version, and index.
• 4. Worker calls OpenSearch bulk API; duplicate IDs overwrite existing chunks.
• 5. Worker deletes orphaned chunks if chunk count decreases.
• 6. Worker handles per‑item bulk errors by retrying failed chunks.
• 7. Worker updates DynamoDB status table to COMPLETED or FAILED.
• 8. Frontend polls status endpoint to display progress and eventual searchability.

证据

The embedding API would time out after 28 seconds on page 38, returning a 504 Gateway Timeout.

来源

I chose SQS because I wanted an explicit buffering boundary between uploads and workers.

来源

The worker needed to be idempotent. The simplest way to achieve this in a vector database is to use a deterministic document ID.

来源