Why Production RAG Pipelines Need More Than a Vector Database on AWS

Описание и применение

В продакшн‑пайплайнах RAG синхронная обработка приводит к тайм‑аутам и дублированию. Решение: асинхронный пайплайн на S3, SQS, Lambda и OpenSearch с детерминированными ID, DLQ и статус‑трекингом в DynamoDB.

Задача

In production RAG pipelines, synchronous ingestion causes timeouts, duplicate processing, and unreliable scaling when embedding APIs are slow or rate‑limited.

Результат

The asynchronous pipeline decouples upload from processing, eliminates timeouts, ensures idempotent indexing, and provides retry and failure handling, at the cost of eventual consistency and added status‑tracking complexity.

Рабочий процесс

• 1. Client uploads file to S3 via API; API returns 200 OK immediately.
• 2. S3 event triggers SQS message (or via SQS directly).
• 3. Lambda worker pulls message, chunks document, generates deterministic chunk IDs using S3 key, version, and index.
• 4. Worker calls OpenSearch bulk API; duplicate IDs overwrite existing chunks.
• 5. Worker deletes orphaned chunks if chunk count decreases.
• 6. Worker handles per‑item bulk errors by retrying failed chunks.
• 7. Worker updates DynamoDB status table to COMPLETED or FAILED.
• 8. Frontend polls status endpoint to display progress and eventual searchability.

Доказательства

The embedding API would time out after 28 seconds on page 38, returning a 504 Gateway Timeout.

Источник

I chose SQS because I wanted an explicit buffering boundary between uploads and workers.

Источник

The worker needed to be idempotent. The simplest way to achieve this in a vector database is to use a deterministic document ID.

Источник