Architecting a Zero-Loss Webhook Ingestion Engine with n8n and Redis (5k req/s DLQ)

说明与用途

该架构通过 Redis Streams 和 n8n 实现零丢失的 webhook 摄取，使用死信队列和 SHA‑256 去重键。

问题

Webhook payloads are lost when downstream services fail or rate‑limit, causing data loss and manual re‑processing.

结果

The architecture isolates ingestion from downstream consumers, enforces exponential backoff with jitter, preserves metadata, and guarantees exactly‑once processing using SHA‑256 idempotency checks.

工作流

• Accept webhook and return 202 Accepted
• Generate deterministic SHA‑256 idempotency key and sanitize headers
• Push payload to Redis Stream (webhook:stream:raw)
• Worker reads stream with XREADGROUP
• If downstream call fails with retryable status, enqueue to Redis ZSET with backoff
• Compute exponential backoff with full jitter
• Re‑enqueue to stream after delay
• After max retries, move to isolated DLQ (webhook:dlq:dead) with sanitized headers
• Provide manual replay via idempotent sync

证据

By default, native webhook nodes in low‑code orchestration engines (like standard n8n workflows, Zapier, or Make) execute synchronously.

来源

If downstream services timeout or throw an uncaught 5xx error, the payload vanishes into log oblivion.

来源

This architecture isolates the ingestion layer from downstream consumers, enforces exponential backoff with full jitter, preserves metadata, and guarantees exactly‑once processing using SHA‑256 idempotency checks.

来源

After $N$ failed retries (e.g., 5 attempts), the payload is sent to webhook:dlq:dead, sanitizing sensitive headers while preserving headers, execution error stacks, and original timestamps.

来源