Architecting a Zero-Loss Webhook Ingestion Engine with n8n and Redis (5k req/s DLQ)

Описание и применение

Архитектура обеспечивает нулевую потерю вебхуков, используя Redis Streams и n8n, с дед‑лифт‑кью и idempotency‑ключами SHA‑256.

Задача

Webhook payloads are lost when downstream services fail or rate‑limit, causing data loss and manual re‑processing.

Результат

The architecture isolates ingestion from downstream consumers, enforces exponential backoff with jitter, preserves metadata, and guarantees exactly‑once processing using SHA‑256 idempotency checks.

Рабочий процесс

• Accept webhook and return 202 Accepted
• Generate deterministic SHA‑256 idempotency key and sanitize headers
• Push payload to Redis Stream (webhook:stream:raw)
• Worker reads stream with XREADGROUP
• If downstream call fails with retryable status, enqueue to Redis ZSET with backoff
• Compute exponential backoff with full jitter
• Re‑enqueue to stream after delay
• After max retries, move to isolated DLQ (webhook:dlq:dead) with sanitized headers
• Provide manual replay via idempotent sync

Доказательства

By default, native webhook nodes in low‑code orchestration engines (like standard n8n workflows, Zapier, or Make) execute synchronously.

Источник

If downstream services timeout or throw an uncaught 5xx error, the payload vanishes into log oblivion.

Источник

This architecture isolates the ingestion layer from downstream consumers, enforces exponential backoff with full jitter, preserves metadata, and guarantees exactly‑once processing using SHA‑256 idempotency checks.

Источник

After $N$ failed retries (e.g., 5 attempts), the payload is sent to webhook:dlq:dead, sanitizing sensitive headers while preserving headers, execution error stacks, and original timestamps.

Источник