The Webhook Burst That Buried My Queue: Backpressure in n8n

Описание и применение

Внедрение легковесного обработчика вебхуков, ограниченного воркера, приоритизированной очереди и сигналов обратного давления уменьшило задержки и предотвратило устаревшие доставки при всплесках событий.

Задача

A webhook workflow received a sudden burst of over a thousand events in one minute, causing downstream 429 responses, queue backlogs, and stale deliveries.

Результат

Implementing a fast, lightweight webhook handler with a bounded worker, priority queue, and backpressure signals reduced latency, prevented stale deliveries, and enabled graceful degradation during bursts.

Рабочий процесс

• 1. Webhook handler validates signature and deduplicates by delivery ID.
• 2. Enqueue payload to durable queue and return 200.
• 3. If queue depth exceeds derived threshold, return 429 with Retry-After.
• 4. Worker pulls a fixed number of jobs (e.g., 2) concurrently.
• 5. Worker processes jobs in priority order, calling downstream APIs.
• 6. Completed jobs are marked done; refused jobs are logged with delivery ID.
• 7. Alerting monitors projected staleness (depth/drain rate) and triggers before backlog burial.

Доказательства

In sixty seconds it received over a thousand.

Источник

The webhook handler does almost nothing. It validates the signature, checks the delivery id for duplicates, writes the payload to a durable queue, and returns 200.

Источник

The worker pulls a fixed number of jobs at a time. Mine is small — two concurrent executions for a provider with a documented rate limit.

Источник

When the queue depth passes a threshold, the webhook handler stops returning 200 and starts returning 429 with a Retry-After header.

Источник

My queue rows carry a priority, and the worker claims the highest-priority oldest job first.

Источник