The Delivery ID Gap: Deduping Webhook Replays When the Provider Sends No ID

说明与用途

在 n8n 中实现的自动化方案，用于处理缺少交付 ID 的 webhook，使用哈希回退、计数回退触发次数，并根据证据强度调整去重窗口和速率限制。

问题

Webhook providers sometimes omit a delivery ID, making it hard to deduplicate replayed events and detect accidental double-processing.

结果

The recipe introduces a robust fallback mechanism for providers lacking delivery IDs, improves observability by counting fallback triggers, and adapts grace windows and rate limits based on evidence strength, reducing accidental double-processing while maintaining security.

工作流

• 1. Verify the webhook signature to confirm sender authenticity.
• 2. Extract the delivery ID from headers or body; if absent, compute a deterministic hash of the timestamp-normalized payload plus sender key.
• 3. Generate a dedup key: use "sender:deliveryId" for strong keys, "sender:fb:hash" for weak keys.
• 4. Increment a per-provider fallback counter whenever the hash fallback is used.
• 5. Check the dedup store for the key; if present, throw a replay error.
• 6. Store the key with a TTL: full grace window for strong keys, shorter window for weak keys.
• 7. Apply a keyed rate limit based on the sender to mitigate hammering.
• 8. Monitor fallback rates per provider; when a provider stops triggering the fallback, switch to using the real delivery ID and tighten its defenses.
• 9. Review providers that never send IDs and route them to a review queue for manual inspection.

证据

some of my providers don't send a delivery ID at all.

来源

Be honest about what this buys: it kills accidental double-processing, not malicious replay.

来源

increment a per-provider counter every time the hash fallback fires.

来源

Shorter grace TTL. The dedup store TTLs at grace-window length plus clock skew; for fallback-only senders I shrink the window itself.

来源