Quarantine, Don't Drop: What I Do With Webhook Payloads That Fail Validation

Description and use

Problem: invalid webhook payloads are lost when dropped. Solution: quarantine them in a table, monitor, and replay after parser fixes, recovering 214 events.

Problem

Webhook payloads often fail validation due to undocumented fields, type changes, or null values, leading to lost events when dropped.

Outcome

Recovered 214 events last month from a provider’s undocumented field rename with minimal human effort.

Workflow

• Validate payload
• If valid, process
• If invalid, insert raw body, errors, timestamp, source, and status into quarantine table
• Schedule alerts on quarantine depth per source
• When parser fixed, run replay workflow that reads new rows, feeds raw body into same processing sub-workflow, uses idempotency keys, marks as replayed or leaves new on failure

Evidence

Failing validation doesn't make a payload worthless — it makes it evidence.

Source

The quarantine is not a log line. It's a table row with four things: raw body, validation errors, received timestamp + source, status.

Source

Last month this recovered 214 events from a provider's undocumented field rename.

Source