Why Dropping "await" is Broken: Building a Resilient Background Task Engine with Node.js, Redis Streams, and PostgreSQL

Description and use

Dropping await blocks client connections and causes duplicate actions. Mailman‑Relay uses Redis Streams and PostgreSQL to provide asynchronous task processing with atomic deduplication, fault tolerance, and zero job loss.

Problem

Holding the client’s HTTP connection hostage while executing slow external I/O causes UI freezes, duplicate actions, and unreliable side‑effects.

Outcome

Mailman‑Relay achieved ~592 req/s with 0.00% error, P50 ≈ 30 ms, P95 ≈ 50 ms, P99 ≈ 58 ms under 50 concurrent users, and zero job loss after worker crashes or network latency injection.

Workflow

• Accept task via HTTP POST and validate input
• Acquire atomic idempotency lock with Redis SET NX EX
• Insert task metadata into PostgreSQL
• Append task payload to Redis Streams
• Return 202 Accepted to client
• Worker pool consumes stream entries, handles rate limits, retries, and acknowledges via XACK
• Supervisor thread uses XAUTOCLAIM to recover orphaned tasks

Evidence

When developers notice this latency, the most common "quick fix" is simply omitting the await keyword in Node.js

Source

Atomic Deduplication (Double-Click Shield) ... SET idempotency:{key} {job_id} NX EX {ttl}

Source

High-Concurrency Saturation (Grafana k6) ... Throughput: ~592 requests/second sustained

Source

Worker Crash Recovery (SIGKILL) ... Total jobs lost: Zero

Source