CDC without Kafka: keeping Postgres, OpenSearch, and Redis in sync with one binary

Description and use

VentStream is a single‑binary Rust tool that consumes PostgreSQL logical replication directly, performs joins on the fly, and writes to OpenSearch or Redis, eliminating Kafka and reducing latency.

Problem

Keeping Postgres in sync with search indexes or caches typically required a Kafka stack, adding unnecessary infrastructure for simple change‑to‑target pipelines.

Outcome

VentStream achieved zero drift after 250,000 mutations with repeated hard kills, performed joins during CDC, and handled TOAST columns correctly, eliminating the need for Kafka and reducing batch latency.

Workflow

• 1. Configure VentStream with source (Postgres) and sink (OpenSearch/Redis) in a YAML file.
• 2. VentStream connects to the source’s native change feed (pgoutput).
• 3. It maintains local join state and composes enriched documents as changes arrive.
• 4. Documents are written to the sink via bulk API or keyspace updates.
• 5. VentStream advances the replication slot only after the sink acknowledges writes, ensuring no data loss.
• 6. In case of errors, bad rows are sent to a dead‑letter queue file and the pipeline continues.

Evidence

I tested this by pushing 250,000 mutations through the pipeline and repeatedly hard‑killing the process. The source and sink ended up with zero drift in those validation runs.

Source

VentStream only advances the replication watermark after the sink has acknowledged the write.

Source