CDC without Kafka: keeping Postgres, OpenSearch, and Redis in sync with one binary

Описание и применение

VentStream – это однопроцессный инструмент на Rust, который читает изменения из PostgreSQL без Kafka, выполняет необходимые соединения и обновляет OpenSearch или Redis, обеспечивая нулевую задержку и отказоустойчивость.

Задача

Keeping Postgres in sync with search indexes or caches typically required a Kafka stack, adding unnecessary infrastructure for simple change‑to‑target pipelines.

Результат

VentStream achieved zero drift after 250,000 mutations with repeated hard kills, performed joins during CDC, and handled TOAST columns correctly, eliminating the need for Kafka and reducing batch latency.

Рабочий процесс

• 1. Configure VentStream with source (Postgres) and sink (OpenSearch/Redis) in a YAML file.
• 2. VentStream connects to the source’s native change feed (pgoutput).
• 3. It maintains local join state and composes enriched documents as changes arrive.
• 4. Documents are written to the sink via bulk API or keyspace updates.
• 5. VentStream advances the replication slot only after the sink acknowledges writes, ensuring no data loss.
• 6. In case of errors, bad rows are sent to a dead‑letter queue file and the pipeline continues.

Доказательства

I tested this by pushing 250,000 mutations through the pipeline and repeatedly hard‑killing the process. The source and sink ended up with zero drift in those validation runs.

Источник

VentStream only advances the replication watermark after the sink has acknowledged the write.

Источник