At the edge, the number that matters is memory - not throughput (specially in Ramageddon)

Description and use

Rekuiper, written in Rust, delivers 100 k msg/s on a single core while keeping memory between 5–10 MB across five realistic MQTT workloads, whereas Go‑based engines consume 0.5–1 GB or fail.

Problem

Edge stream‑processing engines often optimize for peak throughput on large servers, but real deployments run on single‑core, low‑memory devices where bounded memory and single‑core performance are critical.

Outcome

Rekuiper delivers 100 k msg/s on a single core while keeping memory between 5–10 MB across five realistic MQTT workloads, whereas Go‑based engines consume 0.5–1 GB or fail. The benchmark’s exact output verification uncovered a correctness bug that throughput‑only tests would have missed.

Workflow

• 1. Reimplement eKuiper’s REST API, SQL dialect, and CLI in Rust to preserve compatibility.
• 2. Design bounded queues with real backpressure: reserve‑then‑commit batches to all subscribers, rejecting partially delivered batches.
• 3. Implement incremental window aggregation: keep one accumulator per group per aggregate, avoiding per‑message buffering.
• 4. Add an offline sink cache that spills to disk before dropping records, ensuring recoverable failures are retried.
• 5. Build a benchmark that runs each engine in a 1‑core, 1 GB container, feeds identical MQTT workloads via a load generator, and verifies output exactly against proofs.
• 6. Run five realistic workloads (telemetry filter, per‑device windows, ESPHome states, vehicle windows, EV charger sessions) at rates up to 100 k msg/s.
• 7. Compare throughput, memory usage, and correctness across Rekuiper, eKuiper, Telegraf, and Redpanda Connect.
• 8. Analyze results: Rekuiper maintains bounded memory and correct output; other engines either exceed memory limits or fail correctness checks.

Evidence

rekuiper produced complete, correct output at 100,000 messages per second on one core in every workload

Source

rekuiper's memory stayed between 5 and 10 MB while the Go-based engines climbed to half a gigabyte to a full gigabyte

Source