At the edge, the number that matters is memory - not throughput (specially in Ramageddon)

说明与用途

Rekuiper（Rust实现）在单核、1 GB内存环境下，能够以100 k msg/s的吞吐量运行五种真实MQTT工作负载，并保持5–10 MB的内存占用，而Go实现的引擎则会消耗0.5–1 GB或失败。

问题

Edge stream‑processing engines often optimize for peak throughput on large servers, but real deployments run on single‑core, low‑memory devices where bounded memory and single‑core performance are critical.

结果

Rekuiper delivers 100 k msg/s on a single core while keeping memory between 5–10 MB across five realistic MQTT workloads, whereas Go‑based engines consume 0.5–1 GB or fail. The benchmark’s exact output verification uncovered a correctness bug that throughput‑only tests would have missed.

工作流

• 1. Reimplement eKuiper’s REST API, SQL dialect, and CLI in Rust to preserve compatibility.
• 2. Design bounded queues with real backpressure: reserve‑then‑commit batches to all subscribers, rejecting partially delivered batches.
• 3. Implement incremental window aggregation: keep one accumulator per group per aggregate, avoiding per‑message buffering.
• 4. Add an offline sink cache that spills to disk before dropping records, ensuring recoverable failures are retried.
• 5. Build a benchmark that runs each engine in a 1‑core, 1 GB container, feeds identical MQTT workloads via a load generator, and verifies output exactly against proofs.
• 6. Run five realistic workloads (telemetry filter, per‑device windows, ESPHome states, vehicle windows, EV charger sessions) at rates up to 100 k msg/s.
• 7. Compare throughput, memory usage, and correctness across Rekuiper, eKuiper, Telegraf, and Redpanda Connect.
• 8. Analyze results: Rekuiper maintains bounded memory and correct output; other engines either exceed memory limits or fail correctness checks.

证据

rekuiper produced complete, correct output at 100,000 messages per second on one core in every workload

来源

rekuiper's memory stayed between 5 and 10 MB while the Go-based engines climbed to half a gigabyte to a full gigabyte

来源