Why We Don’t Always Need Kafka: High-Throughput Background Queues in Spring Boot with Redis & Virtual Threads

说明与用途

OxMQ 通过 Redis 和 Java 21 虚拟线程实现高吞吐量后台处理，消除了许多场景下对 Kafka 的需求。

问题

Engineering teams often default to Kafka for background workers, incurring operational complexity and cost for workloads that only need reliable retries, rate limiting, and task orchestration.

结果

OxMQ enables 800+ concurrent workers on 35 OS threads, reduces memory footprint to ~1 KB per task, and provides native rate limiting and retry logic, eliminating the need for Kafka in many background processing scenarios.

工作流

• Add OxMQ Spring Boot starter dependency
• Configure Redis URI, namespace, and worker concurrency in application.yml
• Enqueue jobs with add()/addBulk() and JobOptions for retries and retention
• Declare @OxmqListener components with concurrency settings
• OxMQ workers use virtual threads to process jobs, automatically handling retries and progress updates
• Optional: enable RateLimitSentinelPlugin or AdaptiveRateLimiterPlugin for external API throttling

证据

If you don't need petabyte-scale append-only event streaming with weeks of historical replay, Kafka is often the wrong tool for task queues.

来源

OxMQ executes BullMQ’s official 49 battle-tested Lua scripts directly inside Redis.

来源

With Java 21 Project Loom: Virtual Threads are managed by the JVM, consuming only ~1 KB of heap memory each.

来源