Why We Don’t Always Need Kafka: High-Throughput Background Queues in Spring Boot with Redis & Virtual Threads

Description and use

OxMQ enables high‑throughput background processing with Redis and Java 21 virtual threads, removing the need for Kafka in many use cases.

Problem

Engineering teams often default to Kafka for background workers, incurring operational complexity and cost for workloads that only need reliable retries, rate limiting, and task orchestration.

Outcome

OxMQ enables 800+ concurrent workers on 35 OS threads, reduces memory footprint to ~1 KB per task, and provides native rate limiting and retry logic, eliminating the need for Kafka in many background processing scenarios.

Workflow

• Add OxMQ Spring Boot starter dependency
• Configure Redis URI, namespace, and worker concurrency in application.yml
• Enqueue jobs with add()/addBulk() and JobOptions for retries and retention
• Declare @OxmqListener components with concurrency settings
• OxMQ workers use virtual threads to process jobs, automatically handling retries and progress updates
• Optional: enable RateLimitSentinelPlugin or AdaptiveRateLimiterPlugin for external API throttling

Evidence

If you don't need petabyte-scale append-only event streaming with weeks of historical replay, Kafka is often the wrong tool for task queues.

Source

OxMQ executes BullMQ’s official 49 battle-tested Lua scripts directly inside Redis.

Source

With Java 21 Project Loom: Virtual Threads are managed by the JVM, consuming only ~1 KB of heap memory each.

Source