Why We Don’t Always Need Kafka: High-Throughput Background Queues in Spring Boot with Redis & Virtual Threads

Описание и применение

OxMQ позволяет обрабатывать тысячи задач в фоне, используя Redis и виртуальные потоки Java 21, устраняя необходимость в Kafka.

Задача

Engineering teams often default to Kafka for background workers, incurring operational complexity and cost for workloads that only need reliable retries, rate limiting, and task orchestration.

Результат

OxMQ enables 800+ concurrent workers on 35 OS threads, reduces memory footprint to ~1 KB per task, and provides native rate limiting and retry logic, eliminating the need for Kafka in many background processing scenarios.

Рабочий процесс

• Add OxMQ Spring Boot starter dependency
• Configure Redis URI, namespace, and worker concurrency in application.yml
• Enqueue jobs with add()/addBulk() and JobOptions for retries and retention
• Declare @OxmqListener components with concurrency settings
• OxMQ workers use virtual threads to process jobs, automatically handling retries and progress updates
• Optional: enable RateLimitSentinelPlugin or AdaptiveRateLimiterPlugin for external API throttling

Доказательства

If you don't need petabyte-scale append-only event streaming with weeks of historical replay, Kafka is often the wrong tool for task queues.

Источник

OxMQ executes BullMQ’s official 49 battle-tested Lua scripts directly inside Redis.

Источник

With Java 21 Project Loom: Virtual Threads are managed by the JVM, consuming only ~1 KB of heap memory each.

Источник