Beyond Kafka and Redis, Part 2: An AI Chat Backend on NATS 2.15

Описание и применение

NATS‑бэкенд заменяет Kafka, Redis, SQS, Postgres и cron, обеспечивая очередь GPU, поток токенов, историю чата, метрику использования и таймауты.

Задача

Replacing Kafka, Redis, SQS, Postgres and cron services with a unified NATS backend for an AI chat product.

Результат

The NATS‑based backend replaces multiple services with a single unified system, achieving GPU queue throttling, token streaming resilience, usage metering, and timeout handling without external queues, caches, or cron jobs.

Рабочий процесс

• Define subject layout for all state streams
• Create INFERENCE stream and consumer with priority groups for GPU and burst workers
• Use consumer.fetch with min_pending to throttle burst workers
• Publish token chunks to CHAT stream with per‑message TTL
• Browser consumes CHAT stream, tracks last sequence, resumes on reconnect
• Use delayed messages (@at) for timeouts and reminders
• Use recurring schedules (@every) for periodic jobs
• Use KV leases to track worker ownership
• Use distributed counters for per‑region and global usage
• Use atomic batch publish for all-or-nothing writes

Доказательства

The system is the backend for an AI chat product. It has a GPU work queue, token streaming to the browser, conversation history, per-tenant usage metering, run timeouts, and worker ownership of conversations.

Источник

In a typical stack that's Kafka or SQS for the queue, Redis for counters and locks, a cron service for timeouts, Postgres for history, and a WebSocket gateway for streaming.

Источник

NATS has shipped four minor releases since part 1: 2.11, 2.12, 2.14, and 2.15 (2.13 was skipped). Several of the new features close gaps that used to push you toward one of those other systems.

Источник

With 60 jobs queued and both pools pulling 10 at a time: round 0: on-prem=10 burst=10 pending=40 round 1: on-prem=10 burst=8 pending=22 round 2: on-prem=10 burst=0 pending=12 round 3: on-prem=10 burst=0 pending=2 round 4: on-prem=2 burst=0 pending=0

Источник