Beyond Kafka and Redis, Part 2: An AI Chat Backend on NATS 2.15

Description and use

A NATS‑based backend replaces Kafka, Redis, SQS, Postgres and cron, providing GPU queueing, token streaming, chat history, usage metering and timeouts.

Problem

Replacing Kafka, Redis, SQS, Postgres and cron services with a unified NATS backend for an AI chat product.

Outcome

The NATS‑based backend replaces multiple services with a single unified system, achieving GPU queue throttling, token streaming resilience, usage metering, and timeout handling without external queues, caches, or cron jobs.

Workflow

• Define subject layout for all state streams
• Create INFERENCE stream and consumer with priority groups for GPU and burst workers
• Use consumer.fetch with min_pending to throttle burst workers
• Publish token chunks to CHAT stream with per‑message TTL
• Browser consumes CHAT stream, tracks last sequence, resumes on reconnect
• Use delayed messages (@at) for timeouts and reminders
• Use recurring schedules (@every) for periodic jobs
• Use KV leases to track worker ownership
• Use distributed counters for per‑region and global usage
• Use atomic batch publish for all-or-nothing writes

Evidence

The system is the backend for an AI chat product. It has a GPU work queue, token streaming to the browser, conversation history, per-tenant usage metering, run timeouts, and worker ownership of conversations.

Source

In a typical stack that's Kafka or SQS for the queue, Redis for counters and locks, a cron service for timeouts, Postgres for history, and a WebSocket gateway for streaming.

Source

NATS has shipped four minor releases since part 1: 2.11, 2.12, 2.14, and 2.15 (2.13 was skipped). Several of the new features close gaps that used to push you toward one of those other systems.

Source

With 60 jobs queued and both pools pulling 10 at a time: round 0: on-prem=10 burst=10 pending=40 round 1: on-prem=10 burst=8 pending=22 round 2: on-prem=10 burst=0 pending=12 round 3: on-prem=10 burst=0 pending=2 round 4: on-prem=2 burst=0 pending=0

Source