Beyond Kafka and Redis, Part 2: An AI Chat Backend on NATS 2.15

说明与用途

NATS 后端替代 Kafka、Redis、SQS、Postgres 和 cron，提供 GPU 队列、令牌流、聊天历史、使用计量和超时功能。

问题

Replacing Kafka, Redis, SQS, Postgres and cron services with a unified NATS backend for an AI chat product.

结果

The NATS‑based backend replaces multiple services with a single unified system, achieving GPU queue throttling, token streaming resilience, usage metering, and timeout handling without external queues, caches, or cron jobs.

工作流

• Define subject layout for all state streams
• Create INFERENCE stream and consumer with priority groups for GPU and burst workers
• Use consumer.fetch with min_pending to throttle burst workers
• Publish token chunks to CHAT stream with per‑message TTL
• Browser consumes CHAT stream, tracks last sequence, resumes on reconnect
• Use delayed messages (@at) for timeouts and reminders
• Use recurring schedules (@every) for periodic jobs
• Use KV leases to track worker ownership
• Use distributed counters for per‑region and global usage
• Use atomic batch publish for all-or-nothing writes

证据

The system is the backend for an AI chat product. It has a GPU work queue, token streaming to the browser, conversation history, per-tenant usage metering, run timeouts, and worker ownership of conversations.

来源

In a typical stack that's Kafka or SQS for the queue, Redis for counters and locks, a cron service for timeouts, Postgres for history, and a WebSocket gateway for streaming.

来源

NATS has shipped four minor releases since part 1: 2.11, 2.12, 2.14, and 2.15 (2.13 was skipped). Several of the new features close gaps that used to push you toward one of those other systems.

来源

With 60 jobs queued and both pools pulling 10 at a time: round 0: on-prem=10 burst=10 pending=40 round 1: on-prem=10 burst=8 pending=22 round 2: on-prem=10 burst=0 pending=12 round 3: on-prem=10 burst=0 pending=2 round 4: on-prem=2 burst=0 pending=0

来源