Move Slow LLM Calls Off the Request Path with BullMQ in Node.js

说明与用途

将慢速 LLM 调用从请求路径移到 BullMQ 队列中，可通过立即返回 202 并在工作进程中处理重试、并发和死信队列，提升可靠性和可扩展性。

问题

LLM calls that take 20 s+ cause timeouts, duplicate work, provider rate‑limit hits, and lost work on deploys when executed inline in HTTP handlers.

结果

API latency reduced, retries isolated to the queue, provider rate limits respected, and job durability improved.

工作流

• Client POST /summaries → API enqueues job in BullMQ and returns jobId
• Worker picks job, loads document, calls LLM, updates progress stages
• Worker saves result and completes job
• Client GET /summaries/:jobId/events streams progress until completion
• Failed jobs move to dead‑letter queue for manual inspection

证据

If an LLM call can take 20 seconds or more, keep it out of your HTTP handler.

来源

Put the work on a BullMQ queue backed by Redis, return a job ID right away with a 202 Accepted, and let a separate worker make the slow call while the client gets progress updates over Server‑Sent Events.

来源

With a queue, the API stays fast and the slow part has its own retry policy, its own concurrency cap and its own failure handling.

来源