Move Slow LLM Calls Off the Request Path with BullMQ in Node.js

Description and use

Moving slow LLM calls off the request path into a BullMQ queue improves reliability and scalability by returning a 202 immediately, handling retries, concurrency, and dead‑lettering in a worker.

Problem

LLM calls that take 20 s+ cause timeouts, duplicate work, provider rate‑limit hits, and lost work on deploys when executed inline in HTTP handlers.

Outcome

API latency reduced, retries isolated to the queue, provider rate limits respected, and job durability improved.

Workflow

• Client POST /summaries → API enqueues job in BullMQ and returns jobId
• Worker picks job, loads document, calls LLM, updates progress stages
• Worker saves result and completes job
• Client GET /summaries/:jobId/events streams progress until completion
• Failed jobs move to dead‑letter queue for manual inspection

Evidence

If an LLM call can take 20 seconds or more, keep it out of your HTTP handler.

Source

Put the work on a BullMQ queue backed by Redis, return a job ID right away with a 202 Accepted, and let a separate worker make the slow call while the client gets progress updates over Server‑Sent Events.

Source

With a queue, the API stays fast and the slow part has its own retry policy, its own concurrency cap and its own failure handling.

Source