Move Slow LLM Calls Off the Request Path with BullMQ in Node.js

Описание и применение

Перенос медленных вызовов LLM из HTTP‑обработчика в очередь BullMQ повышает надёжность и масштабируемость, обеспечивая быстрый ответ клиенту, управление повторными попытками и обработку ошибок.

Задача

LLM calls that take 20 s+ cause timeouts, duplicate work, provider rate‑limit hits, and lost work on deploys when executed inline in HTTP handlers.

Результат

API latency reduced, retries isolated to the queue, provider rate limits respected, and job durability improved.

Рабочий процесс

• Client POST /summaries → API enqueues job in BullMQ and returns jobId
• Worker picks job, loads document, calls LLM, updates progress stages
• Worker saves result and completes job
• Client GET /summaries/:jobId/events streams progress until completion
• Failed jobs move to dead‑letter queue for manual inspection

Доказательства

If an LLM call can take 20 seconds or more, keep it out of your HTTP handler.

Источник

Put the work on a BullMQ queue backed by Redis, return a job ID right away with a 202 Accepted, and let a separate worker make the slow call while the client gets progress updates over Server‑Sent Events.

Источник

With a queue, the API stays fast and the slow part has its own retry policy, its own concurrency cap and its own failure handling.

Источник