Nginx proxy_buffering Ate My LLM Stream: 11s to First Token

Описание и применение

В продакшене поток LLM задерживался 11 с из‑за буферизации nginx, gzip, CDN и приложения. Отключив каждый слой, удалось сократить время до первого токена до 0,7 с и устранить повторные генерации.

Задача

LLM streaming responses were delayed by 11 s in production, causing user frustration and wasted compute.

Результат

Reduced first‑token latency from 11.4 s to 0.7 s, cut duplicate report generations, and improved user experience.

Рабочий процесс

• Identify delay by comparing curl -N to origin vs proxy
• Disable nginx proxy_buffering
• Disable gzip for text/event-stream
• Set correct content‑type and no‑transform headers at CDN
• Remove application‑level token chunking
• Verify time to first byte with curl -N

Доказательства

If your LLM stream arrives all at once in production but streams fine locally, a proxy is buffering it.

Источник

Four things buffered my tokens. Here’s the measured time to first visible token in the browser, on the same 1,800-token report, fixing one layer at a time:

Источник

curl -N -s -o /dev/null -w 'ttfb %{time_starttransfer}s\n' https://api.example.com/api/report/stream

Источник