Nginx proxy_buffering Ate My LLM Stream: 11s to First Token

说明与用途

在生产环境中，LLM 流因 nginx、gzip、CDN 和应用层缓冲导致 11 秒延迟。逐层禁用后，首个可见 token 的时间降至 0.7 秒，消除了重复生成。

问题

LLM streaming responses were delayed by 11 s in production, causing user frustration and wasted compute.

结果

Reduced first‑token latency from 11.4 s to 0.7 s, cut duplicate report generations, and improved user experience.

工作流

• Identify delay by comparing curl -N to origin vs proxy
• Disable nginx proxy_buffering
• Disable gzip for text/event-stream
• Set correct content‑type and no‑transform headers at CDN
• Remove application‑level token chunking
• Verify time to first byte with curl -N

证据

If your LLM stream arrives all at once in production but streams fine locally, a proxy is buffering it.

来源

Four things buffered my tokens. Here’s the measured time to first visible token in the browser, on the same 1,800-token report, fixing one layer at a time:

来源

curl -N -s -o /dev/null -w 'ttfb %{time_starttransfer}s\n' https://api.example.com/api/report/stream

来源