Nginx proxy_buffering Ate My LLM Stream: 11s to First Token

Description and use

In production, LLM streaming was delayed 11 s due to buffering by nginx, gzip, CDN, and the app. Disabling each layer reduced first‑token latency to 0.7 s and eliminated duplicate generations.

Problem

LLM streaming responses were delayed by 11 s in production, causing user frustration and wasted compute.

Outcome

Reduced first‑token latency from 11.4 s to 0.7 s, cut duplicate report generations, and improved user experience.

Workflow

• Identify delay by comparing curl -N to origin vs proxy
• Disable nginx proxy_buffering
• Disable gzip for text/event-stream
• Set correct content‑type and no‑transform headers at CDN
• Remove application‑level token chunking
• Verify time to first byte with curl -N

Evidence

If your LLM stream arrives all at once in production but streams fine locally, a proxy is buffering it.

Source

Four things buffered my tokens. Here’s the measured time to first visible token in the browser, on the same 1,800-token report, fixing one layer at a time:

Source

curl -N -s -o /dev/null -w 'ttfb %{time_starttransfer}s\n' https://api.example.com/api/report/stream

Source