nginx streams your tokens fine. HAProxy holds them for 206ms

Description and use

HAProxy’s default configuration buffers small SSE frames, causing a 206 ms first‑token delay and bursty delivery. Larger frames mitigate this. Nginx’s proxy_buffering only adds latency when buffers fill. The X‑Accel‑Buffering header is ineffective for HAProxy.

Problem

When streaming Server‑Sent Events (SSE) through proxies, small token‑sized frames can be delayed or coalesced, causing higher latency and bursty delivery.

Outcome

HAProxy’s default configuration buffers small SSE frames, causing a 206 ms first‑token delay and bursty delivery. Larger frames mitigate this. Nginx’s proxy_buffering only adds latency when buffers fill. The X‑Accel‑Buffering header is ineffective for HAProxy.

Workflow

• 1. Deploy a synthetic SSE emitter that sends 50 ms intervals.
• 2. Configure four reverse proxies (nginx, Caddy, Traefik, HAProxy) with identical upstream settings.
• 3. Measure first‑token latency, gap between frames, and frames per read using a client that reads promptly.
• 4. Repeat measurements with varying frame sizes (≈60 B vs ≈1 KB).
• 5. Test the X‑Accel‑Buffering header on HAProxy.
• 6. Validate results against a real model endpoint (gpt‑oss‑20b).
• 7. Apply guards: discard runs where emitter pacing drifts or direct path is irregular.
• 8. Record and compare metrics.
• 9. Draw conclusions on proxy buffering behavior.
• 10. Document findings and recommendations.

Evidence

HAProxy 3.4.4: 206 ms first token, 0.0 ms gap, 5.12 frames per read

Source

Increasing frame size to ~1.1KB reduced HAProxy delay to 53 ms and frames per read to 1.46

Source

X‑Accel‑Buffering header had no effect on HAProxy (206 ms vs 214 ms)

Source

Nginx proxy_buffering on added 53 ms first‑token latency for 328 KB payload with slow client

Source