Your Keep-Alive Is Lying to You: Six Connection Myths I Measured on a Free Model Server

Description and use

On free model APIs, latency spikes 4‑5× after idle periods. Measure real latency and adjust retry/timeout logic accordingly.

Problem

Developers rely on keep‑alive and retry advice without measuring real latency on free model endpoints, leading to mis‑tuned timeout and retry logic.

Outcome

Free tier endpoints exhibit significantly higher latency after idle periods; keep‑alive does not mitigate this. Developers should measure latency and adjust retry/timeout logic accordingly.

Workflow

• 1. Write a probe script that warms the connection and records latency.
• 2. Run the probe with a specified idle gap (e.g., 300 s) for multiple rounds.
• 3. Record and analyze latency for warm and idle requests.
• 4. Repeat the test using a persistent connection (keep‑alive) to compare performance.
• 5. Adjust client retry and timeout settings based on observed latency patterns.

Evidence

Notice something? The warm baseline is ~400 ms. The idle requests are 4-5x slower. Every single time.

Source

Free tiers have SLAs. They have capacity guarantees. Free tiers don't.

Source