How I Keep My Site’s Chatbot Under a 9‑Second Serverless Timeout

Description and use

A chatbot on Cloudflare Workers consistently completes inference within 9 seconds by using MiniLM‑Chat‑6B, caching the model in KV, and applying a 6‑second timeout.

Problem

Cloudflare Workers enforce a 9‑second CPU limit, so the chatbot must finish inference and respond before that deadline.

Outcome

The chatbot now reliably completes inference and returns a response within 9 seconds, avoiding timeouts and maintaining user experience.

Workflow

• Choose a lightweight model that fits the 128 MB memory budget – MiniLM‑Chat‑6B exported to ggml and run with llama.cpp.
• Store the model file in Cloudflare Workers KV and fetch it on the first cold start, caching it in a global variable for subsequent requests.
• Implement a handler that loads the model if not ready, then runs inference with a 6‑second timeout using a custom withTimeout helper.
• If inference succeeds, return the answer and duration; if it times out, fall back to a deterministic rule‑based reply.
• Log inference duration to KV for monitoring.

Evidence

I run a personal automation server at home that powers the chatbot on my Dream Futures Tech website. The serverless platform I use (Cloudflare Workers) imposes a hard 9‑second execution limit, so every request has to finish well before that deadline or the user sees a timeout error.

Source

After testing a few options locally I settled on MiniLM‑Chat‑6B from Hugging Face, exported to the ggml format and run with llama.cpp. The model loads in ~2 seconds and generates a short answer in under 5 seconds on the free tier of Cloudflare Workers KV.

Source

Because the model file lives in KV, I have to fetch it on the first cold start and cache it in a global variable. Subsequent requests reuse the cached instance, keeping the load time negligible.

Source

Run inference with a safety margin (6 seconds) ... leaving 3 seconds for network overhead and response serialization.

Source