The FastAPI Backend Boilerplate I Wish I Had When I Started Building LLM Apps

Description and use

FastAPI scaffold for LLM apps removes repetitive boilerplate: non‑blocking streaming, long agent runs without timeouts, and multi‑turn WebSocket sessions.

Problem

LLM backends repeatedly face the same tedious problems: blocking streaming, agent run timeouts, and session isolation.

Outcome

The scaffold eliminates repetitive boilerplate, enabling teams to focus on agent logic rather than infrastructure concerns.

Workflow

• Implement FastAPI factory and middleware stack
• Configure Pydantic BaseSettings for env vars
• Define /chat/completions endpoint using StreamingResponse for SSE
• Create /agents/run endpoint that submits tasks to an asyncio TaskQueue and returns a task_id
• Implement /agents/{task_id}/status endpoint for polling
• Add WebSocket /ws/chat for multi-turn sessions
• Include health probes and rate limiting middleware
• Wrap LLM calls in services/llm.py
• Write full test suite with mock LLM
• Build Dockerfile and docker-compose for deployment

Evidence

FastAPI for the endpoints, a basic /chat route, and then a week of solving the same problems…

Source

The scaffold's streaming endpoint uses StreamingResponse with a proper async generator — no blocking, backpressure handled, Cache-Control and X-Accel-Buffering headers set correctly for nginx

Source

The scaffold's background task pattern returns immediately with a task_id. The client polls for the result

Source