The FastAPI Backend Boilerplate I Wish I Had When I Started Building LLM Apps

说明与用途

FastAPI LLM 应用模板消除了重复的繁琐任务：非阻塞流式传输、长时间代理运行无超时以及多轮 WebSocket 会话。

问题

LLM backends repeatedly face the same tedious problems: blocking streaming, agent run timeouts, and session isolation.

结果

The scaffold eliminates repetitive boilerplate, enabling teams to focus on agent logic rather than infrastructure concerns.

工作流

• Implement FastAPI factory and middleware stack
• Configure Pydantic BaseSettings for env vars
• Define /chat/completions endpoint using StreamingResponse for SSE
• Create /agents/run endpoint that submits tasks to an asyncio TaskQueue and returns a task_id
• Implement /agents/{task_id}/status endpoint for polling
• Add WebSocket /ws/chat for multi-turn sessions
• Include health probes and rate limiting middleware
• Wrap LLM calls in services/llm.py
• Write full test suite with mock LLM
• Build Dockerfile and docker-compose for deployment

证据

FastAPI for the endpoints, a basic /chat route, and then a week of solving the same problems…

来源

The scaffold's streaming endpoint uses StreamingResponse with a proper async generator — no blocking, backpressure handled, Cache-Control and X-Accel-Buffering headers set correctly for nginx

来源

The scaffold's background task pattern returns immediately with a task_id. The client polls for the result

来源