The FastAPI Backend Boilerplate I Wish I Had When I Started Building LLM Apps

Описание и применение

Шаблон FastAPI для LLM‑приложений устраняет повторяющиеся трудоемкие задачи: потоковая передача без блокировки, длительные агентские операции без таймаутов и многопользовательские WebSocket‑сессии.

Задача

LLM backends repeatedly face the same tedious problems: blocking streaming, agent run timeouts, and session isolation.

Результат

The scaffold eliminates repetitive boilerplate, enabling teams to focus on agent logic rather than infrastructure concerns.

Рабочий процесс

• Implement FastAPI factory and middleware stack
• Configure Pydantic BaseSettings for env vars
• Define /chat/completions endpoint using StreamingResponse for SSE
• Create /agents/run endpoint that submits tasks to an asyncio TaskQueue and returns a task_id
• Implement /agents/{task_id}/status endpoint for polling
• Add WebSocket /ws/chat for multi-turn sessions
• Include health probes and rate limiting middleware
• Wrap LLM calls in services/llm.py
• Write full test suite with mock LLM
• Build Dockerfile and docker-compose for deployment

Доказательства

FastAPI for the endpoints, a basic /chat route, and then a week of solving the same problems…

Источник

The scaffold's streaming endpoint uses StreamingResponse with a proper async generator — no blocking, backpressure handled, Cache-Control and X-Accel-Buffering headers set correctly for nginx

Источник

The scaffold's background task pattern returns immediately with a task_id. The client polls for the result

Источник