hwdsl2/self-hosted-ai-stack — Deploy a complete self-hosted AI stack with Docker Compose: Ollama, LiteLLM, AnythingLLM, Whisper, W…

Description and use

**Brief Summary (2‑3 sentences)**
hwdsl2/self‑hosted‑ai‑stack provides a ready‑to‑run Docker‑Compose bundle that brings together Ollama, LiteLLM, AnythingLLM, Whisper, WhisperLive, Kokoro, embeddings, Docling and an MCP gateway. It delivers a private‑by‑default, multi‑arch (amd64 / arm64) AI platform with optional HTTPS and NVIDIA CUDA acceleration, letting teams run full‑stack LLM, RAG and speech‑to‑text services on‑premise.

**Value**
The stack turns a collection of disparate open‑source AI components into a single, cohesive platform, exposing a standard Model Context Protocol (MCP) interface. This lets developers plug AI agents directly into real tools, data stores, and custom services without writing glue code, accelerating the creation of “AI‑as‑a‑service” products and internal assistants while keeping data under full control.

**Practical Adoption Path**

| Step | Action | Reason |
|------|--------|--------|
| 1️⃣ | Clone the repo and run docker compose up -d (or use the provided .env to enable CUDA/HTTPS). | One‑command bootstrap; works on both x86_64 and ARM hardware. |
| 2️⃣ | Verify each service (Ollama, LiteLLM, Whisper, etc.) via their health‑check endpoints. | Confirms the stack is healthy before integration. |
| 3️⃣ | Register the MCP gateway URL in your AI agent or RAG pipeline. | Provides a uniform API for chat, embeddings, transcription, etc. |
| 4️⃣ | Extend or replace modules (e.g., swap Ollama for a different LLM) by editing the compose file—no code changes needed. | Keeps the stack flexible for evolving requirements. |
| 5️⃣ | Deploy to production (Docker Swarm, Kubernetes, or a VM) and enable TLS/Authentication via the built‑in HTTPS toggle. | Guarantees secure, scalable operation. |

**Production Readiness**

• **Activity & Community**: 107 ⭐, 21 forks, recent commits (as of 2026‑06‑24) and a healthy issue/PR flow indicate active maintenance.
• **Architecture**: Containerized, multi‑arch, and optionally GPU‑accelerated, matching typical enterprise deployment patterns.
• **Security & Compliance**: Private‑by‑default stance, optional TLS, and clear separation of services reduce attack surface; however, a final license and vulnerability audit is still required.
• **Scalability**: Individual services can be scaled out via Docker Compose overrides or orchestrated in Kubernetes, making the stack suitable for pilot projects up to production workloads.

Overall, the project is mature enough for a serious pilot, offering a low‑friction path to a self‑hosted, standards‑based AI backend while still needing the usual final security and licensing review before full production rollout.

Problem

hwdsl2/self-hosted-ai-stack helps connect AI assistants to real tools and data through a standard protocol.

Outcome

High for an OSS candidate: recent activity, adoption, and ecosystem signals are strong enough for a serious pilot.

Workflow

• connect AI agents to tools
• ship Model Context Protocol servers
• standardize integrations

Evidence

Deploy a complete self-hosted AI stack with Docker Compose: Ollama, LiteLLM, AnythingLLM, Whisper, WhisperLive, Kokoro, Embeddings, Docling and MCP Gateway. Local-first, private by default, with lightweight stacks, optional HTTPS and NVIDIA CUDA acceleration. Multi-arch: amd64, arm64.

Source