kubeflow/mcp-apache-spark-history-server — MCP Server and CLI for Apache Spark History Server.

Description and use

**Brief Summary (2‑3 sentences)**
The kubeflow/mcp-apache-spark‑history‑server project provides a Model‑Context‑Protocol (MCP) server and accompanying CLI that expose Apache Spark’s History Server functionality through a standard, AI‑friendly API. By wrapping Spark’s UI and REST endpoints, it lets AI agents, scripts, or terminal users query job metadata, logs, and metrics in a uniform way, making Spark debugging and observability programmatically accessible.

**Value**

• **AI‑ready integration** – The MCP layer translates Spark’s native data into a protocol that large language models and other AI assistants can consume, enabling natural‑language queries (“Why did this stage fail?”) and automated root‑cause analysis.
• **Unified tooling** – Developers can interact with Spark history data via a CLI, Python SDK, or direct API calls, eliminating the need to parse HTML UI or maintain custom scrapers.
• **Standardized ecosystem** – By adhering to the Model Context Protocol, the server can be swapped with any other MCP‑compatible service, fostering plug‑and‑play integrations across Kubeflow, MLOps pipelines, and observability stacks.

**Practical Adoption Path**

1. **Prototype** – Deploy the Docker image (or Helm chart) alongside an existing Spark History Server in a test namespace. Use the provided CLI (mcp-spark-history) to fetch job summaries and validate that the MCP responses match the UI.
2. **Integrate** – Replace ad‑hoc Spark‑history scripts with calls to the MCP API from your AI agents, CI pipelines, or monitoring dashboards. The Python SDK can be added as a lightweight dependency in existing notebooks or services.
3. **Scale** – Configure the server behind a load balancer, enable authentication (e.g., OIDC) and TLS, and point production AI assistants or automated remediation bots to the MCP endpoint.

**Production Readiness**

• **Activity & Community** – 168 ★, 58 forks, recent commits (last update 2026‑05‑12) and active issue handling indicate a healthy maintainer base.
• **Maturity** – Core functionality (API/CLI/SDK) is complete, with clear documentation and a small, well‑defined Python codebase.
• **Risk Profile** – No major licensing or security red flags have been identified, though a final audit of dependencies and maintainer responsiveness is advisable.

Overall, the project is sufficiently stable for a serious pilot in production environments, especially where AI‑driven Spark debugging or automated observability is a priority.

Problem

kubeflow/mcp-apache-spark-history-server helps connect AI assistants to real tools and data through a standard protocol.

Outcome

High for an OSS candidate: recent activity, adoption, and ecosystem signals are strong enough for a serious pilot.

Workflow

• connect AI agents to tools
• ship Model Context Protocol servers
• standardize integrations

Evidence

MCP Server and CLI for Apache Spark History Server. Debug Spark applications from AI agents, scripts, or the terminal.

Source