Build a Kafka MCP Server: Safe Consumer Lag and Topic Diagnostics for AI Agents

Описание и применение

Создайте сервер MCP для Kafka, который безопасно предоставляет AI‑агентам метаданные о задержке потребителей, здоровье топиков и брокеров, используя только Describe‑ACL и AdminAPI.

Задача

Incidents in event-driven systems often stem from consumer groups lagging millions of messages, yet diagnosing lag is hard because naive tools consume messages or alter offsets, risking data loss or performance impact.

Результат

The MCP server provides AI agents with instant, safe diagnostics of consumer lag and Kafka health without risking data exposure or altering consumer state.

Рабочий процесс

• 1. Create a SCRAM user (agent‑ro) with only Describe and DescribeConfigs ACLs on topics, groups, and cluster.
• 2. Configure Kafka AdminClient in the MCP server to use this user over SASL_SSL.
• 3. Implement four FastMCP tools: group_lag, all_groups_lag, topic_health, cluster_health, each querying AdminClient for offsets, end offsets, and configs.
• 4. Cap returned lists (MAX_PARTITIONS, MAX_GROUPS) to keep model input manageable.
• 5. Deploy the MCP server and expose it to AI agents via FastMCP endpoint.

Доказательства

Half of "the API is slow" incidents in an event-driven system are actually "a consumer group is 2 million messages behind."

Источник

The server follows the same rule as the Postgres one: statistics and metadata only, never contents.

Источник

Create a dedicated SCRAM user and grant it exactly one verb. Describe on topics covers metadata and end offsets; Describe on groups covers committed offsets and member state; Describe plus DescribeConfigs on the cluster covers broker and config lookups.

Источник