Build a Kafka MCP Server: Safe Consumer Lag and Topic Diagnostics for AI Agents

Description and use

Build a Kafka MCP server that safely exposes consumer lag, topic health, and broker status to AI agents via AdminAPI, using a read‑only SCRAM user.

Problem

Incidents in event-driven systems often stem from consumer groups lagging millions of messages, yet diagnosing lag is hard because naive tools consume messages or alter offsets, risking data loss or performance impact.

Outcome

The MCP server provides AI agents with instant, safe diagnostics of consumer lag and Kafka health without risking data exposure or altering consumer state.

Workflow

• 1. Create a SCRAM user (agent‑ro) with only Describe and DescribeConfigs ACLs on topics, groups, and cluster.
• 2. Configure Kafka AdminClient in the MCP server to use this user over SASL_SSL.
• 3. Implement four FastMCP tools: group_lag, all_groups_lag, topic_health, cluster_health, each querying AdminClient for offsets, end offsets, and configs.
• 4. Cap returned lists (MAX_PARTITIONS, MAX_GROUPS) to keep model input manageable.
• 5. Deploy the MCP server and expose it to AI agents via FastMCP endpoint.

Evidence

Half of "the API is slow" incidents in an event-driven system are actually "a consumer group is 2 million messages behind."

Source

The server follows the same rule as the Postgres one: statistics and metadata only, never contents.

Source

Create a dedicated SCRAM user and grant it exactly one verb. Describe on topics covers metadata and end offsets; Describe on groups covers committed offsets and member state; Describe plus DescribeConfigs on the cluster covers broker and config lookups.

Source