kafka-mcp 0.1.0

package/README.md

@@ -0,0 +1,172 @@
+# Kafka MCP CLI
+
+`kafka-mcp` 是一个基于 TypeScript 开发的 Kafka 查询 CLI，同时配套了一份可供 AI Agent 使用的 Skill 定义。
+
+当前支持的能力包括：
+
+- 从配置文件或环境变量读取 Kafka 地址并建立连接
+- 获取 Topic 列表并支持搜索
+- 获取指定 Topic 的消息并支持搜索
+- 获取消费者组列表
+- 查看指定消费者组在某个 Topic 上的消费进度
+
+## 功能与命令
+
+当前 CLI 提供以下 5 个命令：
+
+### 1. `topics`
+
+列出所有 Topic，并支持按名称搜索。
+
+示例：
+
+```bash
+node dist/cli.js topics
+node dist/cli.js topics --search order
+```
+
+### 2. `topic`
+
+查看某个 Topic 的元数据，包括：
+
+- 分区编号
+- leader
+- replicas
+- ISR
+
+示例：
+
+```bash
+node dist/cli.js topic --topic orders
+```
+
+### 3. `messages`
+
+读取指定 Topic 的消息，并支持按关键字搜索。
+
+搜索范围包括：
+
+- Topic 名称
+- Message Key
+- Message Value
+- Message Headers
+
+常用参数：
+
+- `--topic`：指定 Topic
+- `--search`：搜索关键字
+- `--limit`：最多返回多少条匹配消息，默认 `10`
+- `--timeout-ms`：读取超时时间，默认 `5000`
+- `--latest`：从最新 offset 开始读取，默认从最早 offset 开始
+
+示例：
+
+```bash
+node dist/cli.js messages --topic orders
+node dist/cli.js messages --topic orders --search paid --limit 20
+node dist/cli.js messages --topic orders --latest
+```
+
+### 4. `groups`
+
+列出消费者组，并支持按组名搜索。
+
+示例：
+
+```bash
+node dist/cli.js groups
+node dist/cli.js groups --search billing
+```
+
+### 5. `group-progress`
+
+查看某个消费者组在某个 Topic 上的消费进度。
+
+输出内容包括：
+
+- 分区编号
+- 当前负责该分区的消费者成员
+- 已提交 offset
+- Topic 最新 offset
+- Lag
+
+示例：
+
+```bash
+node dist/cli.js group-progress --group billing-consumers --topic orders
+```
+
+## 配置方式
+
+CLI 会按以下顺序读取配置，找到第一个存在的配置源后使用：
+
+1. 当前目录下的 `.kafka-mcp.json`
+2. `~/.config/kafka-mcp/config.json`
+3. 环境变量会覆盖上述配置
+
+最小配置示例：
+
+```json
+{
+  "brokers": ["localhost:9092"],
+  "clientId": "kafka-mcp-cli",
+  "ssl": false
+}
+```
+
+如果 Kafka 开启了 SASL，也可以在配置文件中增加认证信息：
+
+```json
+{
+  "brokers": ["localhost:9092"],
+  "clientId": "kafka-mcp-cli",
+  "sasl": {
+    "mechanism": "plain",
+    "username": "username",
+    "password": "password"
+  }
+}
+```
+
+支持的环境变量：
+
+- `KAFKA_BROKERS`
+- `KAFKA_CLIENT_ID`
+- `KAFKA_SSL`
+- `KAFKA_SASL_MECHANISM`
+- `KAFKA_SASL_USERNAME`
+- `KAFKA_SASL_PASSWORD`
+
+## 安装
+
+```bash
+npm install
+npm run build
+```
+
+如果要作为全局命令使用，可以在发布到 npm 后执行：
+
+```bash
+npm install -g kafka-mcp
+```
+
+## 使用示例
+
+```bash
+npx tsx src/cli.ts topics --search order
+npx tsx src/cli.ts topic --topic orders
+npx tsx src/cli.ts messages --topic orders --search paid --limit 20
+npx tsx src/cli.ts groups --search billing
+npx tsx src/cli.ts group-progress --group billing-consumers --topic orders
+```
+
+如果已经执行过构建，也可以直接运行：
+
+```bash
+node dist/cli.js topics
+node dist/cli.js messages --topic orders --search paid
+```
+
+## Skill
+
+可复用的 Skill 定义位于 `skills/kafka-cli/SKILL.md`，后续可以基于这个 CLI 继续扩展成面向 Codex、Claude 或其他 Agent 的 Kafka 操作能力。

package/dist/cli.d.ts

@@ -0,0 +1,2 @@
+#!/usr/bin/env node
+export {};

package/dist/cli.js

@@ -0,0 +1,96 @@
+#!/usr/bin/env node
+import { Command } from "commander";
+import { formatRows } from "./lib/format.js";
+import { KafkaCliService } from "./lib/kafka.js";
+const program = new Command();
+let service;
+function getService() {
+    service ??= new KafkaCliService();
+    return service;
+}
+program
+    .name("kafka-mcp")
+    .description("Kafka CLI for topics, messages, and consumer group inspection")
+    .version("0.1.0");
+program
+    .command("topics")
+    .description("List topics")
+    .option("-s, --search <keyword>", "Filter topic names")
+    .action(async (options) => {
+    const topics = await getService().listTopics(options.search);
+    console.log(formatRows(topics.map((topic) => ({ topic }))));
+});
+program
+    .command("topic")
+    .description("Show metadata for a topic")
+    .requiredOption("-t, --topic <topic>", "Topic name")
+    .action(async (options) => {
+    const topic = await getService().getTopicMetadata(options.topic);
+    if (!topic) {
+        throw new Error(`Topic not found: ${options.topic}`);
+    }
+    console.log(formatRows(topic.partitions.map((partition) => ({
+        partition: partition.partitionId,
+        leader: partition.leader,
+        replicas: partition.replicas.join(","),
+        isr: partition.isr.join(",")
+    }))));
+});
+program
+    .command("messages")
+    .description("Read messages from a topic")
+    .requiredOption("-t, --topic <topic>", "Topic name")
+    .option("-s, --search <keyword>", "Filter by topic, key, value, or headers")
+    .option("-l, --limit <number>", "Max matching messages", "10")
+    .option("--timeout-ms <number>", "Stop after N milliseconds", "5000")
+    .option("--latest", "Read from latest offsets instead of beginning")
+    .action(async (options) => {
+    const messages = await getService().consumeMessages({
+        topic: options.topic,
+        search: options.search,
+        limit: Number.parseInt(options.limit, 10),
+        timeoutMs: Number.parseInt(options.timeoutMs, 10),
+        fromBeginning: !options.latest
+    });
+    console.log(formatRows(messages.map((message) => ({
+        partition: message.partition,
+        offset: message.offset,
+        timestamp: message.timestamp ?? "",
+        key: message.key ?? "",
+        value: message.value ?? ""
+    }))));
+});
+program
+    .command("groups")
+    .description("List consumer groups")
+    .option("-s, --search <keyword>", "Filter group ids")
+    .action(async (options) => {
+    const groups = await getService().listConsumerGroups(options.search);
+    console.log(formatRows(groups.map((groupId) => ({ groupId }))));
+});
+program
+    .command("group-progress")
+    .description("Show a consumer group's progress on a topic")
+    .requiredOption("-g, --group <groupId>", "Consumer group id")
+    .requiredOption("-t, --topic <topic>", "Topic name")
+    .action(async (options) => {
+    const progress = await getService().getGroupTopicProgress(options.group, options.topic);
+    console.log(`group: ${progress.groupId}`);
+    console.log(`topic: ${progress.topic}`);
+    console.log(`state: ${progress.state}`);
+    console.log("");
+    console.log(formatRows(progress.partitions.map((partition) => ({
+        partition: partition.partition,
+        memberId: partition.memberId ?? "",
+        clientId: partition.clientId ?? "",
+        consumerHost: partition.consumerId ?? "",
+        committed: partition.committedOffset,
+        latest: partition.latestOffset,
+        lag: partition.lag ?? ""
+    }))));
+});
+program.parseAsync(process.argv).catch((error) => {
+    const message = error instanceof Error ? error.message : String(error);
+    console.error(`Error: ${message}`);
+    process.exitCode = 1;
+});

package/dist/lib/config.d.ts

@@ -0,0 +1,2 @@
+import type { KafkaCliConfig } from "./types.js";
+export declare function loadConfig(): KafkaCliConfig;

package/dist/lib/config.js

@@ -0,0 +1,60 @@
+import { existsSync, readFileSync } from "node:fs";
+import { homedir } from "node:os";
+import { join, resolve } from "node:path";
+const DEFAULT_CLIENT_ID = "kafka-mcp-cli";
+const CONFIG_CANDIDATES = [
+    resolve(process.cwd(), ".kafka-mcp.json"),
+    join(homedir(), ".config", "kafka-mcp", "config.json")
+];
+function isSupportedMechanism(value) {
+    return value === "plain" || value === "scram-sha-256" || value === "scram-sha-512";
+}
+function readConfigFile() {
+    for (const filePath of CONFIG_CANDIDATES) {
+        if (!existsSync(filePath)) {
+            continue;
+        }
+        const parsed = JSON.parse(readFileSync(filePath, "utf8"));
+        return parsed;
+    }
+    return {};
+}
+export function loadConfig() {
+    const fileConfig = readConfigFile();
+    const brokersFromEnv = process.env.KAFKA_BROKERS
+        ?.split(",")
+        .map((item) => item.trim())
+        .filter(Boolean);
+    const brokers = brokersFromEnv ?? fileConfig.brokers ?? [];
+    if (brokers.length === 0) {
+        throw new Error("Kafka brokers are missing. Set KAFKA_BROKERS or create .kafka-mcp.json / ~/.config/kafka-mcp/config.json.");
+    }
+    const config = {
+        brokers,
+        clientId: process.env.KAFKA_CLIENT_ID || fileConfig.clientId || DEFAULT_CLIENT_ID
+    };
+    const ssl = process.env.KAFKA_SSL;
+    if (ssl !== undefined) {
+        config.ssl = ssl === "true";
+    }
+    else if (fileConfig.ssl !== undefined) {
+        config.ssl = fileConfig.ssl;
+    }
+    const mechanism = process.env.KAFKA_SASL_MECHANISM ??
+        fileConfig.sasl?.mechanism;
+    const username = process.env.KAFKA_SASL_USERNAME ??
+        fileConfig.sasl?.username;
+    const password = process.env.KAFKA_SASL_PASSWORD ??
+        fileConfig.sasl?.password;
+    if (mechanism && username && password) {
+        if (!isSupportedMechanism(mechanism)) {
+            throw new Error(`Unsupported SASL mechanism: ${mechanism}`);
+        }
+        config.sasl = {
+            mechanism,
+            username,
+            password
+        };
+    }
+    return config;
+}

package/dist/lib/format.d.ts

@@ -0,0 +1 @@
+export declare function formatRows(rows: Array<Record<string, string | number | null>>): string;

package/dist/lib/format.js

@@ -0,0 +1,16 @@
+export function formatRows(rows) {
+    if (rows.length === 0) {
+        return "No results.";
+    }
+    const headers = Object.keys(rows[0]);
+    const widths = headers.map((header) => Math.max(header.length, ...rows.map((row) => String(row[header] ?? "").length)));
+    const formatLine = (values) => values.map((value, index) => value.padEnd(widths[index])).join("  ");
+    const lines = [
+        formatLine(headers),
+        formatLine(widths.map((width) => "-".repeat(width))),
+    ];
+    for (const row of rows) {
+        lines.push(formatLine(headers.map((header) => String(row[header] ?? ""))));
+    }
+    return lines.join("\n");
+}

package/dist/lib/kafka.d.ts

@@ -0,0 +1,17 @@
+import { type ITopicMetadata } from "kafkajs";
+import type { GroupTopicProgress, TopicMessage } from "./types.js";
+export declare class KafkaCliService {
+    private readonly kafka;
+    constructor();
+    listTopics(search?: string): Promise<string[]>;
+    getTopicMetadata(topic: string): Promise<ITopicMetadata | undefined>;
+    consumeMessages(options: {
+        topic: string;
+        search?: string;
+        limit: number;
+        timeoutMs: number;
+        fromBeginning: boolean;
+    }): Promise<TopicMessage[]>;
+    listConsumerGroups(search?: string): Promise<string[]>;
+    getGroupTopicProgress(groupId: string, topic: string): Promise<GroupTopicProgress>;
+}

package/dist/lib/kafka.js

@@ -0,0 +1,210 @@
+import { AssignerProtocol, Kafka } from "kafkajs";
+import { loadConfig } from "./config.js";
+import { filterTopics, matchMessage } from "./search.js";
+function decodeValue(value) {
+    if (typeof value === "string") {
+        return value;
+    }
+    return value ? value.toString("utf8") : null;
+}
+function decodeHeaderValue(value) {
+    if (value === undefined) {
+        return "";
+    }
+    if (Array.isArray(value)) {
+        return value.map((item) => decodeValue(item) ?? "").join(",");
+    }
+    return decodeValue(value) ?? "";
+}
+function toBigIntOrNull(value) {
+    if (!value || value === "-1") {
+        return null;
+    }
+    return BigInt(value);
+}
+export class KafkaCliService {
+    kafka;
+    constructor() {
+        const config = loadConfig();
+        this.kafka = new Kafka({
+            clientId: config.clientId,
+            brokers: config.brokers,
+            ssl: config.ssl,
+            sasl: toKafkaSasl(config.sasl)
+        });
+    }
+    async listTopics(search) {
+        const admin = this.kafka.admin();
+        await admin.connect();
+        try {
+            const metadata = await admin.fetchTopicMetadata();
+            return filterTopics(metadata.topics.map((topic) => topic.name).sort((left, right) => left.localeCompare(right)), search);
+        }
+        finally {
+            await admin.disconnect();
+        }
+    }
+    async getTopicMetadata(topic) {
+        const admin = this.kafka.admin();
+        await admin.connect();
+        try {
+            const metadata = await admin.fetchTopicMetadata({ topics: [topic] });
+            return metadata.topics.find((item) => item.name === topic);
+        }
+        finally {
+            await admin.disconnect();
+        }
+    }
+    async consumeMessages(options) {
+        const consumer = this.kafka.consumer({
+            groupId: `kafka-mcp-cli-preview-${Date.now()}`
+        });
+        const matches = [];
+        await consumer.connect();
+        await consumer.subscribe({ topic: options.topic, fromBeginning: options.fromBeginning });
+        const timer = setTimeout(async () => {
+            await consumer.stop();
+        }, options.timeoutMs);
+        try {
+            await consumer.run({
+                autoCommit: false,
+                eachMessage: async ({ topic, partition, message }) => {
+                    const normalized = {
+                        topic,
+                        partition,
+                        offset: message.offset,
+                        timestamp: message.timestamp,
+                        key: decodeMessageValue(message, "key"),
+                        value: decodeMessageValue(message, "value"),
+                        headers: Object.fromEntries(Object.entries(message.headers ?? {}).map(([header, value]) => [
+                            header,
+                            decodeHeaderValue(value)
+                        ]))
+                    };
+                    if (matchMessage(normalized, options.search)) {
+                        matches.push(normalized);
+                    }
+                    if (matches.length >= options.limit) {
+                        await consumer.stop();
+                    }
+                }
+            });
+        }
+        finally {
+            clearTimeout(timer);
+            await consumer.disconnect();
+        }
+        return matches;
+    }
+    async listConsumerGroups(search) {
+        const admin = this.kafka.admin();
+        await admin.connect();
+        try {
+            const groups = await admin.listGroups();
+            return groups.groups
+                .map((group) => group.groupId)
+                .filter((groupId) => !search || groupId.toLowerCase().includes(search.toLowerCase()))
+                .sort((left, right) => left.localeCompare(right));
+        }
+        finally {
+            await admin.disconnect();
+        }
+    }
+    async getGroupTopicProgress(groupId, topic) {
+        const admin = this.kafka.admin();
+        await admin.connect();
+        try {
+            const groupDescriptions = await admin.describeGroups([groupId]);
+            const description = groupDescriptions.groups[0];
+            if (!description) {
+                throw new Error(`Consumer group not found: ${groupId}`);
+            }
+            const topicOffsets = await admin.fetchTopicOffsets(topic);
+            const groupOffsets = await admin.fetchOffsets({ groupId, topics: [topic] });
+            const groupTopicOffsets = groupOffsets[0];
+            const topicAssignments = decodeAssignmentsForTopic(description, topic);
+            const assignmentMap = buildAssignmentMap(description, topicAssignments);
+            const partitions = topicOffsets.map((partitionOffset) => {
+                const committed = groupTopicOffsets?.partitions.find((partition) => partition.partition === partitionOffset.partition);
+                const latest = toBigIntOrNull(partitionOffset.high);
+                const committedOffset = toBigIntOrNull(committed?.offset ?? "-1");
+                const lag = latest !== null && committedOffset !== null && latest >= committedOffset
+                    ? (latest - committedOffset).toString()
+                    : null;
+                const member = assignmentMap.get(partitionOffset.partition);
+                return {
+                    partition: partitionOffset.partition,
+                    memberId: member?.memberId ?? null,
+                    clientId: member?.clientId ?? null,
+                    consumerId: member?.consumerId ?? null,
+                    committedOffset: committed?.offset ?? "-1",
+                    latestOffset: partitionOffset.high,
+                    lag
+                };
+            });
+            return {
+                groupId,
+                topic,
+                state: description.state,
+                members: description.members.map((member) => ({
+                    memberId: member.memberId,
+                    clientId: member.clientId,
+                    consumerId: member.clientHost,
+                    partitions: topicAssignments.get(member.memberId) ?? []
+                })),
+                partitions
+            };
+        }
+        finally {
+            await admin.disconnect();
+        }
+    }
+}
+function buildAssignmentMap(description, decodedAssignments) {
+    const assignment = new Map();
+    for (const member of description.members) {
+        for (const partition of decodedAssignments.get(member.memberId) ?? []) {
+            assignment.set(partition, {
+                memberId: member.memberId,
+                clientId: member.clientId,
+                consumerId: member.clientHost
+            });
+        }
+    }
+    return assignment;
+}
+function decodeAssignmentsForTopic(description, topic) {
+    const assignments = new Map();
+    for (const member of description.members) {
+        const decoded = AssignerProtocol.MemberAssignment.decode(member.memberAssignment);
+        assignments.set(member.memberId, decoded?.assignment[topic] ?? []);
+    }
+    return assignments;
+}
+function decodeMessageValue(message, field) {
+    return decodeValue(message[field]);
+}
+function toKafkaSasl(sasl) {
+    if (!sasl) {
+        return undefined;
+    }
+    if (sasl.mechanism === "plain") {
+        return {
+            mechanism: "plain",
+            username: sasl.username,
+            password: sasl.password
+        };
+    }
+    if (sasl.mechanism === "scram-sha-256") {
+        return {
+            mechanism: "scram-sha-256",
+            username: sasl.username,
+            password: sasl.password
+        };
+    }
+    return {
+        mechanism: "scram-sha-512",
+        username: sasl.username,
+        password: sasl.password
+    };
+}

package/dist/lib/search.d.ts

@@ -0,0 +1,4 @@
+import type { TopicMessage } from "./types.js";
+export declare function includesInsensitive(input: string, search?: string): boolean;
+export declare function filterTopics(topics: string[], search?: string): string[];
+export declare function matchMessage(message: TopicMessage, search?: string): boolean;

package/dist/lib/search.js

@@ -0,0 +1,21 @@
+export function includesInsensitive(input, search) {
+    if (!search) {
+        return true;
+    }
+    return input.toLowerCase().includes(search.toLowerCase());
+}
+export function filterTopics(topics, search) {
+    return topics.filter((topic) => includesInsensitive(topic, search));
+}
+export function matchMessage(message, search) {
+    if (!search) {
+        return true;
+    }
+    const haystacks = [
+        message.topic,
+        message.key ?? "",
+        message.value ?? "",
+        JSON.stringify(message.headers)
+    ];
+    return haystacks.some((item) => includesInsensitive(item, search));
+}

package/dist/lib/types.d.ts

@@ -0,0 +1,40 @@
+export interface KafkaCliConfig {
+    brokers: string[];
+    clientId: string;
+    ssl?: boolean;
+    sasl?: {
+        mechanism: "plain" | "scram-sha-256" | "scram-sha-512";
+        username: string;
+        password: string;
+    };
+}
+export interface TopicMessage {
+    topic: string;
+    partition: number;
+    offset: string;
+    timestamp?: string;
+    key: string | null;
+    value: string | null;
+    headers: Record<string, string>;
+}
+export interface GroupPartitionProgress {
+    partition: number;
+    memberId: string | null;
+    clientId: string | null;
+    consumerId: string | null;
+    committedOffset: string;
+    latestOffset: string;
+    lag: string | null;
+}
+export interface GroupTopicProgress {
+    groupId: string;
+    topic: string;
+    state: string;
+    members: Array<{
+        memberId: string;
+        clientId: string;
+        consumerId: string;
+        partitions: number[];
+    }>;
+    partitions: GroupPartitionProgress[];
+}

package/dist/lib/types.js

@@ -0,0 +1 @@
+export {};

package/package.json

@@ -0,0 +1,44 @@
+{
+  "name": "kafka-mcp",
+  "version": "0.1.0",
+  "description": "Kafka inspection CLI with topic, message, and consumer group utilities",
+  "type": "module",
+  "main": "./dist/cli.js",
+  "types": "./dist/cli.d.ts",
+  "bin": {
+    "kafka-mcp": "./dist/cli.js"
+  },
+  "files": [
+    "dist",
+    "README.md",
+    "skills"
+  ],
+  "scripts": {
+    "build": "tsc -p tsconfig.json",
+    "dev": "tsx src/cli.ts",
+    "test": "vitest run",
+    "lint": "tsc --noEmit -p tsconfig.vitest.json",
+    "prepublishOnly": "npm run build && npm run lint && npm test"
+  },
+  "keywords": [
+    "kafka",
+    "cli",
+    "consumer-group",
+    "topic",
+    "codex-skill"
+  ],
+  "engines": {
+    "node": ">=18.18"
+  },
+  "license": "MIT",
+  "dependencies": {
+    "commander": "^14.0.0",
+    "kafkajs": "^2.2.4"
+  },
+  "devDependencies": {
+    "@types/node": "^24.6.0",
+    "tsx": "^4.20.5",
+    "typescript": "^5.9.3",
+    "vitest": "^3.2.4"
+  }
+}

package/skills/kafka-cli/SKILL.md

@@ -0,0 +1,29 @@
+---
+name: kafka-cli-inspector
+description: Use this skill when you need to inspect Kafka topics, preview messages, search Kafka data, or analyze a consumer group's progress for a topic through the local kafka-mcp CLI.
+---
+
+# Kafka CLI Inspector
+
+Use this skill when a user wants Kafka read-only inspection from the local workspace.
+
+## Commands
+
+- `npx tsx src/cli.ts topics --search <keyword>` lists topics and supports fuzzy substring search.
+- `npx tsx src/cli.ts topic --topic <topic>` shows partition metadata for a topic.
+- `npx tsx src/cli.ts messages --topic <topic> --search <keyword> --limit <n>` previews messages from a topic and filters by topic, key, value, or headers.
+- `npx tsx src/cli.ts groups --search <keyword>` lists consumer groups and supports search.
+- `npx tsx src/cli.ts group-progress --group <groupId> --topic <topic>` shows committed offsets, latest offsets, lag, and the current member assignment per partition.
+
+## Workflow
+
+1. Check that Kafka brokers are configured through `.kafka-mcp.json`, `~/.config/kafka-mcp/config.json`, or `KAFKA_BROKERS`.
+2. Use `topics` or `groups` first if the exact topic or consumer group is unknown.
+3. Use `messages` for lightweight investigation. Prefer small limits and a search term.
+4. Use `group-progress` when the task is about backlog, lag, stuck partitions, or which consumer currently owns a partition.
+
+## Output guidance
+
+- Summarize only the relevant topics, messages, or lagging partitions.
+- Call out when a search returns no results rather than dumping empty tables.
+- When reporting lag, mention the topic, partition, committed offset, latest offset, and lag value.