mcp-multitool 0.1.11 → 0.1.12

package/README.md

@@ -172,47 +172,44 @@ moveFileOrDir  from="config.json"  to="dest/"  overwrite=true
 
 ### `readLogFile`
 
-Compress a log file using semantic pattern extraction (60-90% token reduction). Creates stateful drains for incremental reads. Use `flushLogFile` to release.
+Compress a log file using semantic pattern extraction. Groups similar lines into templates with `<*>` wildcards for variable parts. **Stateless** — each call processes the file fresh.
 
-**Stateful drains:** On first call for a file, creates a stateful drain. Subsequent calls append only new lines to the existing drain, preserving template IDs. This enables incremental log analysis as files grow. When any drain is active, a dynamic `flushLogFile` tool appears to release drains.
+**Content-hashed template IDs:** Template IDs are 12-character base64URL hashes derived from the pattern itself. The same pattern **always** gets the same ID, regardless of file order or when you call the tool. This means drill-down always works if the pattern still exists.
 
-| Parameter      | Type      | Required | Description                                      |
-| -------------- | --------- | -------- | ------------------------------------------------ |
-| `path`         | `string`  | ✅       | Path to the log file.                            |
-| `format`       | `string`  | ✅       | Output format: `summary`, `detailed`, or `json`. |
-| `depth`        | `integer` | ✅       | Parse tree depth (2-8).                          |
-| `simThreshold` | `number`  | ✅       | Similarity threshold (0-1).                      |
-| `tail`         | `integer` | —        | Last N lines (first read only).                  |
-| `head`         | `integer` | —        | First N lines (first read only).                 |
-| `grep`         | `string`  | —        | Regex filter for lines.                          |
+| Parameter      | Type      | Required | Description                                                        |
+| -------------- | --------- | -------- | ------------------------------------------------------------------ |
+| `path`         | `string`  | ✅       | Path to the log file.                                              |
+| `simThreshold` | `number`  | ✅       | Similarity threshold (0-1). Lower values group more aggressively.  |
+| `tail`         | `integer` | —        | Last N lines.                                                      |
+| `head`         | `integer` | —        | First N lines.                                                     |
+| `grep`         | `string`  | —        | Regex filter for lines before compression.                         |
+| `templateId`   | `string`  | —        | Drill into a specific template by its hash ID for sample captures. |
 
-**Response:** Compressed log summary showing unique templates and occurrence counts.
+**Response:** Compressed log summary showing template IDs, occurrence counts, and patterns with `<*>` wildcards.
 
 **Examples:**
 
 ```
-readLogFile  path="/var/log/app.log"  format="summary"  depth=4  simThreshold=0.4
-readLogFile  path="./logs/server.log"  format="detailed"  depth=4  simThreshold=0.4  tail=1000
-readLogFile  path="app.log"  format="json"  depth=6  simThreshold=0.3  grep="ERROR|WARN"
+readLogFile  path="/var/log/app.log"  simThreshold=0.4
+readLogFile  path="./logs/server.log"  simThreshold=0.4  tail=1000
+readLogFile  path="app.log"  simThreshold=0.3  grep="ERROR|WARN"
+readLogFile  path="app.log"  simThreshold=0.4  templateId="aB3x_Yz7Q2Kf"
 ```
 
----
+<details>
+<summary><strong>Algorithm Notes</strong></summary>
 
-### `flushLogFile` (dynamic)
+This tool implements the [Drain algorithm](https://jiemingzhu.github.io/pub/pjhe_icws2017.pdf) (He et al., ICWS 2017) for online log parsing with content-hashed template IDs:
 
-Release a log drain to free memory. Next `readLogFile` creates fresh drain. **This tool only appears when at least one drain is active.** When the last drain is flushed, the tool is automatically removed.
+**Tree routing:** Lines are routed by token count → first N tokens (default N=2, configurable via `readLogFileRoutingDepth`). This deterministic routing ensures lines with different prefixes are never compared, preventing "cross-contamination" between unrelated patterns.
 
-| Parameter | Type     | Required | Description                    |
-| --------- | -------- | -------- | ------------------------------ |
-| `path`    | `string` | ✅       | Path to the log file to flush. |
+**Content-hashed template IDs:** Template IDs are 12-character base64URL hashes derived from the pattern itself. The same pattern always produces the same ID, enabling stateless drill-down across calls.
 
-**Response:** `"Flushed {filename}. Released N templates from M lines."`
+**Wildcard matching:** Variable tokens (timestamps, IDs, numbers) are replaced with `<*>` wildcards. Tokens starting with digits or matching hex patterns are automatically routed to wildcard buckets.
 
-**Example:**
+**Tuning:** If you see all-wildcard templates (e.g., `<*> <*> <*> <*>`), try increasing `readLogFileRoutingDepth` to 3 or 4. If you hit memory issues on very large logs, reduce it to 1.
 
-```
-flushLogFile  path="/var/log/app.log"
-```
+</details>
 
 ---
 
@@ -257,18 +254,19 @@ wait  durationSeconds=1  reason="animation to complete"
 
 ## Environment Variables
 
-| Variable                 | Default | Description                                                                                                    |
-| ------------------------ | ------- | -------------------------------------------------------------------------------------------------------------- |
-| `waitMaxDurationSeconds` | `300`   | Override the maximum allowed `durationSeconds`. Must be a positive number. Server refuses to start if invalid. |
-| `readLogFileTimeoutMs`   | `5000`  | Override the timeout for `readLogFile` processing in milliseconds. Server refuses to start if invalid.         |
-| `astGrepSearch`          | _(on)_  | Set to `"false"` to disable the `astGrepSearch` tool at startup.                                               |
-| `checkFileOrDir`         | _(on)_  | Set to `"false"` to disable the `checkFileOrDir` tool at startup.                                              |
-| `cloneFileOrDir`         | _(on)_  | Set to `"false"` to disable the `cloneFileOrDir` tool at startup.                                              |
-| `deleteFileOrDir`        | _(on)_  | Set to `"false"` to disable the `deleteFileOrDir` tool at startup.                                             |
-| `moveFileOrDir`          | _(on)_  | Set to `"false"` to disable the `moveFileOrDir` tool at startup.                                               |
-| `readLogFile`            | _(on)_  | Set to `"false"` to disable the `readLogFile` tool at startup.                                                 |
-| `renameFileOrDir`        | _(on)_  | Set to `"false"` to disable the `renameFileOrDir` tool at startup.                                             |
-| `wait`                   | _(on)_  | Set to `"false"` to disable the `wait` tool at startup.                                                        |
+| Variable                  | Default | Description                                                                                                                     |
+| ------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------- |
+| `waitMaxDurationSeconds`  | `300`   | Override the maximum allowed `durationSeconds`. Must be a positive number. Server refuses to start if invalid.                  |
+| `readLogFileTimeoutMs`    | `5000`  | Override the timeout for `readLogFile` processing in milliseconds. Server refuses to start if invalid.                          |
+| `readLogFileRoutingDepth` | `2`     | Tree routing depth (1-5). Higher values isolate more but increase memory. Tune if you see all-wildcard templates or OOM errors. |
+| `astGrepSearch`           | _(on)_  | Set to `"false"` to disable the `astGrepSearch` tool at startup.                                                                |
+| `checkFileOrDir`          | _(on)_  | Set to `"false"` to disable the `checkFileOrDir` tool at startup.                                                               |
+| `cloneFileOrDir`          | _(on)_  | Set to `"false"` to disable the `cloneFileOrDir` tool at startup.                                                               |
+| `deleteFileOrDir`         | _(on)_  | Set to `"false"` to disable the `deleteFileOrDir` tool at startup.                                                              |
+| `moveFileOrDir`           | _(on)_  | Set to `"false"` to disable the `moveFileOrDir` tool at startup.                                                                |
+| `readLogFile`             | _(on)_  | Set to `"false"` to disable the `readLogFile` tool at startup.                                                                  |
+| `renameFileOrDir`         | _(on)_  | Set to `"false"` to disable the `renameFileOrDir` tool at startup.                                                              |
+| `wait`                    | _(on)_  | Set to `"false"` to disable the `wait` tool at startup.                                                                         |
 
 ### Disabling Individual Tools
 

package/dist/tools/readLogFile.js

@@ -1,7 +1,7 @@
+import { createHash } from "node:crypto";
 import { readFile } from "node:fs/promises";
-import { resolve, basename } from "node:path";
+import { resolve } from "node:path";
 import { z } from "zod";
-import { createDrain } from "logpare";
 const timeoutMs = (() => {
     const env = process.env.readLogFileTimeoutMs;
     if (!env)
@@ -13,33 +13,181 @@ const timeoutMs = (() => {
     }
     return n;
 })();
-const drains = new Map();
-let flushTool = null;
+const routingDepth = (() => {
+    const env = process.env.readLogFileRoutingDepth;
+    if (!env)
+        return 2;
+    const n = Number(env);
+    if (!Number.isInteger(n) || n < 1 || n > 5) {
+        process.stderr.write(`Invalid readLogFileRoutingDepth: "${env}". Must be 1-5.\n`);
+        process.exit(1);
+    }
+    return n;
+})();
+const WILDCARD = "<*>";
+const WILDCARD_KEY = "<WILDCARD>";
+const MAX_SAMPLES = 3;
+function tokenize(line) {
+    return line.split(/(\s+|[{}()\[\],:;="'`<>])/g).filter((t) => t.trim());
+}
+function looksLikeVariable(token) {
+    if (token === WILDCARD)
+        return true;
+    const first = token.charAt(0);
+    if (first >= "0" && first <= "9")
+        return true;
+    if (/^[0-9a-fA-F]+$/.test(token) && token.length > 8)
+        return true;
+    return false;
+}
+function similarity(tokens, template) {
+    if (tokens.length !== template.length)
+        return 0;
+    let matches = 0;
+    for (let i = 0; i < tokens.length; i++) {
+        if (template[i] === WILDCARD || tokens[i] === template[i])
+            matches++;
+    }
+    return matches / tokens.length;
+}
+function mergeTokens(tokens, template) {
+    return template.map((t, i) => t === WILDCARD || t !== tokens[i] ? WILDCARD : t);
+}
+function extractVariables(tokens, template) {
+    const vars = [];
+    for (let i = 0; i < template.length; i++) {
+        if (template[i] === WILDCARD && tokens[i])
+            vars.push(tokens[i]);
+    }
+    return vars;
+}
+function tokensToPattern(tokens) {
+    return tokens.join(" ");
+}
+function getRouteKey(token) {
+    if (!token)
+        return WILDCARD_KEY;
+    return looksLikeVariable(token) ? WILDCARD_KEY : token;
+}
+function getRouteKeys(tokens, depth) {
+    const keys = [];
+    for (let i = 0; i < depth; i++) {
+        keys.push(getRouteKey(tokens[i]));
+    }
+    return keys;
+}
+function createNode() {
+    return { children: new Map(), templates: [] };
+}
+/**
+ * Tree-based Drain algorithm with configurable routing depth.
+ * Structure: root → length → token[0] → token[1] → ... → token[depth-1] → templates[]
+ * Deeper routing prevents cross-contamination but increases memory usage.
+ */
+class DrainTree {
+    simThreshold;
+    depth;
+    root = new Map();
+    constructor(simThreshold, depth) {
+        this.simThreshold = simThreshold;
+        this.depth = depth;
+    }
+    navigate(length, keys, create) {
+        let lengthNode = this.root.get(length);
+        if (!lengthNode) {
+            if (!create)
+                return undefined;
+            lengthNode = createNode();
+            this.root.set(length, lengthNode);
+        }
+        let current = lengthNode;
+        for (const key of keys) {
+            let child = current.children.get(key);
+            if (!child) {
+                if (!create)
+                    return undefined;
+                child = createNode();
+                current.children.set(key, child);
+            }
+            current = child;
+        }
+        return current;
+    }
+    addLine(line) {
+        const tokens = tokenize(line);
+        if (!tokens.length)
+            return;
+        const length = tokens.length;
+        const keys = getRouteKeys(tokens, this.depth);
+        const node = this.navigate(length, keys, false);
+        let bestMatch = null;
+        let bestSim = 0;
+        if (node) {
+            for (const template of node.templates) {
+                const sim = similarity(tokens, template.tokens);
+                if (sim >= this.simThreshold && sim > bestSim) {
+                    bestSim = sim;
+                    bestMatch = template;
+                }
+            }
+        }
+        if (bestMatch) {
+            bestMatch.tokens = mergeTokens(tokens, bestMatch.tokens);
+            bestMatch.pattern = tokensToPattern(bestMatch.tokens);
+            bestMatch.count++;
+            if (bestMatch.samples.length < MAX_SAMPLES) {
+                bestMatch.samples.push(extractVariables(tokens, bestMatch.tokens));
+            }
+        }
+        else {
+            const targetNode = this.navigate(length, keys, true);
+            targetNode.templates.push({
+                tokens,
+                pattern: tokensToPattern(tokens),
+                count: 1,
+                samples: [],
+            });
+        }
+    }
+    getTemplates() {
+        const result = [];
+        const collectFromNode = (node) => {
+            result.push(...node.templates);
+            for (const child of node.children.values()) {
+                collectFromNode(child);
+            }
+        };
+        for (const lengthNode of this.root.values()) {
+            collectFromNode(lengthNode);
+        }
+        return result;
+    }
+}
+function compress(lines, simThreshold) {
+    const tree = new DrainTree(simThreshold, routingDepth);
+    for (const line of lines) {
+        tree.addLine(line);
+    }
+    return tree.getTemplates();
+}
+// --- MCP Tool ---
+function hashTemplateId(pattern) {
+    return createHash("sha256").update(pattern).digest("base64url").slice(0, 12);
+}
 const schema = z.object({
     path: z.string().min(1).describe("Path to the log file."),
-    format: z.enum(["summary", "detailed", "json"]).describe("Output format."),
-    depth: z.number().int().min(2).max(8).describe("Parse tree depth (2-8)."),
     simThreshold: z
         .number()
         .min(0)
         .max(1)
-        .describe("Similarity threshold (0-1)."),
-    tail: z
-        .number()
-        .int()
-        .min(1)
-        .optional()
-        .describe("Last N lines (first read only)."),
-    head: z
-        .number()
-        .int()
-        .min(1)
-        .optional()
-        .describe("First N lines (first read only)."),
+        .describe("Similarity threshold (0-1). Lower = more aggressive grouping."),
+    tail: z.number().int().min(1).optional().describe("Last N lines."),
+    head: z.number().int().min(1).optional().describe("First N lines."),
     grep: z.string().optional().describe("Regex filter for lines."),
-});
-const flushSchema = z.object({
-    path: z.string().min(1).describe("Path to the log file to flush."),
+    templateId: z
+        .string()
+        .optional()
+        .describe("Drill into a specific template by its hash ID."),
 });
 const ok = (text) => ({ content: [{ type: "text", text }] });
 const err = (text) => ({
@@ -48,103 +196,78 @@ const err = (text) => ({
 });
 export function register(server) {
     server.registerTool("readLogFile", {
-        description: "Compress a log file using semantic pattern extraction (60-90% reduction). Creates stateful drains for incremental reads. Use flushLogFile to release.",
+        description: "Compress a log file using the Drain algorithm for semantic pattern extraction. Groups similar lines into templates. Stateless — each call processes the file fresh. Template IDs are content-hashed so the same pattern always has the same ID.",
         inputSchema: schema,
         annotations: { readOnlyHint: true, openWorldHint: false },
     }, async (input) => {
         try {
-            return ok(await Promise.race([processLog(server, input), timeout()]));
+            return ok(await Promise.race([processLog(input), timeout()]));
         }
         catch (e) {
             return err(String(e));
         }
     });
 }
-async function processLog(server, input) {
+async function processLog(input) {
     const path = resolve(process.cwd(), input.path);
-    let state = drains.get(path);
     if (input.head && input.tail) {
         return "Cannot use both head and tail.";
     }
-    if (state &&
-        (state.depth !== input.depth || state.simThreshold !== input.simThreshold)) {
-        return `Error: Drain exists with depth=${state.depth}, simThreshold=${state.simThreshold}. Flush first.`;
-    }
     const content = await readFile(path, "utf-8");
     let lines = content.split(/\r?\n/).filter(Boolean);
-    if (!state) {
-        const wasEmpty = drains.size === 0;
-        if (input.head)
-            lines = lines.slice(0, input.head);
-        else if (input.tail)
-            lines = lines.slice(-input.tail);
-        if (input.grep)
-            lines = lines.filter((l) => new RegExp(input.grep).test(l));
-        if (!lines.length)
-            return "No log lines to process.";
-        const drain = createDrain({
-            depth: input.depth,
-            simThreshold: input.simThreshold,
-        });
-        drain.addLogLines(lines);
-        state = {
-            drain,
-            lastLine: lines.length,
-            depth: input.depth,
-            simThreshold: input.simThreshold,
-        };
-        drains.set(path, state);
-        if (wasEmpty)
-            registerFlush(server);
-        return `${state.drain.getResult(input.format).formatted}\n\n[New drain. ${state.lastLine} lines. Use flushLogFile when done.]`;
-    }
-    const newLines = lines.slice(state.lastLine);
-    if (!newLines.length) {
-        return `${state.drain.getResult(input.format).formatted}\n\n[No new lines. Total: ${state.lastLine}]`;
+    if (input.head)
+        lines = lines.slice(0, input.head);
+    else if (input.tail)
+        lines = lines.slice(-input.tail);
+    if (input.grep)
+        lines = lines.filter((l) => new RegExp(input.grep).test(l));
+    if (!lines.length)
+        return "No log lines to process.";
+    const templates = compress(lines, input.simThreshold);
+    const templateMap = buildTemplateMap(templates);
+    if (input.templateId) {
+        const template = templateMap.get(input.templateId);
+        if (!template) {
+            const available = [...templateMap.keys()].slice(0, 10).join(", ");
+            return `Template "${input.templateId}" not found. Available: ${available}${templateMap.size > 10 ? ` (+${templateMap.size - 10} more)` : ""}`;
+        }
+        return formatDrillDown(template);
     }
-    const filtered = input.grep ?
-        newLines.filter((l) => new RegExp(input.grep).test(l))
-        : newLines;
-    if (filtered.length)
-        state.drain.addLogLines(filtered);
-    state.lastLine = lines.length;
-    return `${state.drain.getResult(input.format).formatted}\n\n[+${newLines.length} lines. Total: ${state.lastLine}]`;
-}
-function registerFlush(server) {
-    if (flushTool)
-        return;
-    try {
-        flushTool = server.registerTool("flushLogFile", {
-            description: "Release a log drain to free memory. Next readLogFile creates fresh drain.",
-            inputSchema: flushSchema,
-            annotations: { destructiveHint: false, idempotentHint: true },
-        }, async (input) => {
-            try {
-                const path = resolve(process.cwd(), input.path);
-                const state = drains.get(path);
-                if (!state) {
-                    if (!drains.size)
-                        return ok("No active drains.");
-                    return ok(`No drain for "${basename(path)}". Active: ${[...drains.keys()].map((p) => basename(p)).join(", ")}`);
-                }
-                const { totalClusters, lastLine } = {
-                    totalClusters: state.drain.totalClusters,
-                    lastLine: state.lastLine,
-                };
-                drains.delete(path);
-                if (!drains.size && flushTool) {
-                    flushTool.remove();
-                    flushTool = null;
-                }
-                return ok(`Flushed ${basename(path)}. Released ${totalClusters} templates from ${lastLine} lines.`);
-            }
-            catch (e) {
-                return err(String(e));
-            }
+    return formatOverview(templateMap, lines.length);
+}
+function buildTemplateMap(templates) {
+    const map = new Map();
+    for (const t of templates) {
+        const id = hashTemplateId(t.pattern);
+        map.set(id, {
+            id,
+            pattern: t.pattern,
+            count: t.count,
+            samples: t.samples,
         });
-        server.sendToolListChanged();
     }
-    catch { }
+    return map;
+}
+function formatOverview(templateMap, lineCount) {
+    const sorted = [...templateMap.values()].sort((a, b) => b.count - a.count);
+    const reduction = Math.round((1 - templateMap.size / lineCount) * 100);
+    const header = `=== Log Compression ===\n${lineCount} lines → ${templateMap.size} templates (${reduction}% reduction)\n`;
+    const top20 = sorted.slice(0, 20);
+    const topLines = top20
+        .map((t) => `${t.id} [${t.count}x] ${t.pattern}`)
+        .join("\n");
+    const remaining = sorted.length - 20;
+    const footer = remaining > 0 ? `\n... and ${remaining} more templates` : "";
+    return `${header}\n${topLines}${footer}`;
+}
+function formatDrillDown(template) {
+    const header = `Template: ${template.id}\nPattern: ${template.pattern}\nMatches: ${template.count}\n`;
+    if (!template.samples.length)
+        return header;
+    const samples = template.samples
+        .map((vars, i) => `  ${i + 1}. Variables: ${vars.join(", ")}`)
+        .join("\n");
+    return `${header}\nSample variable captures:\n${samples}`;
 }
 function timeout() {
     return new Promise((_, rej) => setTimeout(() => rej(new Error(`Timeout: ${timeoutMs}ms`)), timeoutMs));

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mcp-multitool",
-  "version": "0.1.11",
+  "version": "0.1.12",
   "description": "MCP server with file operations (delete, move) and timing utilities.",
   "license": "MIT",
   "type": "module",
@@ -31,7 +31,6 @@
   "dependencies": {
     "@ast-grep/napi": "^0.42.1",
     "@modelcontextprotocol/sdk": "1.29.0",
-    "logpare": "^0.1.0",
     "zod": "^3.24.0"
   },
   "devDependencies": {