context-mode 0.5.0 → 0.5.2

package/.claude-plugin/marketplace.json

@@ -13,7 +13,7 @@
       "name": "context-mode",
       "source": "./",
       "description": "Claude Code MCP plugin that saves 94% of your context window. Sandboxed code execution in 10 languages, FTS5 knowledge base with BM25 ranking, and smart truncation.",
-      "version": "0.5.0",
+      "version": "0.5.2",
       "author": {
         "name": "Mert Koseoğlu"
       },

package/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "context-mode",
-  "version": "0.5.0",
+  "version": "0.5.2",
   "description": "Claude Code MCP plugin that saves 94% of your context window. Sandboxed code execution in 10 languages, FTS5 knowledge base with BM25 ranking, and smart truncation.",
   "author": {
     "name": "Mert Koseoğlu",

package/README.md

@@ -75,7 +75,7 @@ Claude calls: execute({ language: "shell", code: "gh pr list --json title,state
 Returns: "3"                  ← 2 bytes instead of 8KB JSON
 ```
 
-**Intent-driven search** (v0.5.0): When you provide an `intent` parameter and output exceeds 5KB, Context Mode uses BM25 search to return only the relevant sections — instead of blind head/tail truncation.
+**Intent-driven search** (v0.5.0): When you provide an `intent` parameter and output exceeds 5KB, Context Mode uses BM25 search to return only the relevant sections matching your intent.
 
 ```
 Claude calls: execute({
@@ -83,7 +83,7 @@ Claude calls: execute({
   code: "cat /var/log/app.log",
   intent: "connection refused database error"
 })
-Returns: only the 3 matching log sections (1.5KB) ← instead of 100KB truncated log
+Returns: only the 3 matching log sections (1.5KB) ← instead of 100KB raw log
 ```
 
 Authenticated CLIs work out of the box — `gh`, `aws`, `gcloud`, `kubectl`, `docker` credentials are passed through securely. Bun auto-detected for 3-5x faster JS/TS.
@@ -163,7 +163,7 @@ Use instead of WebFetch or Context7 when you need documentation — index once,
                                          │  │ • 10 language runtimes │  │
                                          │  │ • Sandboxed subprocess │  │
                                          │  │ • Auth passthrough     │  │
-                                         │  │ • Smart truncation     │  │
+                                         │  │ • Intent-driven search │  │
                                          │  └────────────────────────┘  │
                                          │                              │
                                          │  ┌────────────────────────┐  │
@@ -213,42 +213,29 @@ ORDER BY rank LIMIT 3;
 
 **Lazy singleton:** Database created only when `index` or `search` is first called — zero overhead for sessions that don't use it.
 
-### Smart Truncation
-
-When subprocess output exceeds the 100KB buffer, Context Mode preserves both head and tail:
-
-```
-Head (60%): Initial output with context
-... [47 lines / 3.2KB truncated — showing first 12 + last 8 lines] ...
-Tail (40%): Final output with errors/results
-```
-
-Line-boundary snapping — never cuts mid-line. Error messages at the bottom are always preserved.
-
 ### Intent-Driven Search (v0.5.0)
 
-When `execute` or `execute_file` is called with an `intent` parameter and output exceeds 5KB, Context Mode replaces blind truncation with intelligent BM25 search:
+When `execute` or `execute_file` is called with an `intent` parameter and output exceeds 5KB, Context Mode chunks the output and uses BM25 search to return only the relevant sections:
 
 ```
-Traditional truncation:
-  stdout (100KB) → head(60%) + tail(40%) → ~100KB in context
-  Problem: relevant info in the middle is lost
+Without intent:
+  stdout (100KB) → full output enters context
 
-Intent-driven search:
+With intent:
   stdout (100KB) → chunk by lines → in-memory FTS5 → search(intent) → 2-5KB relevant sections
   Result: only what you need enters context
 ```
 
 Tested across 4 real-world scenarios:
 
-| Scenario | Smart Truncation | Intent Search | Intent Size | Truncation Size |
-|---|---|---|---|---|
-| Server log error (line 347/500) | **missed** | **found** | 1.5 KB | 5.0 KB |
-| 3 test failures among 200 tests | found 2/3 | **found 3/3** | 2.4 KB | 5.0 KB |
-| 2 build warnings among 300 lines | **missed both** | **found both** | 2.1 KB | 5.0 KB |
-| API auth error (line 743/1000) | **missed** | **found** | 1.2 KB | 4.9 KB |
+| Scenario | Without Intent | With Intent | Size Reduction |
+|---|---|---|---|
+| Server log error (line 347/500) | error lost in output | **found** | 1.5 KB vs 5.0 KB |
+| 3 test failures among 200 tests | only 2/3 visible | **all 3 found** | 2.4 KB vs 5.0 KB |
+| 2 build warnings among 300 lines | both lost in output | **both found** | 2.1 KB vs 5.0 KB |
+| API auth error (line 743/1000) | error lost in output | **found** | 1.2 KB vs 4.9 KB |
 
-Smart truncation fails on 3 of 4 scenarios because relevant content is in the dropped middle section. Intent search finds the target every time while using 50-75% fewer bytes.
+Intent search finds the target every time while using 50-75% fewer bytes.
 
 ### HTML to Markdown Conversion
 
@@ -392,9 +379,9 @@ Just ask naturally — Claude automatically routes through Context Mode when it
 
 | Suite | Tests | Coverage |
 |---|---|---|
-| Executor | 55 | 10 languages, sandbox, truncation, concurrency, timeouts |
+| Executor | 55 | 10 languages, sandbox, output handling, concurrency, timeouts |
 | ContentStore | 40 | FTS5 schema, BM25 ranking, chunking, stemming, plain text indexing |
-| Intent Search | 4 | Smart truncation vs intent-driven search across 4 real-world scenarios |
+| Intent Search | 4 | Intent-driven search across 4 real-world scenarios |
 | MCP Integration | 24 | JSON-RPC protocol, all 5 tools, fetch_and_index, errors |
 
 ## Development

package/build/executor.js

@@ -232,23 +232,23 @@ export class PolyglotExecutor {
         switch (language) {
             case "javascript":
             case "typescript":
-                return `const FILE_CONTENT = require("fs").readFileSync(${escaped}, "utf-8");\n${code}`;
+                return `const FILE_CONTENT_PATH = ${escaped};\nconst FILE_CONTENT = require("fs").readFileSync(FILE_CONTENT_PATH, "utf-8");\n${code}`;
             case "python":
-                return `with open(${escaped}, "r") as _f:\n    FILE_CONTENT = _f.read()\n${code}`;
+                return `FILE_CONTENT_PATH = ${escaped}\nwith open(FILE_CONTENT_PATH, "r") as _f:\n    FILE_CONTENT = _f.read()\n${code}`;
             case "shell":
-                return `FILE_CONTENT=$(cat ${escaped})\n${code}`;
+                return `FILE_CONTENT_PATH=${escaped}\nFILE_CONTENT=$(cat ${escaped})\n${code}`;
             case "ruby":
-                return `FILE_CONTENT = File.read(${escaped})\n${code}`;
+                return `FILE_CONTENT_PATH = ${escaped}\nFILE_CONTENT = File.read(FILE_CONTENT_PATH)\n${code}`;
             case "go":
-                return `package main\n\nimport (\n\t"fmt"\n\t"os"\n)\n\nfunc main() {\n\tb, _ := os.ReadFile(${escaped})\n\tFILE_CONTENT := string(b)\n\t_ = FILE_CONTENT\n${code}\n}\n`;
+                return `package main\n\nimport (\n\t"fmt"\n\t"os"\n)\n\nvar FILE_CONTENT_PATH = ${escaped}\n\nfunc main() {\n\tb, _ := os.ReadFile(FILE_CONTENT_PATH)\n\tFILE_CONTENT := string(b)\n\t_ = FILE_CONTENT\n\t_ = fmt.Sprint()\n${code}\n}\n`;
             case "rust":
-                return `use std::fs;\n\nfn main() {\n    let file_content = fs::read_to_string(${escaped}).unwrap();\n${code}\n}\n`;
+                return `use std::fs;\n\nfn main() {\n    let file_content_path = ${escaped};\n    let file_content = fs::read_to_string(file_content_path).unwrap();\n${code}\n}\n`;
             case "php":
-                return `<?php\n$FILE_CONTENT = file_get_contents(${escaped});\n${code}`;
+                return `<?php\n$FILE_CONTENT_PATH = ${escaped};\n$FILE_CONTENT = file_get_contents($FILE_CONTENT_PATH);\n${code}`;
             case "perl":
-                return `open(my $fh, '<', ${escaped}) or die "Cannot open: $!";\nmy $FILE_CONTENT = do { local $/; <$fh> };\nclose($fh);\n${code}`;
+                return `my $FILE_CONTENT_PATH = ${escaped};\nopen(my $fh, '<', $FILE_CONTENT_PATH) or die "Cannot open: $!";\nmy $FILE_CONTENT = do { local $/; <$fh> };\nclose($fh);\n${code}`;
             case "r":
-                return `FILE_CONTENT <- readLines(${escaped}, warn=FALSE)\nFILE_CONTENT <- paste(FILE_CONTENT, collapse="\\n")\n${code}`;
+                return `FILE_CONTENT_PATH <- ${escaped}\nFILE_CONTENT <- readLines(FILE_CONTENT_PATH, warn=FALSE)\nFILE_CONTENT <- paste(FILE_CONTENT, collapse="\\n")\n${code}`;
         }
     }
 }

package/build/server.js

@@ -5,11 +5,12 @@ import { z } from "zod";
 import { PolyglotExecutor } from "./executor.js";
 import { ContentStore } from "./store.js";
 import { detectRuntimes, getRuntimeSummary, getAvailableLanguages, hasBunRuntime, } from "./runtime.js";
+const VERSION = "0.5.2";
 const runtimes = detectRuntimes();
 const available = getAvailableLanguages(runtimes);
 const server = new McpServer({
     name: "context-mode",
-    version: "0.5.0",
+    version: VERSION,
 });
 const executor = new PolyglotExecutor({ runtimes });
 // Lazy singleton — no DB overhead unless index/search is used
@@ -57,8 +58,9 @@ server.registerTool("execute", {
             .string()
             .optional()
             .describe("What you're looking for in the output. When provided and output is large (>5KB), " +
-            "returns only matching sections via BM25 search instead of truncated output. " +
-            "Example: 'find failing tests', 'HTTP 500 errors', 'memory usage statistics'."),
+            "indexes output into knowledge base and returns section titles + previews — not full content. " +
+            "Use search() to retrieve specific sections. Example: 'failing tests', 'HTTP 500 errors'." +
+            "\n\nTIP: Use specific technical terms, not just concepts. Check 'Searchable terms' in the response for available vocabulary."),
     }),
 }, async ({ language, code, timeout, intent }) => {
     try {
@@ -79,7 +81,7 @@ server.registerTool("execute", {
             if (intent && intent.trim().length > 0 && Buffer.byteLength(output) > INTENT_SEARCH_THRESHOLD) {
                 return {
                     content: [
-                        { type: "text", text: intentSearch(output, intent) },
+                        { type: "text", text: intentSearch(output, intent, `execute:${language}:error`) },
                     ],
                     isError: true,
                 };
@@ -96,7 +98,7 @@ server.registerTool("execute", {
         if (intent && intent.trim().length > 0 && Buffer.byteLength(stdout) > INTENT_SEARCH_THRESHOLD) {
             return {
                 content: [
-                    { type: "text", text: intentSearch(stdout, intent) },
+                    { type: "text", text: intentSearch(stdout, intent, `execute:${language}`) },
                 ],
             };
         }
@@ -135,30 +137,79 @@ function indexStdout(stdout, source) {
 // Helper: intent-driven search on execution output
 // ─────────────────────────────────────────────────────────
 const INTENT_SEARCH_THRESHOLD = 5_000; // bytes — ~80-100 lines
-function intentSearch(stdout, intent, maxResults = 5) {
-    const store = new ContentStore(":memory:");
+function intentSearch(stdout, intent, source, maxResults = 5) {
+    const totalLines = stdout.split("\n").length;
+    const totalBytes = Buffer.byteLength(stdout);
+    // Index into the PERSISTENT store so user can search() later
+    const persistent = getStore();
+    const indexed = persistent.indexPlainText(stdout, source);
+    // Search with an ephemeral store to find matching section titles
+    const ephemeral = new ContentStore(":memory:");
     try {
-        const totalLines = stdout.split("\n").length;
-        const totalBytes = Buffer.byteLength(stdout);
-        store.indexPlainText(stdout, "exec-output");
-        const results = store.search(intent, maxResults);
+        ephemeral.indexPlainText(stdout, source);
+        let results = ephemeral.search(intent, maxResults);
+        // Score-based relaxed search: search ALL words, rank by match count
         if (results.length === 0) {
-            return (`[Intent search: no matches for "${intent}" in ${totalLines}-line output. Returning full output.]\n\n` +
-                stdout);
+            const words = intent.trim().split(/\s+/).filter(w => w.length > 2).slice(0, 20);
+            if (words.length > 0) {
+                const sectionScores = new Map();
+                for (const word of words) {
+                    const wordResults = ephemeral.search(word, 10);
+                    for (const r of wordResults) {
+                        const existing = sectionScores.get(r.title);
+                        if (existing) {
+                            existing.score += 1;
+                            if (r.rank < existing.bestRank) {
+                                existing.bestRank = r.rank;
+                                existing.result = r;
+                            }
+                        }
+                        else {
+                            sectionScores.set(r.title, { result: r, score: 1, bestRank: r.rank });
+                        }
+                    }
+                }
+                results = Array.from(sectionScores.values())
+                    .sort((a, b) => b.score - a.score || a.bestRank - b.bestRank)
+                    .slice(0, maxResults)
+                    .map(s => s.result);
+            }
         }
-        const totalChunks = store.getStats().chunks;
-        const header = `[Intent search: ${results.length} of ${totalChunks} sections matched "${intent}" from ${totalLines}-line output (${(totalBytes / 1024).toFixed(1)}KB)]`;
-        const formatted = results
-            .map((r, i) => {
-            const matchLabel = i === 0 ? " (best match)" : "";
-            return `--- ${r.title}${matchLabel} ---\n${r.content}`;
-        })
-            .join("\n\n");
-        const footer = `[Full output: ${totalLines} lines / ${(totalBytes / 1024).toFixed(1)}KB. Re-run without intent to see raw output.]`;
-        return `${header}\n\n${formatted}\n\n${footer}`;
+        // Extract distinctive terms as vocabulary hints for the LLM
+        const distinctiveTerms = persistent.getDistinctiveTerms(indexed.sourceId);
+        if (results.length === 0) {
+            const lines = [
+                `Indexed ${indexed.totalChunks} sections from "${source}" into knowledge base.`,
+                `No sections matched intent "${intent}" in ${totalLines}-line output (${(totalBytes / 1024).toFixed(1)}KB).`,
+            ];
+            if (distinctiveTerms.length > 0) {
+                lines.push("");
+                lines.push(`Searchable terms: ${distinctiveTerms.join(", ")}`);
+            }
+            lines.push("");
+            lines.push("Use search() to explore the indexed content.");
+            return lines.join("\n");
+        }
+        // Return ONLY titles + first-line previews — not full content
+        const lines = [
+            `Indexed ${indexed.totalChunks} sections from "${source}" into knowledge base.`,
+            `${results.length} sections matched "${intent}" (${totalLines} lines, ${(totalBytes / 1024).toFixed(1)}KB):`,
+            "",
+        ];
+        for (const r of results) {
+            const preview = r.content.split("\n")[0].slice(0, 120);
+            lines.push(`  - ${r.title}: ${preview}`);
+        }
+        if (distinctiveTerms.length > 0) {
+            lines.push("");
+            lines.push(`Searchable terms: ${distinctiveTerms.join(", ")}`);
+        }
+        lines.push("");
+        lines.push("Use search() to retrieve full content of any section.");
+        return lines.join("\n");
     }
     finally {
-        store.close();
+        ephemeral.close();
     }
 }
 // ─────────────────────────────────────────────────────────
@@ -223,7 +274,7 @@ server.registerTool("execute_file", {
             if (intent && intent.trim().length > 0 && Buffer.byteLength(output) > INTENT_SEARCH_THRESHOLD) {
                 return {
                     content: [
-                        { type: "text", text: intentSearch(output, intent) },
+                        { type: "text", text: intentSearch(output, intent, `file:${path}:error`) },
                     ],
                     isError: true,
                 };
@@ -239,7 +290,7 @@ server.registerTool("execute_file", {
         if (intent && intent.trim().length > 0 && Buffer.byteLength(stdout) > INTENT_SEARCH_THRESHOLD) {
             return {
                 content: [
-                    { type: "text", text: intentSearch(stdout, intent) },
+                    { type: "text", text: intentSearch(stdout, intent, `file:${path}`) },
                 ],
             };
         }
@@ -337,6 +388,10 @@ server.registerTool("search", {
         "- Look up API signatures ('Supabase RLS policy syntax')\n" +
         "- Get configuration details ('Tailwind responsive breakpoints')\n" +
         "- Find migration steps ('App Router data fetching')\n\n" +
+        "SEARCH TIPS:\n" +
+        "- Use specific technical terms, not concepts ('__proto__' not 'security')\n" +
+        "- Check 'Searchable terms' from execute/execute_file results for available vocabulary\n" +
+        "- Combine multiple specific terms for better results\n\n" +
         "Returns exact content — not summaries. Each result includes heading hierarchy and full section text.",
     inputSchema: z.object({
         query: z.string().describe("Natural language search query"),
@@ -514,7 +569,7 @@ server.registerTool("fetch_and_index", {
 async function main() {
     const transport = new StdioServerTransport();
     await server.connect(transport);
-    console.error("Context Mode MCP server v0.4.0 running on stdio");
+    console.error(`Context Mode MCP server v${VERSION} running on stdio`);
     console.error(`Detected runtimes:\n${getRuntimeSummary(runtimes)}`);
     if (!hasBunRuntime()) {
         console.error("\nPerformance tip: Install Bun for 3-5x faster JS/TS execution");

package/build/store.d.ts

@@ -40,6 +40,7 @@ export declare class ContentStore {
      */
     indexPlainText(content: string, source: string, linesPerChunk?: number): IndexResult;
     search(query: string, limit?: number): SearchResult[];
+    getDistinctiveTerms(sourceId: number, maxTerms?: number): string[];
     getStats(): StoreStats;
     close(): void;
 }

package/build/store.js

@@ -12,6 +12,24 @@ import { readFileSync } from "node:fs";
 import { tmpdir } from "node:os";
 import { join } from "node:path";
 // ─────────────────────────────────────────────────────────
+// Constants
+// ─────────────────────────────────────────────────────────
+const STOPWORDS = new Set([
+    "the", "and", "for", "are", "but", "not", "you", "all", "can", "had",
+    "her", "was", "one", "our", "out", "has", "his", "how", "its", "may",
+    "new", "now", "old", "see", "way", "who", "did", "get", "got", "let",
+    "say", "she", "too", "use", "will", "with", "this", "that", "from",
+    "they", "been", "have", "many", "some", "them", "than", "each", "make",
+    "like", "just", "over", "such", "take", "into", "year", "your", "good",
+    "could", "would", "about", "which", "their", "there", "other", "after",
+    "should", "through", "also", "more", "most", "only", "very", "when",
+    "what", "then", "these", "those", "being", "does", "done", "both",
+    "same", "still", "while", "where", "here", "were", "much",
+    // Common in code/changelogs
+    "update", "updates", "updated", "deps", "dev", "tests", "test",
+    "add", "added", "fix", "fixed", "run", "running", "using",
+]);
+// ─────────────────────────────────────────────────────────
 // Helpers
 // ─────────────────────────────────────────────────────────
 function sanitizeQuery(query) {
@@ -155,6 +173,46 @@ export class ContentStore {
             contentType: r.content_type,
         }));
     }
+    // ── Vocabulary ──
+    getDistinctiveTerms(sourceId, maxTerms = 40) {
+        const stats = this.#db
+            .prepare("SELECT chunk_count FROM sources WHERE id = ?")
+            .get(sourceId);
+        if (!stats || stats.chunk_count < 3)
+            return [];
+        const totalChunks = stats.chunk_count;
+        const minAppearances = 2;
+        const maxAppearances = Math.max(3, Math.ceil(totalChunks * 0.4));
+        const rows = this.#db
+            .prepare("SELECT content FROM chunks WHERE source_id = ?")
+            .all(sourceId);
+        // Count document frequency (how many sections contain each word)
+        const docFreq = new Map();
+        for (const row of rows) {
+            const words = new Set(row.content
+                .toLowerCase()
+                .split(/[^\p{L}\p{N}_-]+/u)
+                .filter((w) => w.length >= 3 && !STOPWORDS.has(w)));
+            for (const word of words) {
+                docFreq.set(word, (docFreq.get(word) ?? 0) + 1);
+            }
+        }
+        const filtered = Array.from(docFreq.entries())
+            .filter(([, count]) => count >= minAppearances && count <= maxAppearances);
+        // Score: IDF (rarity) + length bonus + identifier bonus (underscore/camelCase)
+        const scored = filtered.map(([word, count]) => {
+            const idf = Math.log(totalChunks / count);
+            const lenBonus = Math.min(word.length / 20, 0.5);
+            const hasSpecialChars = /[_]/.test(word);
+            const isCamelOrLong = word.length >= 12;
+            const identifierBonus = hasSpecialChars ? 1.5 : isCamelOrLong ? 0.8 : 0;
+            return { word, score: idf + lenBonus + identifierBonus };
+        });
+        return scored
+            .sort((a, b) => b.score - a.score)
+            .slice(0, maxTerms)
+            .map((s) => s.word);
+    }
     // ── Stats ──
     getStats() {
         const sources = this.#db.prepare("SELECT COUNT(*) as c FROM sources").get()?.c ?? 0;
@@ -246,10 +304,14 @@ export class ContentStore {
             sections.length <= 200 &&
             sections.every((s) => Buffer.byteLength(s) < 5000)) {
             return sections
-                .map((section, i) => ({
-                title: `Section ${i + 1}`,
-                content: section.trim(),
-            }))
+                .map((section, i) => {
+                const trimmed = section.trim();
+                const firstLine = trimmed.split("\n")[0].slice(0, 80);
+                return {
+                    title: firstLine || `Section ${i + 1}`,
+                    content: trimmed,
+                };
+            })
                 .filter((s) => s.content.length > 0);
         }
         const lines = text.split("\n");
@@ -267,8 +329,9 @@ export class ContentStore {
                 break;
             const startLine = i + 1;
             const endLine = Math.min(i + slice.length, lines.length);
+            const firstLine = slice[0]?.trim().slice(0, 80);
             chunks.push({
-                title: `Lines ${startLine}-${endLine}`,
+                title: firstLine || `Lines ${startLine}-${endLine}`,
                 content: slice.join("\n"),
             });
         }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "context-mode",
-  "version": "0.5.0",
+  "version": "0.5.2",
   "type": "module",
   "description": "Claude Code MCP plugin that saves 94% of your context window. Sandboxed code execution, FTS5 knowledge base, and smart truncation.",
   "author": "Mert Koseoğlu",