@nxuss/lemma 1.3.3 → 1.4.0

package/README.md

@@ -1,4 +1,4 @@
-# 🧠 Lemma v1.3.3
+# 🧠 Lemma v1.4.0
 
 [![npm version](https://img.shields.io/npm/v/@nxuss/lemma.svg?color=orange&style=flat-square)](https://www.npmjs.com/package/@nxuss/lemma)
 [![License](https://img.shields.io/npm/l/@nxuss/lemma.svg?color=blue&style=flat-square)](https://github.com/Nxusbets/lemma/blob/main/LICENSE)
@@ -23,7 +23,36 @@ Lemma is a high-performance orchestration layer that sits between your developme
 
 ## 📋 Changelog
 
-### v1.3.0 — Verifiable Caching *(NEW)*
+### v1.4.0 — Correctness-first patching, search, and memory freshness *(NEW)*
+
+An audit of the everyday MCP path — read a file, search the workspace, patch it, remember the
+answer — found the same failure mode repeating: a tool that looked done but silently degraded
+correctness or wasted tokens the moment reality diverged from the happy path. This release fixes
+that path end to end.
+
+| Tool | What changed | Why |
+| :--- | :--- | :--- |
+| `apply_workspace_patch` | Cascading match: exact → whitespace-normalized → indent-anchored → fuzzy (similarity-scored, with an anti-tie margin). On total failure, the error now includes a line-diff against the closest candidate block instead of a generic "not found". | A patch built from a `read_workspace_file(compact:true)` view (comments/imports stripped, whitespace collapsed) could never byte-match the file on disk — a guaranteed failure on the most common read→edit flow, forcing an expensive full re-read every time. |
+| `search_workspace` | Respects `.gitignore`, skips binaries and files over 500KB, supports comma-separated multi-extension filters, caps results with `maxResults`, and groups matches by file (capped per file) instead of one repeated-path line per hit. Code files (`.ts/.tsx/.js/.jsx`) get symbol-level grouping — matches inside the same function/class collapse into one entry with its signature and full line list instead of raw duplicate lines. Repeating an identical query while the workspace is unchanged (tracked via a cheap mtime/size fingerprint, no content re-reads) returns the cached result instantly. | A single broad query could read 300+ files (2.6MB+) unfiltered and return 15,000+ tokens of flat, path-repeated text — often more expensive than just reading the files directly. |
+| `read_workspace_file` | Tracks the content hash last sent for each `(file, compact)` pair for the life of the MCP session. Re-reading an unchanged file returns a short `UNCHANGED` marker instead of the full content again; `force:true` bypasses it. | Re-reading the same file to verify an edit — one of the most common steps in a multi-turn coding session — was resending the full file every time, even when nothing had changed since the last read. |
+| `store_memory` / `search_memory` | `store_memory` accepts an optional `filePaths` array and records a sha256 per file. `search_memory` now splits results into **fresh** (safe to reuse) and **stale** (a tracked file changed since it was stored) — a stale hit is never returned as if it were current. | The Brain's semantic match (BM25 + Jaccard) is intentionally fuzzy so differently-phrased questions still hit — but with no invalidation, a fuzzy hit could resurface a stale answer about code that had since changed, with no signal that it was stale. |
+
+**Net effect**: the four tools an agent reaches for by default — read, search, patch, remember —
+now fail closed (a diff instead of a dead-end error) and skip redundant work automatically instead
+of requiring the calling model to remember to check first.
+
+### Files changed
+- `src/utils/PatchMatcher.ts` *(new)* — cascading match strategies + line-diff generator
+- `src/utils/WorkspaceSearch.ts` *(new)* — gitignore-aware, capped, grouped search
+- `src/utils/StructuralSearch.ts` *(new)* — AST-based symbol grouping for code file matches
+- `src/utils/SearchCache.ts` *(new)* — session-scoped, fingerprint-invalidated search cache
+- `src/utils/ReadWorkspaceCache.ts` *(new)* — session-scoped read-dedup
+- `src/subconscious/TheBrainV2.ts` — `fileHashes` on entries, `checkEntryFreshness`, `fresh`/`staleFiles` on search results
+- `src/mcp/tools.ts` — wired all of the above into `apply_workspace_patch`, `search_workspace`, `read_workspace_file`, `store_memory`, `search_memory`
+
+---
+
+### v1.3.0 — Verifiable Caching
 
 Two new tools built around a single rule: don't ask the model to trust a similarity score or a
 savings estimate — give it something it can actually verify.

package/dist/cjs/mcp/tools.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"tools.d.ts","sourceRoot":"","sources":["../../../src/mcp/tools.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,MAAM,EAAE,MAAM,2CAA2C,CAAC;AAwiCnE,MAAM,WAAW,aAAa;IAC5B,IAAI,EAAE,MAAM,CAAC;IACb,IAAI,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;IAC1B,MAAM,EAAE,IAAI,GAAG,OAAO,GAAG,KAAK,GAAG,MAAM,CAAC;IACxC,OAAO,EAAE,MAAM,CAAC;IAChB,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,KAAK,CAAC,EAAE,MAAM,CAAC;CAChB;AAED,wBAAgB,kBAAkB,CAChC,MAAM,EAAE,MAAM,EACd,UAAU,CAAC,EAAE,CAAC,KAAK,EAAE,aAAa,KAAK,IAAI,GAC1C,IAAI,CA2DN"}
+{"version":3,"file":"tools.d.ts","sourceRoot":"","sources":["../../../src/mcp/tools.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,MAAM,EAAE,MAAM,2CAA2C,CAAC;AA+iCnE,MAAM,WAAW,aAAa;IAC5B,IAAI,EAAE,MAAM,CAAC;IACb,IAAI,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;IAC1B,MAAM,EAAE,IAAI,GAAG,OAAO,GAAG,KAAK,GAAG,MAAM,CAAC;IACxC,OAAO,EAAE,MAAM,CAAC;IAChB,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,KAAK,CAAC,EAAE,MAAM,CAAC;CAChB;AAED,wBAAgB,kBAAkB,CAChC,MAAM,EAAE,MAAM,EACd,UAAU,CAAC,EAAE,CAAC,KAAK,EAAE,aAAa,KAAK,IAAI,GAC1C,IAAI,CA2DN"}

package/dist/cjs/mcp/tools.js

@@ -72,6 +72,10 @@ const SurgicalASTInsert_1 = require("../utils/SurgicalASTInsert");
 const mcp_tools_1 = require("../infra/mcp-tools");
 const StateHashCache_1 = require("../utils/StateHashCache");
 const TokenReceipt_1 = require("../utils/TokenReceipt");
+const PatchMatcher_1 = require("../utils/PatchMatcher");
+const WorkspaceSearch_1 = require("../utils/WorkspaceSearch");
+const ReadWorkspaceCache_1 = require("../utils/ReadWorkspaceCache");
+const SearchCache_1 = require("../utils/SearchCache");
 // ── Pro License Gate ──────────────────────────────────────────────────────────
 const LICENSE_FILE = path_1.default.join(os_1.default.homedir(), '.lemma-cache', 'license.json');
 let _proStatusCache = null;
@@ -128,7 +132,7 @@ const toolDefinitions = [
     },
     {
         name: "search_memory",
-        description: "Search Lemma's semantic memory (The Brain) before answering technical questions. Retrieves past solutions, fixes, and context from ALL your projects globally. If similarity > 50%, use this answer directly.",
+        description: "Search Lemma's semantic memory (The Brain) before investigating something from scratch — retrieves past solutions, fixes, and context from ALL your projects globally. Results are split into 'fresh' (safe to reuse — either untracked general knowledge, or every file it depended on still hashes the same) and 'stale' (a similar question was answered before, but a tracked file changed since — re-verify against current state before reusing). Never treat a stale result as current.",
         inputSchema: {
             type: "object",
             properties: {
@@ -140,13 +144,14 @@ const toolDefinitions = [
     },
     {
         name: "store_memory",
-        description: "Persist a technical solution, bug fix, architecture decision, or key fact into Lemma's Brain.",
+        description: "Persist a technical solution, bug fix, architecture decision, or key fact into Lemma's Brain — so future questions on the same topic (even phrased differently) don't require re-investigating the repo. Pass filePaths for anything derived from specific files (e.g. 'where is X implemented', 'how does Y work') so the memory auto-invalidates the moment those files change, instead of risking a stale answer being reused.",
         inputSchema: {
             type: "object",
             properties: {
                 query: { type: "string", description: "The natural language query key" },
                 response: { type: "string", description: "The complete technical content to memorize" },
                 provider: { type: "string", description: "Optional model provider name", default: "generic" },
+                filePaths: { type: "array", items: { type: "string" }, description: "Paths (relative to project root) this answer depends on. If any changes later, this memory is marked stale instead of being silently reused." },
             },
             required: ["query", "response"],
         },
@@ -175,12 +180,13 @@ const toolDefinitions = [
     },
     {
         name: "read_workspace_file",
-        description: "Read a file inside the workspace. Automatically compresses comments/whitespace and scrubs API keys.",
+        description: "Read a file inside the workspace. Automatically compresses comments/whitespace and scrubs API keys. If this exact file (same compact mode) was already read this session and hasn't changed on disk, returns a short UNCHANGED marker instead of resending the full content — reuse your existing copy. Pass force:true to always get the full content.",
         inputSchema: {
             type: "object",
             properties: {
                 filePath: { type: "string", description: "Path relative to the project root" },
                 compact: { type: "boolean", description: "Whether to compress comments and whitespace", default: true },
+                force: { type: "boolean", description: "Bypass the unchanged-file short-circuit and always return full content", default: false },
             },
             required: ["filePath"],
         },
@@ -211,7 +217,7 @@ const toolDefinitions = [
     },
     {
         name: "apply_workspace_patch",
-        description: "Apply a smart search-and-replace patch to an existing file. Safe against duplicate matches.",
+        description: "Apply a smart search-and-replace patch to an existing file. Falls back through whitespace-normalized, indent-anchored, and fuzzy matching if the exact block isn't found — so a searchContent built from a compacted read still applies. Safe against duplicate matches. On total failure, the error includes a diff against the closest candidate block.",
         inputSchema: {
             type: "object",
             properties: {
@@ -246,12 +252,13 @@ const toolDefinitions = [
     },
     {
         name: "search_workspace",
-        description: "Perform a fast local text search (grep) across all files in the project workspace.",
+        description: "Perform a fast local text search (grep) across all files in the project workspace. Respects .gitignore, skips binaries and files over 500KB, groups results by file (capped per file), and stops at maxResults. For code files (.ts/.tsx/.js/.jsx), matches inside the same function/class/method are collapsed into one entry with its signature and line range instead of raw duplicate lines. Repeating an identical query while the workspace is unchanged returns the cached result instantly instead of re-scanning.",
         inputSchema: {
             type: "object",
             properties: {
                 query: { type: "string", description: "The keyword or text pattern to search for" },
-                extension: { type: "string", description: "Optional file extension filter (e.g., 'ts', 'json')" },
+                extension: { type: "string", description: "Optional file extension filter, comma-separated for multiple (e.g., 'ts,tsx')" },
+                maxResults: { type: "number", description: "Maximum total matches to return before truncating", default: 200 },
             },
             required: ["query"],
         },
@@ -1152,12 +1159,27 @@ async function handleSearchMemory(args) {
         if (results.length === 0) {
             return { content: [{ type: "text", text: "No relevant memories found in Lemma's Brain." }] };
         }
-        const formatted = results
-            .map((r, i) => {
-            return `Result ${i + 1} (Similarity: ${(r.similarity * 100).toFixed(1)}%)\nPrompt: ${r.query.substring(0, 300)}...\nResponse: ${typeof r.response === "string" ? r.response : JSON.stringify(r.response, null, 2)}`;
-        })
-            .join("\n\n---\n\n");
-        return { content: [{ type: "text", text: `Lemma found ${results.length} memories:\n\n${formatted}` }] };
+        const fresh = results.filter((r) => r.fresh);
+        const stale = results.filter((r) => !r.fresh);
+        const formatResult = (r, i) => `Result ${i + 1} (Similarity: ${(r.similarity * 100).toFixed(1)}%)\nPrompt: ${r.query.substring(0, 300)}...\nResponse: ${typeof r.response === "string" ? r.response : JSON.stringify(r.response, null, 2)}`;
+        if (fresh.length > 0) {
+            // Only a fresh hit (or an untracked, purely conceptual entry) counts as a real
+            // avoided re-investigation — this is the only case worth crediting as savings.
+            const bestFresh = fresh[0];
+            const tokensSaved = Math.max(100, Math.floor(String(bestFresh.response).length / 4));
+            (0, reportSavings_1.reportSavings)({ source: "cache", tokens: tokensSaved, toolName: "search_memory", query: query.substring(0, 100) });
+        }
+        const parts = [];
+        if (fresh.length > 0) {
+            parts.push(`Lemma found ${fresh.length} fresh ${fresh.length === 1 ? "memory" : "memories"} (from cache — say so if you use this):\n\n${fresh.map(formatResult).join("\n\n---\n\n")}`);
+        }
+        if (stale.length > 0) {
+            const staleText = stale
+                .map((r, i) => `${formatResult(r, i)}\nSTALE — changed since stored: ${r.staleFiles.join(", ")}`)
+                .join("\n\n---\n\n");
+            parts.push(`${stale.length} more ${stale.length === 1 ? "memory is" : "memories are"} similar but STALE (underlying file(s) changed) — do not reuse verbatim, re-verify against current file state:\n\n${staleText}`);
+        }
+        return { content: [{ type: "text", text: parts.join("\n\n===\n\n") }] };
     }
     catch (e) {
         (0, utils_1.logError)("search_memory", e);
@@ -1168,11 +1190,12 @@ async function handleStoreMemory(args) {
     const query = args?.query;
     const responseText = args?.response;
     const provider = args?.provider || "generic";
+    const filePaths = Array.isArray(args?.filePaths) ? args.filePaths : undefined;
     if (!query || !responseText)
         throw new Error("Query and response are required");
     try {
         const brain = (0, TheBrainV2_1.getBrain)();
-        const storeRes = brain.store(query, responseText, provider);
+        const storeRes = brain.store(query, responseText, provider, 0.92, filePaths);
         const tokensSaved = Math.max(100, Math.floor(responseText.length / 4));
         (0, reportSavings_1.reportSavings)({
             source: "cache",
@@ -1180,7 +1203,8 @@ async function handleStoreMemory(args) {
             toolName: "store_memory",
             query: query.substring(0, 100),
         });
-        return { content: [{ type: "text", text: `Success: Memory stored. ${storeRes.reason}` }] };
+        const trackingNote = filePaths && filePaths.length > 0 ? ` Tracking freshness against ${filePaths.length} file(s) — this memory auto-invalidates if they change.` : "";
+        return { content: [{ type: "text", text: `Success: Memory stored. ${storeRes.reason}${trackingNote}` }] };
     }
     catch (e) {
         (0, utils_1.logError)("store_memory", e);
@@ -1225,6 +1249,7 @@ async function handleAutoHeal(args) {
 async function handleReadWorkspaceFile(args) {
     const filePath = args?.filePath;
     const compact = args?.compact !== false;
+    const force = !!args?.force;
     if (!filePath)
         throw new Error("filePath is required");
     const workspaceRoot = process.cwd();
@@ -1232,6 +1257,24 @@ async function handleReadWorkspaceFile(args) {
         const { resolved } = (0, utils_1.safeResolvePath)(workspaceRoot, filePath);
         let content = fs_1.default.readFileSync(resolved, "utf8");
         const originalSize = content.length;
+        const dedup = (0, ReadWorkspaceCache_1.checkAlreadySent)(resolved, compact, content);
+        if (!force && dedup.unchanged) {
+            (0, reportSavings_1.reportSavings)({
+                source: "contextSqueeze",
+                charsBefore: originalSize,
+                charsAfter: 0,
+                toolName: "read_workspace_file",
+                filePath,
+            });
+            return {
+                content: [
+                    {
+                        type: "text",
+                        text: `UNCHANGED: ${filePath} is identical to what you already read this session (hash ${dedup.hash.slice(0, 12)}) — reuse your existing copy. Pass force:true to re-fetch anyway.`,
+                    },
+                ],
+            };
+        }
         if (compact) {
             content = (0, ContextSqueezer_1.squeezeCode)(content);
             if (content.length < originalSize) {
@@ -1252,6 +1295,7 @@ async function handleReadWorkspaceFile(args) {
         catch (err) {
             (0, utils_1.logWarn)("read_workspace_file", "SemanticScrubber failed, returning raw content");
         }
+        (0, ReadWorkspaceCache_1.recordSent)(resolved, compact, dedup.hash);
         return { content: [{ type: "text", text: content }] };
     }
     catch (err) {
@@ -1318,30 +1362,44 @@ async function handleApplyWorkspacePatch(args) {
             return { content: [{ type: "text", text: `Error: File ${filePath} does not exist.` }] };
         }
         const originalContent = fs_1.default.readFileSync(resolved, "utf8");
-        const index = originalContent.indexOf(searchContent);
-        if (index === -1) {
-            return {
-                content: [
-                    {
-                        type: "text",
-                        text: `Error: Target search content not found in file. Make sure the search content matches exactly, including indentation, tabs, spaces, and newlines.`,
-                    },
-                ],
-            };
-        }
-        if (originalContent.indexOf(searchContent, index + 1) !== -1) {
+        const matchResult = (0, PatchMatcher_1.findMatch)(originalContent, searchContent);
+        if (!Array.isArray(matchResult)) {
+            if (matchResult.reason === "ambiguous") {
+                return {
+                    content: [
+                        {
+                            type: "text",
+                            text: `Error: Multiple matches found for the search content. Please provide a larger search block with unique surrounding context lines to make it unambiguous.`,
+                        },
+                    ],
+                };
+            }
+            if (matchResult.candidateDiff) {
+                return {
+                    content: [
+                        {
+                            type: "text",
+                            text: `Error: No match found for the search content. Closest candidate starts at line ${matchResult.candidateStartLine} (- = your searchContent, + = actual file content):\n\n${matchResult.candidateDiff}`,
+                        },
+                    ],
+                };
+            }
             return {
                 content: [
                     {
                         type: "text",
-                        text: `Error: Multiple matches found for the search content. Please provide a larger search block with unique surrounding context lines to make it unambiguous.`,
+                        text: `Error: Target search content not found in file, and no similar block was found to diff against.`,
                     },
                 ],
             };
         }
-        const updatedContent = originalContent.replace(searchContent, replaceContent);
-        fs_1.default.writeFileSync(resolved, updatedContent, "utf8");
-        return { content: [{ type: "text", text: `Success: Patch successfully applied to ${filePath}` }] };
+        const match = matchResult[0];
+        const contentLines = originalContent.split("\n");
+        const finalReplace = match.strategy === "exact" ? replaceContent : (0, PatchMatcher_1.reindentReplacement)(replaceContent, searchContent, contentLines[match.startLine]);
+        const updatedLines = [...contentLines.slice(0, match.startLine), ...finalReplace.split("\n"), ...contentLines.slice(match.endLine + 1)];
+        fs_1.default.writeFileSync(resolved, updatedLines.join("\n"), "utf8");
+        const note = match.strategy === "exact" ? "" : ` (matched via ${match.strategy}, score ${match.score.toFixed(2)})`;
+        return { content: [{ type: "text", text: `Success: Patch successfully applied to ${filePath}${note}` }] };
     }
     catch (err) {
         (0, utils_1.logError)("apply_workspace_patch", err);
@@ -1442,53 +1500,28 @@ async function handleListWorkspaceDir(args) {
         return { content: [{ type: "text", text: `Error listing directory: ${err.message}` }] };
     }
 }
-function searchDirRecursive(currentDir, relativePath, query, extFilter) {
-    try {
-        const entries = fs_1.default.readdirSync(currentDir, { withFileTypes: true });
-        let result = [];
-        for (const entry of entries) {
-            if (["node_modules", ".git", "dist", "chroma_data", ".lemma"].includes(entry.name))
-                continue;
-            const rel = relativePath ? path_1.default.join(relativePath, entry.name) : entry.name;
-            const full = path_1.default.join(currentDir, entry.name);
-            if (entry.isDirectory()) {
-                result.push(...searchDirRecursive(full, rel, query, extFilter));
-            }
-            else {
-                if (extFilter && !entry.name.endsWith(`.${extFilter}`))
-                    continue;
-                try {
-                    const content = fs_1.default.readFileSync(full, "utf8");
-                    if (content.toLowerCase().includes(query.toLowerCase())) {
-                        const lines = content.split("\n");
-                        lines.forEach((lineText, idx) => {
-                            if (lineText.toLowerCase().includes(query.toLowerCase())) {
-                                result.push({ filePath: rel, line: idx + 1, text: lineText.trim() });
-                            }
-                        });
-                    }
-                }
-                catch { }
-            }
-        }
-        return result;
-    }
-    catch {
-        return [];
-    }
-}
 async function handleSearchWorkspace(args) {
     const query = args?.query;
     const extension = args?.extension;
+    const maxResults = args?.maxResults || 200;
     if (!query)
         throw new Error("query is required");
     const workspaceRoot = process.cwd();
     try {
-        const results = searchDirRecursive(workspaceRoot, "", query, extension);
-        if (results.length === 0) {
-            return { content: [{ type: "text", text: `No matches found in the workspace for "${query}".` }] };
-        }
-        let formatted = results.map((r) => `[${r.filePath}:${r.line}] ${r.text}`).join("\n");
+        const extensions = (0, WorkspaceSearch_1.parseExtensionFilter)(extension);
+        const cached = (0, SearchCache_1.getCachedSearch)(workspaceRoot, query, extensions, maxResults);
+        if (cached.text !== null) {
+            (0, reportSavings_1.reportSavings)({ source: "cache", tokens: Math.floor(cached.text.length / 4), toolName: "search_workspace", query: query.substring(0, 100) });
+            return { content: [{ type: "text", text: cached.text }] };
+        }
+        const { matches, truncated } = (0, WorkspaceSearch_1.searchWorkspace)(workspaceRoot, query, extensions, maxResults);
+        if (matches.length === 0) {
+            const emptyText = `No matches found in the workspace for "${query}".`;
+            (0, SearchCache_1.setCachedSearch)(query, extensions, maxResults, cached.fingerprint, emptyText);
+            return { content: [{ type: "text", text: emptyText }] };
+        }
+        const grouped = (0, WorkspaceSearch_1.groupSearchResults)(matches, truncated, undefined, workspaceRoot);
+        let formatted = grouped.formatted;
         try {
             const scrubber = new SemanticScrubber_1.SemanticScrubber();
             const { maskedPrompt } = scrubber.mask(formatted);
@@ -1497,8 +1530,16 @@ async function handleSearchWorkspace(args) {
         catch (err) {
             (0, utils_1.logWarn)("search_workspace", "SemanticScrubber failed, returning raw results");
         }
+        const truncationNote = grouped.truncated ? ` (truncated at ${maxResults} matches — narrow your query or extension filter for the rest)` : "";
+        const resultText = `Found ${grouped.totalMatches} matches across ${grouped.filesMatched} files${truncationNote}:\n\n${formatted}`;
+        (0, SearchCache_1.setCachedSearch)(query, extensions, maxResults, cached.fingerprint, resultText);
         return {
-            content: [{ type: "text", text: `Found ${results.length} matches:\n\n${formatted}` }],
+            content: [
+                {
+                    type: "text",
+                    text: resultText,
+                },
+            ],
         };
     }
     catch (err) {