@nxuss/lemma 1.8.0 → 1.8.1

package/README.md

@@ -1,11 +1,11 @@
-# 🧠 Lemma v1.8.0
+# 🧠 Lemma v1.8.1
 
 [![npm version](https://img.shields.io/npm/v/@nxuss/lemma.svg?color=orange&style=flat-square)](https://www.npmjs.com/package/@nxuss/lemma)
 [![License](https://img.shields.io/npm/l/@nxuss/lemma.svg?color=blue&style=flat-square)](https://github.com/Nxusbets/lemma/blob/main/LICENSE)
 [![MCP Tools](https://img.shields.io/badge/MCP%20tools-95-ff69b4?style=flat-square)](https://github.com/Nxusbets/lemma)
 [![Security](https://img.shields.io/badge/privacy-firewall%20active-blueviolet?style=flat-square)](https://github.com/Nxusbets/lemma)
 [![Caching](https://img.shields.io/badge/semantic%20memory-enabled-success?style=flat-square)](https://github.com/Nxusbets/lemma)
-[![Tests](https://img.shields.io/badge/tests-293%20passing-00c853?style=flat-square)](https://github.com/Nxusbets/lemma)
+[![Tests](https://img.shields.io/badge/tests-309%20passing-00c853?style=flat-square)](https://github.com/Nxusbets/lemma)
 
 ```text
     __                                
@@ -27,6 +27,26 @@ are advertised to the model each turn. The rest stay one `lemma_toolbox` call aw
 
 ## 📋 Changelog
 
+### v1.8.1 — Two correctness fixes: secret scrubbing and patch data loss *(NEW)*
+
+An audit of the tools with no test coverage turned up two bugs where the tool looked done but
+silently produced wrong output.
+
+- **`apply_workspace_patch` could corrupt a line.** The exact-match strategy computed
+  `startLine`/`endLine` from a raw substring `indexOf`, but the handler always spliced by whole
+  line. A search block that was a *fragment* of a line (not the entire line) collapsed
+  `startLine === endLine`, and the splice replaced the entire line with just the replacement —
+  discarding everything else on it. Exact matches now splice by character offset;
+  whitespace-normalized/indent-anchored/fuzzy matches (already full-line windows) are unaffected.
+- **The privacy scrubber only recognized 3 generic patterns.** Default patterns covered
+  `API_KEY`/`EMAIL`/`Bearer ...` shapes only — vendor-specific secrets (Stripe, AWS, GitHub,
+  Google, Slack, JWTs, PEM private-key blocks) and custom-named secrets (e.g. a project's own
+  `_SALT` or `_WEBHOOK_SECRET` env var) passed through unmasked. Also, a project's custom
+  `lemma.config.json` patterns silently *replaced* the defaults instead of adding to them, so
+  defining one narrow pattern dropped coverage for everything else. Added vendor-specific
+  patterns plus a generic `ENV_SECRET` catch-all keyed on variable-name conventions, and custom
+  patterns now augment the defaults by default (`replaceDefaults: true` opts into full override).
+
 ### v1.8.0 — The Brain learns from project history, not just what it's told *(NEW)*
 
 Every memory in The Brain used to depend on the assistant remembering to call `store_memory`. If
@@ -411,12 +431,29 @@ chmod +x .git/hooks/post-commit
 
 ## 💎 Free vs Pro
 
-### 🆓 Free (34 Tools)
+Of the 95 registered tools, 46 work with no license and 49 require Pro. This split is enforced by
+one allowlist in code (`FREE_TOOLS` in `src/mcp/tools.ts`), not by narrative — if a tool isn't in
+that list, it's Pro, regardless of what a description elsewhere implies.
+
+### 🆓 Free (46 Tools)
 
 These tools work **without a Pro license**. Activate them with just `lemma init`:
 
 | Tool | What It Does |
 | :--- | :--- |
+| `smarter_cache` | Semantic cache lookup before reasoning from scratch |
+| `state_hash_cache` | Reasoning cache keyed to file content hashes — exact invalidation |
+| `token_receipt` | Auditable per-session ledger of how each answer was produced |
+| `search_memory` | Semantic search across The Brain, all projects |
+| `store_memory` | Persist verified solutions/fixes/decisions into The Brain |
+| `get_project_history` | Git log + checkpoints + Brain memories in one call |
+| `token_budget` | Session token/cost dashboard, including Lemma's own overhead |
+| `squeeze_prompt` | Compress code blocks before they enter your own context |
+| `turbosqueeze` | Extreme internal-context compression (never shown to the user) |
+| `read_workspace_file` | Read a file with auto-compression and privacy scrubbing |
+| `scrub_privacy` | Mask PII, API keys, and secrets from any text block |
+| `summarize_long_text` | Summarize via local Ollama, falls back to truncation |
+| `lemma_toolbox` | List, inspect, and call tools whose schemas were withheld |
 | `file_fingerprint` | File hash cache — avoid re-reads |
 | `git_blame_heat` | Git blame + risk scoring |
 | `type_coverage_report` | TypeScript type safety score |
@@ -426,7 +463,6 @@ These tools work **without a Pro license**. Activate them with just `lemma init`
 | `auto_context_bundle` | Full context bundle per file |
 | `repeat_guard` | Duplicate output detector |
 | `conversation_checkpoint` | Session state save/load |
-| `get_project_history` | Git log + checkpoints + Brain memories in one call |
 | `test_autofix_interceptor` | Test failure → Brain auto-fix |
 | `prompt_pattern_cache` | Reusable prompt templates |
 | `infra_discover` | Auto-detect Jenkins, OKD, Terraform |
@@ -452,25 +488,27 @@ These tools work **without a Pro license**. Activate them with just `lemma init`
 | `terraform_state_list` | List Terraform state resources |
 | `terraform_drift` | Detect Terraform drift |
 
-### 💎 Pro (51 Tools)
+### 💎 Pro (49 Tools)
 
 All remaining tools require a Pro license. Activate with `lemma activate <key>` → [Get Pro](https://lemma.nxus.studio/upgrade)
 
 | Category | Tools |
 | :--- | :--- |
-| **AI Proxy** | OpenAI/Anthropic/Gemini proxy, exact-match cache, context squeezer, complexity router |
-| **Privacy** | Privacy firewall (semantic scrubber), PII masking |
-| **The Brain** | Semantic memory, telepathic hints, hybrid consensus, smarter cache, dedup guard |
-| **Zero-Token Static** | entropy_score, coupling_radar, pattern_fossil, git_heatmap_risk, precrime_static, dead_export_necromancer |
-| **Architecture** | depgraph, refactor, ast_flow_visualizer, imports_skeleton_resolver |
-| **Code Intelligence** | get_ast_hologram, validate_patch_sandbox, surgical_ast_insert, get_symbol_surgical_context |
-| **Workspace** | read/write/create files, apply patches, search, list dirs, run commands |
-| **Optimization** | turbosqueeze, squeeze_prompt, wormhole_squeeze, compress_context, token_budget |
-| **PR Review** | review_diff, review_pr, pr_status, generate_pr_workflow |
-| **Session** | test_oracle, schema_extract, changelog_auto, env_snapshot, migration_tracer |
-| **Automation** | AutoPilot, Auto-Heal, Silent Patch Watcher |
-
-> **The MCP server is Pro.** All 51 Pro tools — The Brain, PreCrime, AST Hologram, `depgraph`, `refactor`, and every optimization tool — require a Pro license. The 33 Free tools listed above (including all 22 Infrastructure tools) are accessible without activation.
+| **AI Proxy** | OpenAI/Anthropic/Gemini proxy, exact-match cache, complexity router |
+| **The Brain (advanced)** | `get_telepathic_hints`, `query_hybrid_consensus`, `semantic_dedup_guard`, `local_semantic_autofix` — basic search/store/smarter_cache are Free |
+| **Zero-Token Static** | `entropy_score`, `coupling_radar`, `pattern_fossil`, `git_heatmap_risk`, `precrime_static`, `dead_export_necromancer` |
+| **Architecture** | `depgraph`, `refactor`, `ast_flow_visualizer`, `imports_skeleton_resolver`, `cognitive_map`, `file_intent_index` |
+| **Code Intelligence** | `get_ast_hologram`, `validate_patch_sandbox`, `surgical_ast_insert`, `get_symbol_surgical_context`, `smart_file_slice`, `diff_only` |
+| **Workspace writes** | `write_workspace_file`, `create_workspace_file`, `apply_workspace_patch`, `multi_file_patch`, `search_workspace`, `list_workspace_dir`, `run_workspace_command`, `output_region` |
+| **Optimization (advanced)** | `wormhole_squeeze`, `compress_context`, `prune_conversation_history` — `turbosqueeze`, `squeeze_prompt`, `token_budget` are Free |
+| **PR Review** | `review_diff`, `review_pr`, `pr_status`, `generate_pr_workflow` |
+| **Session & Automation** | `test_oracle`, `schema_extract`, `changelog_auto`, `env_snapshot`, `migration_tracer`, `spec_to_stub`, `batch_tool_calls`, `get_routing_advice`, `get_project_onboarding`, `auto_heal`, `semantic_grep` |
+| **Automation (background)** | AutoPilot, Auto-Heal, Silent Patch Watcher |
+
+> **The MCP server itself is Pro.** All 49 Pro tools — the advanced Brain layer, PreCrime, AST
+> Hologram, `depgraph`, `refactor`, every workspace-write tool, and every PR review tool — require a
+> Pro license. The 46 Free tools above (including all 22 Infrastructure tools and `read_workspace_file`
+> with its automatic privacy scrubbing) are accessible without activation.
 
 ---
 
@@ -687,6 +725,8 @@ lemma start
 | `lemma apply <id>` | Apply a pending fix patch. |
 | `lemma discard <id>` | Discard a pending fix patch. |
 | `lemma activate <key>` | Activate Pro license for full MCP access. |
+| `lemma-checkpoint [projectRoot]` | Standalone bin: derive a session checkpoint from repo state (branch, changed files, recent commits) without an assistant call. See [Session checkpoints](#-session-checkpoints). |
+| `lemma-brain-ingest [projectRoot]` | Standalone bin: feed The Brain from git commits and `CHANGELOG.md`, incrementally. See [Feeding The Brain from project history](#-feeding-the-brain-from-project-history). |
 
 ### Customizing via `lemma.config.json`
 
@@ -726,157 +766,32 @@ Just run `lemma init`. Lemma automatically registers its MCP server in your `~/.
 }
 ```
 
-Once connected, your IDE gains access to **62 MCP tools** — 11 free + 51 Pro. Lemma auto-injects the TurboMode system prompt on handshake.
-
----
-
-## 🛠️ MCP Tools Reference (v1.1.0 — 62 Tools)
-
-### 🆓 Free Tools (11)
-
-> These tools work without a Pro license. All are zero-dependency and run locally.
-
-| Tool | Category | Description |
-| :--- | :--- | :--- |
-| `file_fingerprint` | Smart Read | SHA256 hash cache. Returns 'unchanged' when hash matches, avoiding re-reads. |
-| `git_blame_heat` | Analysis | Git blame + risk scoring per line. Identifies hotspots and high-risk authors. |
-| `type_coverage_report` | Analysis | TypeScript type safety score via AST. Reports typed vs untyped, `any` count. |
-| `import_tree_context` | Architecture | Bidirectional import graph with reverse dependencies. |
-| `read_token_budgeted` | Smart Read | Read file with hard token limit. Returns only relevant sections. |
-| `bulk_file_digest` | Smart Read | Read N files in one call. Returns compressed digest. |
-| `auto_context_bundle` | Smart Read | One-call bundle: content + imports + types + Brain hints. |
-| `repeat_guard` | Optimization | Detects duplicate outputs using Jaccard similarity. |
-| `conversation_checkpoint` | Session | Save/load session state to prevent re-explaining context. |
-| `test_autofix_interceptor` | Automation | Run tests + Brain search for matching fixes. |
-| `prompt_pattern_cache` | Optimization | Find/store optimized prompt templates. |
-
-### 🗂️ Workspace Tools
-
-| Tool | Description |
-| :--- | :--- |
-| `read_workspace_file` | Read a file with automatic token compression and privacy scrubbing. |
-| `write_workspace_file` | Write content to a file, creating parent directories as needed. |
-| `create_workspace_file` | Create a new file (fails if exists). Returns minimal ACK. |
-| `apply_workspace_patch` | Smart search-and-replace patch — safe against duplicate matches. |
-| `multi_file_patch` | Apply patches to multiple files in a single call. |
-| `list_workspace_dir` | Recursively list the project structure up to a configurable depth. |
-| `search_workspace` | Fast text search (grep) across all project files. |
-| `semantic_grep` | Natural language concept search using BM25. |
-| `run_workspace_command` | Execute bash commands in the workspace root. Secured by allowlist. Distills long jest/vitest/tsc/eslint output down to the failures and their locations; full output kept retrievable. |
-| `output_region` | Retrieve any part of a stored command output verbatim (whole thing, one failing test, or a line range). |
-| `file_intent_index` | Group files by domain intent (auth, routing, config). |
-
-### 🔬 Code Intelligence Tools
-
-| Tool | Description |
-| :--- | :--- |
-| `get_ast_hologram` | Holographic AST Map of all exported symbols. 90% token savings. |
-| `validate_patch_sandbox` | Pre-flight validator: runs tsc + syntax check before applying. |
-| `diff_only` | Compact line-by-line diff between known states. Session-scoped. |
-| `surgical_ast_insert` | Insert code at AST anchor point. 95% output token savings. |
-| `get_symbol_surgical_context` | Extract symbol + dependency signatures only. 90% savings. |
-| `smart_file_slice` | Read only relevant lines around a query. 85% savings. |
-| `schema_extract` | Extract interfaces/types/schemas via AST. 80% savings. |
-| `spec_to_stub` | Generate mock stubs from TypeScript interfaces. |
-| `ast_flow_visualizer` | Mermaid flowchart from function control flow. |
-| `imports_skeleton_resolver` | Import signature mapper for quick reference. |
-
-### 🧠 Brain & Memory Tools
-
-| Tool | Description |
-| :--- | :--- |
-| `search_memory` | Semantic search across The Brain from all projects globally. |
-| `store_memory` | Persist solutions, fixes, decisions into The Brain. |
-| `query_hybrid_consensus` | Hybrid cache: 80% threshold → instant answer, 0 cloud tokens. |
-| `get_telepathic_hints` | Proactive hints from The Brain based on active file. |
-| `smarter_cache` | Predictive cache at 75% threshold before any reasoning. |
-| `semantic_dedup_guard` | Brain dedup firewall. Rejects ≥92% similar entries. |
-| `local_semantic_autofix` | Offline self-healing from Brain error index. |
-
-### ⚙️ Optimization & Privacy Tools
-
-| Tool | Description |
-| :--- | :--- |
-| `scrub_privacy` | Mask PII, API keys, secrets from any text block. |
-| `squeeze_prompt` | Compress code blocks. Up to 80% token reduction. |
-| `turbosqueeze` | Extreme compression: strips comments, imports, whitespace. |
-| `wormhole_squeeze` | Reversible unicode compression. 60-70% reduction. |
-| `compress_context` | Summarize old conversation turns. Keeps last N intact. |
-| `prune_conversation_history` | Intelligently decimate history preserving critical context. |
-| `summarize_long_text` | Summarize via Ollama. Falls back to truncation. |
-| `token_budget` | Dashboard: multiplier, savings, strategies, recommendations. |
-| `get_routing_advice` | Optimal model recommendation from Complexity Router. |
-| `get_project_onboarding` | One-shot architectural mental model. |
-| `auto_heal` | Auto-diagnose and repair crashes from live context. |
-| `batch_tool_calls` | Execute multiple tools in parallel via Promise.all. |
-
-### 🔮 Zero-Token Static Intelligence Tools
-
-> All tools in this section use **zero LLM tokens**. Pure local computation.
-
-| Tool | Description |
-| :--- | :--- |
-| `entropy_score` | Mathematical chaos score (0-100) per file. |
-| `coupling_radar` | Dependency graph: circular imports, god modules, dead islands. |
-| `pattern_fossil` | Zombie code detector: var vs const, callbacks vs async/await. |
-| `git_heatmap_risk` | Git churn analysis + co-edit pairs. |
-| `precrime_static` | Composite risk predictor combining entropy + coupling + git. |
-| `dead_export_necromancer` | Dead code auditor: exported but never imported. |
-| `git_blame_heat` | Git blame + per-line risk scoring. |
-| `type_coverage_report` | TypeScript type safety score. |
-| `import_tree_context` | Bidirectional import graph. |
-
-### 🏗️ Architecture & Refactoring Tools
-
-| Tool | Description |
-| :--- | :--- |
-| `depgraph` | Real-time dependency graph with depth, format, circular detection. |
-| `refactor` | Declarative codemod: rename symbols or move files. |
-| `cognitive_map` | Auto-updating knowledge graph of domains and patterns. |
-
-### 🔍 PR Review Agent Tools
-
-| Tool | Description |
-| :--- | :--- |
-| `review_diff` | Analyze raw diff for bugs, security, best practices. |
-| `review_pr` | Full PR review against GitHub, Azure DevOps, GitLab. |
-| `pr_status` | Query last review status for a PR. |
-| `generate_pr_workflow` | Generate CI/CD workflow YAML for PR reviews. |
-
-### 🧪 Testing & Quality Tools
-
-| Tool | Description |
-| :--- | :--- |
-| `test_oracle` | Run tests, return only failures with compressed traces. |
-| `test_autofix_interceptor` | Run tests + Brain search for matching fixes. |
-| `changelog_auto` | Conventional commit changelog from git history. |
-| `env_snapshot` | Node/npm versions, deps, .env keys (values hidden). |
-| `migration_tracer` | Prisma migration schema diff analysis. |
-
-### 📋 Session & Automation Tools
-
-| Tool | Description |
-| :--- | :--- |
-| `conversation_checkpoint` | Save/load session state to `.lemma/session/`. |
-| `prompt_pattern_cache` | Find/store optimized prompt templates. |
-| `review_pr` | Full PR review with auto-approve option. |
+Once connected, your IDE gains access to all **95 MCP tools** (46 Free + 49 Pro — see
+[Free vs Pro](#-free-vs-pro) for the full, current split) and Lemma auto-injects the TurboMode
+system prompt on handshake. By default only the tools your project shows evidence of needing are
+*advertised* each turn — see [Tool profiles](#-tool-profiles) — but every tool is registered and
+reachable via `lemma_toolbox` regardless of profile.
 
 ### ⚡ TurboMode Auto-Injection
 
-When connected via `lemma mcp`, Lemma auto-injects 12 rules into the agent's system prompt on handshake:
-
-1. `turbosqueeze` before writing large code blocks
-2. `smarter_cache` before any reasoning
-3. `store_memory` after every resolved solution
-4. `get_ast_hologram` before reading files
-5. `batch_tool_calls` for parallel operations
-6. `semantic_dedup_guard` before every `store_memory`
-7. No fluff, no chit-chat — maximum density
-8. `auto_context_bundle` or `bulk_file_digest` for multi-file ops
-9. `repeat_guard` before long outputs
-10. `conversation_checkpoint` to restore session context
-11. `test_autofix_interceptor` when tests fail
-12. `prompt_pattern_cache` for repetitive prompts
+When connected via `lemma mcp`, Lemma auto-injects its system prompt on handshake. The real,
+current prompt (`LEMMA_TURBOMODE_PROMPT` in `src/mcp/prompts.ts`) states the goal — reduce what
+enters context without ever costing a correct answer — and enforces it with rules including:
+
+- A tool call costs tokens too: expect a call to save more than its own round-trip, or skip it.
+- `lemma_toolbox` reaches every tool whose schema was withheld from this session — check it
+  before saying a capability is missing.
+- Compression (`turbosqueeze`, `wormhole_squeeze`, `squeeze_prompt`) is for the model's own
+  context only; the user always gets the real, uncompressed file.
+- `smarter_cache` hits must be disclosed as cached, never presented as freshly reasoned, and
+  re-verified before acting on them.
+- `state_hash_cache` (exact file-hash invalidation) is preferred over `smarter_cache` (semantic
+  similarity) whenever the answer depends on specific files.
+- `store_memory` only after a solution is actually verified — an unverified fix stored here
+  becomes a false cache hit later.
+- `get_routing_advice` before assuming a mechanical task needs the biggest available model.
+- `surgical_ast_insert` by default for adding to an existing file, instead of rewriting it whole.
+- `token_receipt` for "how much did we save," never an invented estimate.
 
 ---
 
@@ -921,15 +836,6 @@ See [`.env.security.example`](.env.security.example) for all security-related en
 
 ## 🦜 SDK Integrations
 
-### LangChain
-```typescript
-import { LemmaLangChainProvider } from '@nxuss/lemma/langchain';
-
-const model = new LemmaLangChainProvider({
-  apiKey: process.env.OPENAI_API_KEY,
-});
-```
-
 ### CrewAI
 ```typescript
 import { LemmaCrewAIProvider } from '@nxuss/lemma/crewai';

package/dist/cjs/mcp/tools.js

@@ -1624,10 +1624,19 @@ async function handleApplyWorkspacePatch(args) {
             };
         }
         const match = matchResult[0];
-        const contentLines = originalContent.split("\n");
-        const finalReplace = match.strategy === "exact" ? replaceContent : (0, PatchMatcher_1.reindentReplacement)(replaceContent, searchContent, contentLines[match.startLine]);
-        const updatedLines = [...contentLines.slice(0, match.startLine), ...finalReplace.split("\n"), ...contentLines.slice(match.endLine + 1)];
-        fs_1.default.writeFileSync(resolved, updatedLines.join("\n"), "utf8");
+        let updatedContent;
+        if (match.strategy === "exact" && match.charStart !== undefined && match.charEnd !== undefined) {
+            // Splice by character offset — the search block may be a mid-line fragment, and
+            // a whole-line splice would discard everything else sharing that line.
+            updatedContent = originalContent.slice(0, match.charStart) + replaceContent + originalContent.slice(match.charEnd);
+        }
+        else {
+            const contentLines = originalContent.split("\n");
+            const finalReplace = (0, PatchMatcher_1.reindentReplacement)(replaceContent, searchContent, contentLines[match.startLine]);
+            const updatedLines = [...contentLines.slice(0, match.startLine), ...finalReplace.split("\n"), ...contentLines.slice(match.endLine + 1)];
+            updatedContent = updatedLines.join("\n");
+        }
+        fs_1.default.writeFileSync(resolved, updatedContent, "utf8");
         const note = match.strategy === "exact" ? "" : ` (matched via ${match.strategy}, score ${match.score.toFixed(2)})`;
         return { content: [{ type: "text", text: `Success: Patch successfully applied to ${filePath}${note}` }] };
     }