engramx 0.2.1 → 0.3.1

package/README.md

@@ -15,79 +15,171 @@
   <a href="https://github.com/NickCirv/engram/actions"><img src="https://github.com/NickCirv/engram/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
   <img src="https://img.shields.io/badge/license-Apache%202.0-blue" alt="License">
   <img src="https://img.shields.io/badge/node-%3E%3D20-brightgreen" alt="Node">
-  <img src="https://img.shields.io/badge/tests-132%20passing-brightgreen" alt="Tests">
+  <img src="https://img.shields.io/badge/tests-439%20passing-brightgreen" alt="Tests">
   <img src="https://img.shields.io/badge/LLM%20cost-$0-green" alt="Zero LLM cost">
   <img src="https://img.shields.io/badge/native%20deps-zero-green" alt="Zero native deps">
+  <img src="https://img.shields.io/badge/token%20reduction-82%25-orange" alt="82% token reduction">
 </p>
 
 ---
 
-**Your AI coding assistant forgets everything. We fixed that.**
+# The structural code graph your AI agent can't forget to use.
 
-engram gives AI coding tools persistent memory. One command scans your codebase, builds a knowledge graph, and makes every session start where the last one left off.
+**Context rot is empirically solved.** Chroma's July 2025 research proved that even Claude Opus 4.6 scores only 76% on MRCR v2 8-needle at 1M tokens. Long context windows don't save you — they drown you. engram is the answer: a **structural graph** of your codebase that replaces file reads with ~300-token summaries *before the agent sees them*.
 
-Zero LLM cost. Zero cloud. Works with Claude Code, Cursor, Codex, aider, and any MCP client.
+engram installs a Claude Code hook layer at the tool-call boundary. Every `Read`, `Edit`, `Write`, and `Bash cat` gets intercepted. When the graph has confident coverage of a file, the raw read never happens — the agent sees a structural summary instead.
+
+Not a memory tool. Not a RAG layer. Not a context manager. **A structural code graph with a Claude Code hook layer that turns it into the memory your agent can't forget to exist.**
+
+| What it is | What it isn't |
+|---|---|
+| Structural code graph (AST + git + session miners) | Prose memory like Anthropic's MEMORY.md |
+| Local SQLite, zero cloud, zero native deps | Vector RAG that phones home |
+| Hook-based interception at the tool boundary | A tool the agent has to remember to call |
+| 82% measured token reduction on real code | Another LongMemEval chatbot benchmark |
+| Complements native Claude memory | Competes with native Claude memory |
 
 ```bash
-npx engram init
+npm install -g engramx
+cd ~/my-project
+engram init             # scan codebase → .engram/graph.db (~40 ms, 0 tokens)
+engram install-hook     # wire the Sentinel into Claude Code
 ```
 
+```bash
+npm install -g engramx
+cd ~/my-project
+engram init             # scan codebase → .engram/graph.db
+engram install-hook     # wire into Claude Code (project-local)
 ```
-🔍 Scanning codebase...
-🌳 AST extraction complete (42ms, 0 tokens used)
-   60 nodes, 96 edges from 14 files (2,155 lines)
 
-📊 Token savings: 11x fewer tokens vs relevant files
-   Full corpus: ~15,445 tokens | Graph query: ~285 tokens
+That's it. The next Claude Code session in that directory automatically:
 
-✅ Ready. Your AI now has persistent memory.
-```
+- **Replaces file reads with graph summaries** (Read intercept, deny+reason)
+- **Warns before edits that hit known mistakes** (Edit landmine injection)
+- **Pre-loads relevant context when you ask a question** (UserPromptSubmit pre-query)
+- **Injects a project brief at session start** (SessionStart additionalContext)
+- **Logs every decision for `engram hook-stats`** (PostToolUse observer)
 
-## Why
+## Architecture Diagram
 
-Every AI coding session starts from zero. Claude Code re-reads your files. Cursor reindexes. Copilot has no memory. CLAUDE.md is a sticky note you write by hand.
+An 11-page visual walkthrough of the full lifecycle — the four hook events, the Read handler's 9-branch decision tree (with a real JSON response), the six-layer ecosystem substrate, and measured numbers from engram's own code.
 
-engram fixes this with five things no other tool combines:
+- 📄 **[View the PDF](docs/engram-sentinel-ecosystem.pdf)** — A3 landscape, 11 pages, 1 MB. GitHub renders it inline when clicked.
+- 🌐 **[HTML source](docs/engram-sentinel-ecosystem.html)** — single self-contained file. Download raw and open in any browser for the interactive scroll-reveal version.
 
-1. **Persistent knowledge graph** — survives across sessions, stored in `.engram/graph.db`
-2. **Learns from every session** — decisions, patterns, mistakes are extracted and remembered
-3. **Universal protocol** — MCP server + CLI + auto-generates CLAUDE.md, .cursorrules, AGENTS.md
-4. **Skill-aware** (v0.2) — indexes your `~/.claude/skills/` directory into the graph so queries return code *and* the skill to apply
-5. **Regret buffer** (v0.2) — surfaces past mistakes at the top of query results so your AI stops re-making the same wrong turns
+## The Problem
 
-## Install
+Every Claude Code session burns ~52,500 tokens on things you already told the agent yesterday. Reading the same files, re-exploring the same modules, re-discovering the same patterns. Even with a great CLAUDE.md, the agent still falls back to `Read` because `Read` is what it knows.
 
-```bash
-npx engramx init
+The ceiling isn't the graph's accuracy. It's that the agent has to *remember* to ask. v0.2 of engram was a tool the agent queried ~5 times per session. The other 25 Reads happened uninterrupted.
+
+v0.3 flips this. The hook intercepts at the tool-call boundary, not at the agent's discretion.
+
+```
+v0.2: agent → (remembers to call query_graph) → engram returns summary
+v0.3: agent → Read → Claude Code hook → engram intercepts → summary delivered
 ```
 
-Or install globally:
+**Projected savings: -42,500 tokens per session** (~80% reduction vs v0.2.1 baseline).
+Every number is arithmetic on empirically verified hook mechanisms — not estimates.
+
+## Install
 
 ```bash
 npm install -g engramx
-engram init
 ```
 
 Requires Node.js 20+. Zero native dependencies. No build tools needed.
 
-## Usage
+## Quickstart (v0.3 Sentinel)
+
+```bash
+cd ~/my-project
+engram init                    # scan codebase, build knowledge graph
+engram install-hook            # install Sentinel hooks into .claude/settings.local.json
+engram hook-preview src/auth.ts  # dry-run: see what the hook would do
+```
+
+Open a Claude Code session in that project. When it reads a well-covered file, you'll see a system-reminder with engram's structural summary instead of the full file contents. Run `engram hook-stats` afterwards to see how many reads were intercepted.
+
+```bash
+engram hook-stats              # summarize hook-log.jsonl
+engram hook-disable            # kill switch (keeps install, disables intercepts)
+engram hook-enable             # re-enable
+engram uninstall-hook          # surgical removal, preserves other hooks
+```
+
+## All Commands
+
+### Core (v0.1/v0.2 — unchanged)
 
 ```bash
 engram init [path]              # Scan codebase, build knowledge graph
 engram init --with-skills       # Also index ~/.claude/skills/ (v0.2)
 engram query "how does auth"    # Query the graph (BFS, token-budgeted)
-engram query "auth" --dfs       # DFS traversal (trace specific paths)
+engram query "auth" --dfs       # DFS traversal
 engram gods                     # Show most connected entities
 engram stats                    # Node/edge counts, token savings
 engram bench                    # Token reduction benchmark
 engram path "auth" "database"   # Shortest path between concepts
 engram learn "chose JWT..."     # Teach a decision or pattern
-engram mistakes                 # List known mistakes (v0.2)
+engram mistakes                 # List known landmines
 engram gen                      # Generate CLAUDE.md section from graph
-engram gen --task bug-fix       # Task-aware view (v0.2: general|bug-fix|feature|refactor)
-engram hooks install            # Auto-rebuild on git commit
+engram gen --task bug-fix       # Task-aware view (general|bug-fix|feature|refactor)
+engram hooks install            # Auto-rebuild graph on git commit
+```
+
+### Sentinel (v0.3 — new)
+
+```bash
+engram intercept                        # Hook entry point (called by Claude Code, reads stdin)
+engram install-hook [--scope <s>]       # Install hooks into Claude Code settings
+                                        #   --scope local (default, gitignored)
+                                        #   --scope project (committed)
+                                        #   --scope user (global ~/.claude/settings.json)
+engram install-hook --dry-run           # Preview changes without writing
+engram uninstall-hook                   # Remove engram entries (preserves other hooks)
+engram hook-stats                       # Summarize .engram/hook-log.jsonl
+engram hook-stats --json                # Machine-readable output
+engram hook-preview <file>              # Dry-run Read handler for a specific file
+engram hook-disable                     # Kill switch (touch .engram/hook-disabled)
+engram hook-enable                      # Remove kill switch
 ```
 
+## How the Sentinel Layer Works
+
+Seven hook handlers compose the interception stack:
+
+| Hook | Mechanism | What it does |
+|---|---|---|
+| **`PreToolUse:Read`** | `deny + permissionDecisionReason` | If the file is in the graph with ≥0.7 confidence, blocks the Read and delivers a ~300-token structural summary as the block reason. Claude sees the reason as a system-reminder and uses it as context. The file is never actually read. |
+| **`PreToolUse:Edit`** | `allow + additionalContext` | Never blocks writes. If the file has known past mistakes, injects them as a landmine warning alongside the edit. |
+| **`PreToolUse:Write`** | Same as Edit | Advisory landmine injection. |
+| **`PreToolUse:Bash`** | Parse + delegate | Detects `cat|head|tail|less|more <single-file>` invocations (strict parser, rejects any shell metacharacter) and delegates to the Read handler. Closes the Bash workaround loophole. |
+| **`SessionStart`** | `additionalContext` | Injects a compact project brief (god nodes + graph stats + top landmines + git branch) on source=startup/clear/compact. Passes through on resume. |
+| **`UserPromptSubmit`** | `additionalContext` | Extracts keywords from the user's message, runs a ≤500-token pre-query, injects results. Skipped for short or generic prompts. Raw prompt content is never logged. |
+| **`PostToolUse`** | Observer | Pure logger. Writes tool/path/outputSize/success/decision to `.engram/hook-log.jsonl` for `hook-stats` and v0.3.1 self-tuning. |
+
+### Ten safety invariants, enforced at runtime
+
+1. Any handler error → passthrough (never block Claude Code)
+2. 2-second per-handler timeout
+3. Kill switch (`.engram/hook-disabled`) respected by every handler
+4. Atomic settings.json writes with timestamped backups
+5. Never intercept outside the project root
+6. Never intercept binary files or secrets (.env, .pem, .key, credentials, id_rsa, ...)
+7. Never log user prompt content (privacy invariant asserted in tests)
+8. Never inject >8000 chars per hook response
+9. Stale graph detection (file mtime > graph mtime → passthrough)
+10. Partial-read bypass (Read with explicit `offset` or `limit` → passthrough)
+
+### What you can safely install
+
+Default scope is `.claude/settings.local.json` — gitignored, project-local, zero risk of committing hook config to a shared repo. Idempotent install. Non-destructive uninstall. `--dry-run` shows the diff before writing.
+
+If anything goes wrong, `engram hook-disable` flips the kill switch without uninstalling.
+
 ## How It Works
 
 engram runs three miners on your codebase. None of them use an LLM.

package/dist/{chunk-YQ3FPGPC.js → chunk-IYO4HETA.js}

@@ -310,7 +310,7 @@ function writeToFile(filePath, summary) {
   writeFileSync2(filePath, newContent);
 }
 async function autogen(projectRoot, target, task) {
-  const { getStore } = await import("./core-HWOM7GSU.js");
+  const { getStore } = await import("./core-2TWPNHRQ.js");
   const store = await getStore(projectRoot);
   try {
     let view = VIEWS.general;

package/dist/{chunk-22ADJYQ5.js → chunk-RGDHLGWQ.js}

@@ -1,6 +1,6 @@
 // src/core.ts
-import { join as join4, resolve as resolve2 } from "path";
-import { existsSync as existsSync5, mkdirSync as mkdirSync2, readFileSync as readFileSync5, writeFileSync as writeFileSync2, unlinkSync } from "fs";
+import { join as join4, resolve as resolve2, relative as relative2 } from "path";
+import { existsSync as existsSync5, mkdirSync as mkdirSync2, readFileSync as readFileSync5, writeFileSync as writeFileSync2, unlinkSync, statSync as statSync2 } from "fs";
 import { homedir } from "os";
 
 // src/graph/store.ts
@@ -526,6 +526,106 @@ function renderPath(nodes, edges) {
   }
   return `Path (${edges.length} hops): ${segments.join(" ")}`;
 }
+function renderFileStructure(store, relativeFilePath, tokenBudget = 600) {
+  const allNodes = store.getAllNodes();
+  const fileNodes = allNodes.filter(
+    (n) => n.sourceFile === relativeFilePath && !isHiddenKeyword(n)
+  );
+  if (fileNodes.length === 0) {
+    return {
+      text: "",
+      nodeCount: 0,
+      codeNodeCount: 0,
+      avgConfidence: 0,
+      estimatedTokens: 0
+    };
+  }
+  const codeNodeCount = fileNodes.filter(
+    (n) => n.kind !== "file" && n.kind !== "module"
+  ).length;
+  const avgConfidence = fileNodes.reduce((s, n) => s + n.confidenceScore, 0) / fileNodes.length;
+  const allEdges = store.getAllEdges();
+  const fileNodeIds = new Set(fileNodes.map((n) => n.id));
+  const degreeMap = /* @__PURE__ */ new Map();
+  for (const e of allEdges) {
+    if (fileNodeIds.has(e.source)) {
+      degreeMap.set(e.source, (degreeMap.get(e.source) ?? 0) + 1);
+    }
+    if (fileNodeIds.has(e.target)) {
+      degreeMap.set(e.target, (degreeMap.get(e.target) ?? 0) + 1);
+    }
+  }
+  const byKind = /* @__PURE__ */ new Map();
+  for (const n of fileNodes) {
+    const list = byKind.get(n.kind) ?? [];
+    list.push(n);
+    byKind.set(n.kind, list);
+  }
+  for (const list of byKind.values()) {
+    list.sort(
+      (a, b) => (degreeMap.get(b.id) ?? 0) - (degreeMap.get(a.id) ?? 0)
+    );
+  }
+  const kindOrder = [
+    "class",
+    "interface",
+    "type",
+    "function",
+    "method",
+    "variable",
+    "import",
+    "module",
+    "file",
+    "decision",
+    "pattern",
+    "mistake",
+    "concept"
+  ];
+  const lines = [];
+  lines.push(`[engram] Structural summary for ${relativeFilePath}`);
+  lines.push(
+    `Nodes: ${fileNodes.length} | avg extraction confidence: ${avgConfidence.toFixed(2)}`
+  );
+  lines.push("");
+  for (const kind of kindOrder) {
+    const group = byKind.get(kind);
+    if (!group || group.length === 0) continue;
+    for (const n of group) {
+      const loc = n.sourceLocation ?? "";
+      lines.push(`NODE ${n.label} [${n.kind}] ${loc}`.trim());
+    }
+  }
+  const relevantEdges = allEdges.filter(
+    (e) => fileNodeIds.has(e.source) || fileNodeIds.has(e.target)
+  ).slice(0, 10);
+  if (relevantEdges.length > 0) {
+    lines.push("");
+    lines.push("Key relationships:");
+    for (const e of relevantEdges) {
+      const src = allNodes.find((n) => n.id === e.source);
+      const tgt = allNodes.find((n) => n.id === e.target);
+      if (src && tgt) {
+        lines.push(`EDGE ${src.label} --${e.relation}--> ${tgt.label}`);
+      }
+    }
+  }
+  lines.push("");
+  lines.push(
+    "Note: engram replaced a full-file read with this structural view to save tokens. If you need specific lines, Read this file again with explicit offset/limit parameters \u2014 engram passes partial reads through unchanged."
+  );
+  let text = lines.join("\n");
+  const charBudget = tokenBudget * CHARS_PER_TOKEN;
+  if (text.length > charBudget) {
+    text = sliceGraphemeSafe(text, charBudget) + "\n... (truncated to fit summary budget)";
+  }
+  return {
+    text,
+    nodeCount: fileNodes.length,
+    codeNodeCount,
+    avgConfidence,
+    estimatedTokens: Math.ceil(text.length / CHARS_PER_TOKEN)
+  };
+}
 
 // src/miners/ast-miner.ts
 import { readFileSync as readFileSync2, readdirSync, realpathSync } from "fs";
@@ -1490,6 +1590,111 @@ async function stats(projectRoot) {
     store.close();
   }
 }
+var FILE_CONTEXT_COVERAGE_CEILING = 3;
+async function getFileContext(projectRoot, absFilePath) {
+  const empty = {
+    found: false,
+    confidence: 0,
+    summary: "",
+    nodeCount: 0,
+    codeNodeCount: 0,
+    avgNodeConfidence: 0,
+    graphMtimeMs: 0,
+    fileMtimeMs: null,
+    isStale: false
+  };
+  try {
+    const root = resolve2(projectRoot);
+    const abs = resolve2(absFilePath);
+    const relPath = relative2(root, abs);
+    if (relPath.startsWith("..") || relPath === "") {
+      return empty;
+    }
+    const dbPath = getDbPath(root);
+    let graphMtimeMs = 0;
+    try {
+      graphMtimeMs = statSync2(dbPath).mtimeMs;
+    } catch {
+      return empty;
+    }
+    let fileMtimeMs = null;
+    try {
+      fileMtimeMs = statSync2(abs).mtimeMs;
+    } catch {
+      fileMtimeMs = null;
+    }
+    const isStale = fileMtimeMs !== null && fileMtimeMs > graphMtimeMs;
+    const store = await getStore(root);
+    try {
+      const summary = renderFileStructure(store, relPath);
+      if (summary.codeNodeCount === 0) {
+        return {
+          ...empty,
+          nodeCount: summary.nodeCount,
+          codeNodeCount: 0,
+          graphMtimeMs,
+          fileMtimeMs,
+          isStale
+        };
+      }
+      const coverageScore = Math.min(
+        summary.codeNodeCount / FILE_CONTEXT_COVERAGE_CEILING,
+        1
+      );
+      const confidence = coverageScore * summary.avgConfidence;
+      return {
+        found: true,
+        confidence,
+        summary: summary.text,
+        nodeCount: summary.nodeCount,
+        codeNodeCount: summary.codeNodeCount,
+        avgNodeConfidence: summary.avgConfidence,
+        graphMtimeMs,
+        fileMtimeMs,
+        isStale
+      };
+    } finally {
+      store.close();
+    }
+  } catch {
+    return empty;
+  }
+}
+async function computeKeywordIDF(projectRoot, keywords) {
+  if (keywords.length === 0) return [];
+  try {
+    const root = resolve2(projectRoot);
+    const dbPath = getDbPath(root);
+    if (!existsSync5(dbPath)) return [];
+    const store = await getStore(root);
+    try {
+      const allNodes = store.getAllNodes();
+      const total = allNodes.length;
+      if (total === 0) return [];
+      const labels = allNodes.map((n) => n.label.toLowerCase());
+      const results = [];
+      for (const kw of keywords) {
+        const kwLower = kw.toLowerCase();
+        let df = 0;
+        for (const label of labels) {
+          if (label.includes(kwLower)) df += 1;
+        }
+        const idf = df === 0 ? 0 : Math.log(total / df);
+        results.push({
+          keyword: kw,
+          documentFrequency: df,
+          idf
+        });
+      }
+      results.sort((a, b) => b.idf - a.idf);
+      return results;
+    } finally {
+      store.close();
+    }
+  } catch {
+    return [];
+  }
+}
 async function learn(projectRoot, text, sourceLabel = "manual") {
   const { nodes, edges } = learnFromSession(text, sourceLabel);
   if (nodes.length === 0 && edges.length === 0) return { nodesAdded: 0 };
@@ -1505,6 +1710,10 @@ async function mistakes(projectRoot, options = {}) {
   const store = await getStore(projectRoot);
   try {
     let items = store.getAllNodes().filter((n) => n.kind === "mistake");
+    if (options.sourceFile !== void 0) {
+      const target = options.sourceFile;
+      items = items.filter((m) => m.sourceFile === target);
+    }
     if (options.sinceDays !== void 0) {
       const cutoff = Date.now() - options.sinceDays * 24 * 60 * 60 * 1e3;
       items = items.filter((m) => m.lastVerified >= cutoff);
@@ -1603,6 +1812,8 @@ export {
   path,
   godNodes,
   stats,
+  getFileContext,
+  computeKeywordIDF,
   learn,
   mistakes,
   benchmark