engramx 0.2.0 → 0.3.0

package/README.md

@@ -15,79 +15,148 @@
   <a href="https://github.com/NickCirv/engram/actions"><img src="https://github.com/NickCirv/engram/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
   <img src="https://img.shields.io/badge/license-Apache%202.0-blue" alt="License">
   <img src="https://img.shields.io/badge/node-%3E%3D20-brightgreen" alt="Node">
-  <img src="https://img.shields.io/badge/tests-132%20passing-brightgreen" alt="Tests">
+  <img src="https://img.shields.io/badge/tests-439%20passing-brightgreen" alt="Tests">
   <img src="https://img.shields.io/badge/LLM%20cost-$0-green" alt="Zero LLM cost">
   <img src="https://img.shields.io/badge/native%20deps-zero-green" alt="Zero native deps">
 </p>
 
 ---
 
-**Your AI coding assistant forgets everything. We fixed that.**
+**Context as infra for your AI coding tools.**
 
-engram gives AI coding tools persistent memory. One command scans your codebase, builds a knowledge graph, and makes every session start where the last one left off.
+engram installs a Claude Code hook layer that intercepts every `Read`, `Edit`, `Write`, and `Bash cat` — replacing full file reads with ~300-token structural graph summaries *before the agent even sees them*. No more re-exploring the codebase every session. No more agents forgetting to use the tool you gave them.
 
-Zero LLM cost. Zero cloud. Works with Claude Code, Cursor, Codex, aider, and any MCP client.
+**v0.3 "Sentinel":** the agent can't forget to use engram because engram sits between the agent and the filesystem.
+
+Zero LLM cost. Zero cloud. Zero native deps. Works today in Claude Code.
 
 ```bash
-npx engram init
+npm install -g engramx
+cd ~/my-project
+engram init             # scan codebase → .engram/graph.db
+engram install-hook     # wire into Claude Code (project-local)
 ```
 
-```
-🔍 Scanning codebase...
-🌳 AST extraction complete (42ms, 0 tokens used)
-   60 nodes, 96 edges from 14 files (2,155 lines)
+That's it. The next Claude Code session in that directory automatically:
 
-📊 Token savings: 11x fewer tokens vs relevant files
-   Full corpus: ~15,445 tokens | Graph query: ~285 tokens
+- **Replaces file reads with graph summaries** (Read intercept, deny+reason)
+- **Warns before edits that hit known mistakes** (Edit landmine injection)
+- **Pre-loads relevant context when you ask a question** (UserPromptSubmit pre-query)
+- **Injects a project brief at session start** (SessionStart additionalContext)
+- **Logs every decision for `engram hook-stats`** (PostToolUse observer)
 
-✅ Ready. Your AI now has persistent memory.
-```
+## The Problem
 
-## Why
+Every Claude Code session burns ~52,500 tokens on things you already told the agent yesterday. Reading the same files, re-exploring the same modules, re-discovering the same patterns. Even with a great CLAUDE.md, the agent still falls back to `Read` because `Read` is what it knows.
 
-Every AI coding session starts from zero. Claude Code re-reads your files. Cursor reindexes. Copilot has no memory. CLAUDE.md is a sticky note you write by hand.
+The ceiling isn't the graph's accuracy. It's that the agent has to *remember* to ask. v0.2 of engram was a tool the agent queried ~5 times per session. The other 25 Reads happened uninterrupted.
 
-engram fixes this with five things no other tool combines:
+v0.3 flips this. The hook intercepts at the tool-call boundary, not at the agent's discretion.
+
+```
+v0.2: agent → (remembers to call query_graph) → engram returns summary
+v0.3: agent → Read → Claude Code hook → engram intercepts → summary delivered
+```
 
-1. **Persistent knowledge graph** — survives across sessions, stored in `.engram/graph.db`
-2. **Learns from every session** — decisions, patterns, mistakes are extracted and remembered
-3. **Universal protocol** — MCP server + CLI + auto-generates CLAUDE.md, .cursorrules, AGENTS.md
-4. **Skill-aware** (v0.2) — indexes your `~/.claude/skills/` directory into the graph so queries return code *and* the skill to apply
-5. **Regret buffer** (v0.2) — surfaces past mistakes at the top of query results so your AI stops re-making the same wrong turns
+**Projected savings: -42,500 tokens per session** (~80% reduction vs v0.2.1 baseline).
+Every number is arithmetic on empirically verified hook mechanisms — not estimates.
 
 ## Install
 
 ```bash
-npx engramx init
+npm install -g engramx
 ```
 
-Or install globally:
+Requires Node.js 20+. Zero native dependencies. No build tools needed.
+
+## Quickstart (v0.3 Sentinel)
 
 ```bash
-npm install -g engramx
-engram init
+cd ~/my-project
+engram init                    # scan codebase, build knowledge graph
+engram install-hook            # install Sentinel hooks into .claude/settings.local.json
+engram hook-preview src/auth.ts  # dry-run: see what the hook would do
 ```
 
-Requires Node.js 20+. Zero native dependencies. No build tools needed.
+Open a Claude Code session in that project. When it reads a well-covered file, you'll see a system-reminder with engram's structural summary instead of the full file contents. Run `engram hook-stats` afterwards to see how many reads were intercepted.
+
+```bash
+engram hook-stats              # summarize hook-log.jsonl
+engram hook-disable            # kill switch (keeps install, disables intercepts)
+engram hook-enable             # re-enable
+engram uninstall-hook          # surgical removal, preserves other hooks
+```
+
+## All Commands
 
-## Usage
+### Core (v0.1/v0.2 — unchanged)
 
 ```bash
 engram init [path]              # Scan codebase, build knowledge graph
 engram init --with-skills       # Also index ~/.claude/skills/ (v0.2)
 engram query "how does auth"    # Query the graph (BFS, token-budgeted)
-engram query "auth" --dfs       # DFS traversal (trace specific paths)
+engram query "auth" --dfs       # DFS traversal
 engram gods                     # Show most connected entities
 engram stats                    # Node/edge counts, token savings
 engram bench                    # Token reduction benchmark
 engram path "auth" "database"   # Shortest path between concepts
 engram learn "chose JWT..."     # Teach a decision or pattern
-engram mistakes                 # List known mistakes (v0.2)
+engram mistakes                 # List known landmines
 engram gen                      # Generate CLAUDE.md section from graph
-engram gen --task bug-fix       # Task-aware view (v0.2: general|bug-fix|feature|refactor)
-engram hooks install            # Auto-rebuild on git commit
+engram gen --task bug-fix       # Task-aware view (general|bug-fix|feature|refactor)
+engram hooks install            # Auto-rebuild graph on git commit
 ```
 
+### Sentinel (v0.3 — new)
+
+```bash
+engram intercept                        # Hook entry point (called by Claude Code, reads stdin)
+engram install-hook [--scope <s>]       # Install hooks into Claude Code settings
+                                        #   --scope local (default, gitignored)
+                                        #   --scope project (committed)
+                                        #   --scope user (global ~/.claude/settings.json)
+engram install-hook --dry-run           # Preview changes without writing
+engram uninstall-hook                   # Remove engram entries (preserves other hooks)
+engram hook-stats                       # Summarize .engram/hook-log.jsonl
+engram hook-stats --json                # Machine-readable output
+engram hook-preview <file>              # Dry-run Read handler for a specific file
+engram hook-disable                     # Kill switch (touch .engram/hook-disabled)
+engram hook-enable                      # Remove kill switch
+```
+
+## How the Sentinel Layer Works
+
+Seven hook handlers compose the interception stack:
+
+| Hook | Mechanism | What it does |
+|---|---|---|
+| **`PreToolUse:Read`** | `deny + permissionDecisionReason` | If the file is in the graph with ≥0.7 confidence, blocks the Read and delivers a ~300-token structural summary as the block reason. Claude sees the reason as a system-reminder and uses it as context. The file is never actually read. |
+| **`PreToolUse:Edit`** | `allow + additionalContext` | Never blocks writes. If the file has known past mistakes, injects them as a landmine warning alongside the edit. |
+| **`PreToolUse:Write`** | Same as Edit | Advisory landmine injection. |
+| **`PreToolUse:Bash`** | Parse + delegate | Detects `cat|head|tail|less|more <single-file>` invocations (strict parser, rejects any shell metacharacter) and delegates to the Read handler. Closes the Bash workaround loophole. |
+| **`SessionStart`** | `additionalContext` | Injects a compact project brief (god nodes + graph stats + top landmines + git branch) on source=startup/clear/compact. Passes through on resume. |
+| **`UserPromptSubmit`** | `additionalContext` | Extracts keywords from the user's message, runs a ≤500-token pre-query, injects results. Skipped for short or generic prompts. Raw prompt content is never logged. |
+| **`PostToolUse`** | Observer | Pure logger. Writes tool/path/outputSize/success/decision to `.engram/hook-log.jsonl` for `hook-stats` and v0.3.1 self-tuning. |
+
+### Ten safety invariants, enforced at runtime
+
+1. Any handler error → passthrough (never block Claude Code)
+2. 2-second per-handler timeout
+3. Kill switch (`.engram/hook-disabled`) respected by every handler
+4. Atomic settings.json writes with timestamped backups
+5. Never intercept outside the project root
+6. Never intercept binary files or secrets (.env, .pem, .key, credentials, id_rsa, ...)
+7. Never log user prompt content (privacy invariant asserted in tests)
+8. Never inject >8000 chars per hook response
+9. Stale graph detection (file mtime > graph mtime → passthrough)
+10. Partial-read bypass (Read with explicit `offset` or `limit` → passthrough)
+
+### What you can safely install
+
+Default scope is `.claude/settings.local.json` — gitignored, project-local, zero risk of committing hook config to a shared repo. Idempotent install. Non-destructive uninstall. `--dry-run` shows the diff before writing.
+
+If anything goes wrong, `engram hook-disable` flips the kill switch without uninstalling.
+
 ## How It Works
 
 engram runs three miners on your codebase. None of them use an LLM.

package/dist/{chunk-QKCPFSVU.js → chunk-7KYR4SPZ.js}

@@ -310,7 +310,7 @@ function writeToFile(filePath, summary) {
   writeFileSync2(filePath, newContent);
 }
 async function autogen(projectRoot, target, task) {
-  const { getStore } = await import("./core-H72MM256.js");
+  const { getStore } = await import("./core-MPNNCPFW.js");
   const store = await getStore(projectRoot);
   try {
     let view = VIEWS.general;

package/dist/{chunk-D53DRZZL.js → chunk-NYFDM4FR.js}

@@ -1,6 +1,6 @@
 // src/core.ts
-import { join as join4, resolve as resolve2 } from "path";
-import { existsSync as existsSync5, mkdirSync as mkdirSync2, readFileSync as readFileSync5, writeFileSync as writeFileSync2, unlinkSync } from "fs";
+import { join as join4, resolve as resolve2, relative as relative2 } from "path";
+import { existsSync as existsSync5, mkdirSync as mkdirSync2, readFileSync as readFileSync5, writeFileSync as writeFileSync2, unlinkSync, statSync as statSync2 } from "fs";
 import { homedir } from "os";
 
 // src/graph/store.ts
@@ -315,7 +315,13 @@ function truncateGraphemeSafe(s, max) {
 
 // src/graph/query.ts
 var MISTAKE_SCORE_BOOST = 2.5;
+var KEYWORD_SCORE_DOWNWEIGHT = 0.5;
 var MAX_MISTAKE_LABEL_CHARS = 500;
+function isHiddenKeyword(node) {
+  if (node.kind !== "concept") return false;
+  const meta = node.metadata;
+  return meta?.subkind === "keyword";
+}
 var CHARS_PER_TOKEN = 4;
 function scoreNodes(store, terms) {
   const allNodes = store.getAllNodes();
@@ -330,6 +336,7 @@ function scoreNodes(store, terms) {
     }
     if (score > 0) {
       if (node.kind === "mistake") score *= MISTAKE_SCORE_BOOST;
+      if (isHiddenKeyword(node)) score *= KEYWORD_SCORE_DOWNWEIGHT;
       scored.push({ score, node });
     }
   }
@@ -346,6 +353,12 @@ function queryGraph(store, question, options = {}) {
   for (const n of startNodes) store.incrementQueryCount(n.id);
   const visited = new Set(startNodes.map((n) => n.id));
   const collectedEdges = [];
+  const shouldSkipEdgeFrom = (currentNodeId, edge) => {
+    if (edge.relation !== "triggered_by") return false;
+    const currentNode = store.getNode(currentNodeId);
+    if (!currentNode) return false;
+    return !isHiddenKeyword(currentNode);
+  };
   if (mode === "bfs") {
     let frontier = new Set(startNodes.map((n) => n.id));
     for (let d = 0; d < depth; d++) {
@@ -353,6 +366,7 @@ function queryGraph(store, question, options = {}) {
       for (const nid of frontier) {
         const neighbors = store.getNeighbors(nid);
         for (const { node, edge } of neighbors) {
+          if (shouldSkipEdgeFrom(nid, edge)) continue;
           if (!visited.has(node.id)) {
             nextFrontier.add(node.id);
             collectedEdges.push(edge);
@@ -369,6 +383,7 @@ function queryGraph(store, question, options = {}) {
       if (d > depth) continue;
       const neighbors = store.getNeighbors(id);
       for (const { node, edge } of neighbors) {
+        if (shouldSkipEdgeFrom(id, edge)) continue;
         if (!visited.has(node.id)) {
           visited.add(node.id);
           stack.push({ id: node.id, d: d + 1 });
@@ -446,7 +461,12 @@ function renderSubgraph(nodes, edges, tokenBudget) {
   const charBudget = tokenBudget * CHARS_PER_TOKEN;
   const lines = [];
   const mistakes2 = nodes.filter((n) => n.kind === "mistake");
-  const nonMistakes = nodes.filter((n) => n.kind !== "mistake");
+  const visible = nodes.filter(
+    (n) => n.kind !== "mistake" && !isHiddenKeyword(n)
+  );
+  const hiddenKeywordIds = new Set(
+    nodes.filter(isHiddenKeyword).map((n) => n.id)
+  );
   if (mistakes2.length > 0) {
     lines.push("\u26A0\uFE0F PAST MISTAKES (relevant to your query):");
     for (const m of mistakes2) {
@@ -461,7 +481,7 @@ function renderSubgraph(nodes, edges, tokenBudget) {
     degreeMap.set(e.source, (degreeMap.get(e.source) ?? 0) + 1);
     degreeMap.set(e.target, (degreeMap.get(e.target) ?? 0) + 1);
   }
-  const sorted = [...nonMistakes].sort(
+  const sorted = [...visible].sort(
     (a, b) => (degreeMap.get(b.id) ?? 0) - (degreeMap.get(a.id) ?? 0)
   );
   for (const n of sorted) {
@@ -469,7 +489,18 @@ function renderSubgraph(nodes, edges, tokenBudget) {
       `NODE ${n.label} [${n.kind}] src=${n.sourceFile} ${n.sourceLocation ?? ""}`
     );
   }
+  const skillConceptIds = new Set(
+    nodes.filter(
+      (n) => n.kind === "concept" && n.metadata?.subkind === "skill"
+    ).map((n) => n.id)
+  );
   for (const e of edges) {
+    if (hiddenKeywordIds.has(e.source) || hiddenKeywordIds.has(e.target)) {
+      continue;
+    }
+    if (e.relation === "similar_to" && skillConceptIds.has(e.source) && skillConceptIds.has(e.target)) {
+      continue;
+    }
     const srcNode = nodes.find((n) => n.id === e.source);
     const tgtNode = nodes.find((n) => n.id === e.target);
     if (srcNode && tgtNode) {
@@ -495,6 +526,106 @@ function renderPath(nodes, edges) {
   }
   return `Path (${edges.length} hops): ${segments.join(" ")}`;
 }
+function renderFileStructure(store, relativeFilePath, tokenBudget = 600) {
+  const allNodes = store.getAllNodes();
+  const fileNodes = allNodes.filter(
+    (n) => n.sourceFile === relativeFilePath && !isHiddenKeyword(n)
+  );
+  if (fileNodes.length === 0) {
+    return {
+      text: "",
+      nodeCount: 0,
+      codeNodeCount: 0,
+      avgConfidence: 0,
+      estimatedTokens: 0
+    };
+  }
+  const codeNodeCount = fileNodes.filter(
+    (n) => n.kind !== "file" && n.kind !== "module"
+  ).length;
+  const avgConfidence = fileNodes.reduce((s, n) => s + n.confidenceScore, 0) / fileNodes.length;
+  const allEdges = store.getAllEdges();
+  const fileNodeIds = new Set(fileNodes.map((n) => n.id));
+  const degreeMap = /* @__PURE__ */ new Map();
+  for (const e of allEdges) {
+    if (fileNodeIds.has(e.source)) {
+      degreeMap.set(e.source, (degreeMap.get(e.source) ?? 0) + 1);
+    }
+    if (fileNodeIds.has(e.target)) {
+      degreeMap.set(e.target, (degreeMap.get(e.target) ?? 0) + 1);
+    }
+  }
+  const byKind = /* @__PURE__ */ new Map();
+  for (const n of fileNodes) {
+    const list = byKind.get(n.kind) ?? [];
+    list.push(n);
+    byKind.set(n.kind, list);
+  }
+  for (const list of byKind.values()) {
+    list.sort(
+      (a, b) => (degreeMap.get(b.id) ?? 0) - (degreeMap.get(a.id) ?? 0)
+    );
+  }
+  const kindOrder = [
+    "class",
+    "interface",
+    "type",
+    "function",
+    "method",
+    "variable",
+    "import",
+    "module",
+    "file",
+    "decision",
+    "pattern",
+    "mistake",
+    "concept"
+  ];
+  const lines = [];
+  lines.push(`[engram] Structural summary for ${relativeFilePath}`);
+  lines.push(
+    `Nodes: ${fileNodes.length} | avg extraction confidence: ${avgConfidence.toFixed(2)}`
+  );
+  lines.push("");
+  for (const kind of kindOrder) {
+    const group = byKind.get(kind);
+    if (!group || group.length === 0) continue;
+    for (const n of group) {
+      const loc = n.sourceLocation ?? "";
+      lines.push(`NODE ${n.label} [${n.kind}] ${loc}`.trim());
+    }
+  }
+  const relevantEdges = allEdges.filter(
+    (e) => fileNodeIds.has(e.source) || fileNodeIds.has(e.target)
+  ).slice(0, 10);
+  if (relevantEdges.length > 0) {
+    lines.push("");
+    lines.push("Key relationships:");
+    for (const e of relevantEdges) {
+      const src = allNodes.find((n) => n.id === e.source);
+      const tgt = allNodes.find((n) => n.id === e.target);
+      if (src && tgt) {
+        lines.push(`EDGE ${src.label} --${e.relation}--> ${tgt.label}`);
+      }
+    }
+  }
+  lines.push("");
+  lines.push(
+    "Note: engram replaced a full-file read with this structural view to save tokens. If you need specific lines, Read this file again with explicit offset/limit parameters \u2014 engram passes partial reads through unchanged."
+  );
+  let text = lines.join("\n");
+  const charBudget = tokenBudget * CHARS_PER_TOKEN;
+  if (text.length > charBudget) {
+    text = sliceGraphemeSafe(text, charBudget) + "\n... (truncated to fit summary budget)";
+  }
+  return {
+    text,
+    nodeCount: fileNodes.length,
+    codeNodeCount,
+    avgConfidence,
+    estimatedTokens: Math.ceil(text.length / CHARS_PER_TOKEN)
+  };
+}
 
 // src/miners/ast-miner.ts
 import { readFileSync as readFileSync2, readdirSync, realpathSync } from "fs";
@@ -1459,6 +1590,76 @@ async function stats(projectRoot) {
     store.close();
   }
 }
+var FILE_CONTEXT_COVERAGE_CEILING = 3;
+async function getFileContext(projectRoot, absFilePath) {
+  const empty = {
+    found: false,
+    confidence: 0,
+    summary: "",
+    nodeCount: 0,
+    codeNodeCount: 0,
+    avgNodeConfidence: 0,
+    graphMtimeMs: 0,
+    fileMtimeMs: null,
+    isStale: false
+  };
+  try {
+    const root = resolve2(projectRoot);
+    const abs = resolve2(absFilePath);
+    const relPath = relative2(root, abs);
+    if (relPath.startsWith("..") || relPath === "") {
+      return empty;
+    }
+    const dbPath = getDbPath(root);
+    let graphMtimeMs = 0;
+    try {
+      graphMtimeMs = statSync2(dbPath).mtimeMs;
+    } catch {
+      return empty;
+    }
+    let fileMtimeMs = null;
+    try {
+      fileMtimeMs = statSync2(abs).mtimeMs;
+    } catch {
+      fileMtimeMs = null;
+    }
+    const isStale = fileMtimeMs !== null && fileMtimeMs > graphMtimeMs;
+    const store = await getStore(root);
+    try {
+      const summary = renderFileStructure(store, relPath);
+      if (summary.codeNodeCount === 0) {
+        return {
+          ...empty,
+          nodeCount: summary.nodeCount,
+          codeNodeCount: 0,
+          graphMtimeMs,
+          fileMtimeMs,
+          isStale
+        };
+      }
+      const coverageScore = Math.min(
+        summary.codeNodeCount / FILE_CONTEXT_COVERAGE_CEILING,
+        1
+      );
+      const confidence = coverageScore * summary.avgConfidence;
+      return {
+        found: true,
+        confidence,
+        summary: summary.text,
+        nodeCount: summary.nodeCount,
+        codeNodeCount: summary.codeNodeCount,
+        avgNodeConfidence: summary.avgConfidence,
+        graphMtimeMs,
+        fileMtimeMs,
+        isStale
+      };
+    } finally {
+      store.close();
+    }
+  } catch {
+    return empty;
+  }
+}
 async function learn(projectRoot, text, sourceLabel = "manual") {
   const { nodes, edges } = learnFromSession(text, sourceLabel);
   if (nodes.length === 0 && edges.length === 0) return { nodesAdded: 0 };
@@ -1474,6 +1675,10 @@ async function mistakes(projectRoot, options = {}) {
   const store = await getStore(projectRoot);
   try {
     let items = store.getAllNodes().filter((n) => n.kind === "mistake");
+    if (options.sourceFile !== void 0) {
+      const target = options.sourceFile;
+      items = items.filter((m) => m.sourceFile === target);
+    }
     if (options.sinceDays !== void 0) {
       const cutoff = Date.now() - options.sinceDays * 24 * 60 * 60 * 1e3;
       items = items.filter((m) => m.lastVerified >= cutoff);
@@ -1572,6 +1777,7 @@ export {
   path,
   godNodes,
   stats,
+  getFileContext,
   learn,
   mistakes,
   benchmark