pi-mega-compact 0.6.2 → 0.6.4

package/README.md

@@ -6,28 +6,39 @@ sessions into a **local SQLite store** and offers **deduped inline recall** —
 running **locally inside the extension**, with **no remote MCP server** and
 **zero network calls at runtime** (PREVENT-PI-004).
 
-> **Current version:** `v0.6.1` — storage backend is **`node:sqlite`**
+> **Current version:** `v0.6.2` — storage backend is **`node:sqlite`**
 > (`DatabaseSync`, a Node ≥22.13 built-in), replacing the old `better-sqlite3`
 > native addon and the per-session gzipped JSON checkpoint files. **Zero native
 > build step, fully local, zero network at runtime.** Legacy
 > `.checkpoints.json.gz` snapshots are retained as disaster-recovery fallbacks
-> and auto-imported on first run. Cross-repo recall, durable memory, and a
-> localhost dashboard round out the continuity story.
+> and auto-imported on first run. The S24 line ties auto-compact, the tier
+> label, trim depth, and durable-memory review to one **unified pressure
+> signal**, adds a **cross-repo memory-RAG index**, and relieves context
+> **during team runs** (not just at the end).
 
 ---
 
 ## What this is (the 30-second version)
 
-pi's context window is finite. When a session gets long, pi-mega-compact:
-
-1. **Watches** context usage and, past a threshold, **compacts** the older part of
-   the conversation into a short structured summary + key facts ("a checkpoint").
-2. **Stores** each checkpoint in a **local vector database** (SQLite) with an
-   embedding, so similar regions can be found later — and so **duplicate
-   work is never stored twice**.
-3. **Recalls** the right checkpoints automatically when you resume a session or
-   invoke a recall command, re-injecting only what's relevant (deduped against
-   what's already in view).
+pi's context window is finite. When a session gets long — especially a team run
+with sub-agents — pi-mega-compact keeps it going without overflowing:
+
+1. **Watches one signal.** A single live `pressure = currentTokens / thresholdTokens`
+   drives everything — the tier label, how aggressively the live trim drops
+   context, and how often durable memory is reviewed. As context fills, the whole
+   system reacts together; as it's relieved, it backs off.
+2. **Compacts in two layers.** On every LLM call it returns a **live, compacted
+   view** (the model sees a summary + recent anchor, non-destructively). And it
+   persists a durable **checkpoint** — and, at each agent settle during a team
+   run, fires pi's **native durable trim** so the on-disk transcript is actually
+   truncated (context relieves mid-run, and resume reloads the trimmed transcript
+   instead of a 150k window).
+3. **Stores** each checkpoint in a **local vector database** (SQLite) with an
+   embedding, so similar regions are found later and **duplicate work is never
+   stored twice**.
+4. **Recalls** the right context automatically — same-repo checkpoints on resume,
+   plus **cross-repo memory-RAG**: decisions you saved in one repo are inlined as
+   context when you start a session in another.
 
 Everything lives on **your disk**. No telemetry, no API, no MCP server, no cloud.
 The only optional network surface is a **user-triggered localhost dashboard** you
@@ -35,11 +46,13 @@ open yourself.
 
 ### Why "mega"?
 
-The compaction pipeline is a **Trident** — three deterministic stages that run
-over your conversation before anything is persisted. The checkpoint it produces
-is small (a summary + key decisions + next steps + files touched), so the same
-session that would otherwise overflow its window keeps going on a fraction of the
-tokens.
+The compaction pipeline is a **Trident** — three deterministic stages
+(supersede → collapse → cluster) that run over your conversation before anything
+is persisted. The checkpoint it produces is small (a summary + key decisions +
+next steps + files touched), so the same session that would otherwise overflow
+its window keeps going on a fraction of the tokens. On top of the Trident, a
+single pressure signal orchestrates the live trim, the durable trim, and memory
+review as one coherent system rather than four independent triggers.
 
 ---
 
@@ -52,9 +65,12 @@ Layer 3  Cluster (vectorize)  local vector index → semantic dedup + recall
 Layer 2  Collapse (summarize)  summarizeMessages() heuristic + agent summary on /mega-compact
 Layer 1  Supersede (prune)     drop obsolete file-reads / superseded turns (zero cost)
 ─────────────────────────────────────────────────────────────────────────
-Trigger   context/turn_end → % gate → auto_compact_check → fire
+Trigger   context → token fast-gate → autoCompactCheck → live trim (per call)
+Durable    agent_end (idle + over threshold) → ctx.compact() → session_before_compact
+          supplies the summary; pi truncates the transcript (relieves context)
+Live      context handler returns { messages:[summary, …recent] } — model sees a
+          compacted window every LLM call; the on-disk transcript is untouched
 Marker    insert compact-marker; dedupe so repeated triggers cost ~0 tokens
-Cancel    session_before_compact → { cancel:true } once persisted (no double-compact)
 ```
 
 **One store, three ways to read it back — one dedup engine:**
@@ -168,7 +184,7 @@ building.
 pi-mega-compact uses a dual local backend — **zero network, no native build step**:
 
 - **`node:sqlite`** (`DatabaseSync`, Node ≥22.13 built-in) — the synchronous source of truth for checkpoints, session state, and the dedup index. No dependency, no install script, survives pi's `install-scripts` block.
-- **PGlite + `@electric-sql/pglite-pgvector`** (WASM Postgres + HNSW `vector_cosine_ops`) — an optional, best-effort async vector index for **cross-repo recall** at `~/.pi/mega-compact-vector`. The sync store stays authoritative; the index degrades to the sync per-session scan on any failure.
+- **PGlite + `@electric-sql/pglite-pgvector`** (WASM Postgres + HNSW `vector_cosine_ops`) — an optional, best-effort async vector index for **cross-repo recall** at `~/.pi/mega-compact-vector`. It holds both checkpoint embeddings and durable-memory embeddings, so decisions saved in one repo are findable from another. The sync store stays authoritative; the index degrades to the sync per-session scan on any failure.
 
 Kill-switch: `MEGACOMPACT_PGLITE_DISABLED=1` fully disables the PGlite index (falls back to sync scan). Requires Node ≥22.13 (`engines.node`).
 
@@ -178,7 +194,7 @@ On resume, recall augments from other repos' checkpoints when this repo's store
 
 ### Memory
 
-pi-mega-compact auto-reviews the conversation every 10 turns and writes durable `decision`/`fact`/`preference` memories to SQLite (local, hallucination-guarded). Relevant memories are injected as RAG context on recall (capped, deduped). Manual: `/mega-memory save|list|forget`.
+pi-mega-compact auto-reviews the conversation every 10 turns (the cadence shortens as pressure climbs) and writes durable `decision`/`fact`/`preference` memories to SQLite (local, hallucination-guarded). Relevant memories are injected as RAG context on recall (capped, deduped). **Cross-repo memory-RAG (S24):** every memory write is mirrored into the PGlite/HNSW index, so when same-repo recall is thin the system augments with the nearest memories from *other* repos (stricter `MEGACOMPACT_CROSSREPO_COSINE` floor, deduped against what's already in view). Manual: `/mega-memory save|list|forget` (or `/m`).
 
 ### Uninstall
 
@@ -352,7 +368,7 @@ The extension entry adapts between the engine and pi's runtime types.
 
 ```bash
 npm run build      # tsc
-npm test           # build + node --test on dist/**/*.test.js (346 tests)
+npm test           # build + node --test on dist/**/*.test.js (353 tests)
 npm run lint       # tsc --noEmit + guardrails-scan
 npm run guardrails # regression_check + guardrails-scan
 ```

package/dist/extensions/conflict-scan.js

@@ -12,10 +12,18 @@
  *
  * Pi-agnostic: reads package.json + greps source. No pi runtime types, so it is
  * unit-testable against a fixture node_modules tree.
+ *
+ * SCAN-SCOPE FIX (S24 follow-up): the original scanner only walked the npm
+ * `node_modules` tree, so user-level extensions installed outside npm (e.g.
+ * `pi-hermes-memory`, a data-only `MEMORY.md` + `sessions.db` memory store)
+ * were never inspected — that gap let the 5000-char file-buffer error slip
+ * through undetected. We now also scan the user-level extension dir and detect
+ * memory stores that ship with no package.json / source to grep.
  */
 import { existsSync, readFileSync, readdirSync, statSync } from "node:fs";
 import { join, dirname } from "node:path";
 import { fileURLToPath } from "node:url";
+import { homedir } from "node:os";
 // Marker sets. A package is flagged when its source matches a marker in a
 // category. File-grep (not AST) keeps this dependency-free and fast.
 const MARKERS = {
@@ -38,10 +46,7 @@ const MARKERS = {
         "memoryTool",
     ],
     // Tool-output shaping (compact/summarize tool results) — overlap, not a rival.
-    toolOutput: [
-        "tool_result",
-        "ToolResult",
-    ],
+    toolOutput: ["tool_result", "ToolResult"],
 };
 /** Resolve the node_modules dir that contains this package (or env override). */
 export function resolveExtensionRoot(selfDir = dirname(fileURLToPath(import.meta.url))) {
@@ -62,6 +67,41 @@ export function resolveExtensionRoot(selfDir = dirname(fileURLToPath(import.meta
     }
     return null;
 }
+/**
+ * Resolve every directory that may hold pi extensions to scan.
+ *
+ * - `MEGACOMPACT_EXT_SCAN_DIR` (if set) replaces the whole list — a single
+ *   fixture/override root for tests or custom layouts.
+ * - Otherwise: the node_modules that holds this package (classic npm layout) AND
+ *   the user-level extension dir (`~/.pi/agent`), which is where extensions
+ *   installed outside npm actually live. The original scanner only walked
+ *   node_modules, so user-level memory extensions were never flagged — that is
+ *   the gap that let the 5000-char `MEMORY.md` buffer error slip through.
+ */
+export function collectScanRoots() {
+    const override = process.env.MEGACOMPACT_EXT_SCAN_DIR;
+    if (override && override.trim() !== "")
+        return [override];
+    const roots = [];
+    const nm = resolveExtensionRoot();
+    if (nm && existsSync(nm))
+        roots.push(nm);
+    const userDir = process.env.MEGACOMPACT_EXT_USER_DIR?.trim() ||
+        join(homedir(), ".pi", "agent");
+    if (existsSync(userDir))
+        roots.push(userDir);
+    return roots;
+}
+/**
+ * True when a directory is a pi memory-store container rather than a normal code
+ * extension. `pi-hermes-memory` ships as exactly this: no package.json, no
+ * source — just `MEMORY.md` + `sessions.db`. The marker-grep path misses it,
+ * so we also detect the on-disk memory signature.
+ */
+function isMemoryStoreDir(pkgDir) {
+    return (existsSync(join(pkgDir, "sessions.db")) ||
+        existsSync(join(pkgDir, "MEMORY.md")));
+}
 /** Recursively collect source-ish files under a package, capped to avoid scans. */
 function collectFiles(root, max = 400) {
     const out = [];
@@ -131,70 +171,103 @@ function matchMarkers(pkgDir, keys) {
  * @param selfName package name to skip (defaults to this package's name).
  */
 export function detectConflicts(selfName = "pi-mega-compact") {
-    const root = resolveExtensionRoot();
+    const roots = collectScanRoots();
     const scanned = [];
     const conflicts = [];
-    if (!root || !existsSync(root))
-        return { scanned, conflicts };
-    let entries;
-    try {
-        entries = readdirSync(root);
-    }
-    catch {
-        return { scanned, conflicts };
-    }
-    for (const name of entries) {
-        const pkgDir = join(root, name);
-        if (!statSync(pkgDir).isDirectory())
-            continue;
-        const pkgJson = join(pkgDir, "package.json");
-        if (!existsSync(pkgJson))
+    for (const root of roots) {
+        if (!existsSync(root))
             continue;
-        let pkg;
+        let entries;
         try {
-            pkg = JSON.parse(readFileSync(pkgJson, "utf-8"));
+            entries = readdirSync(root);
         }
         catch {
             continue;
         }
-        const pkgName = pkg.name ?? name;
-        if (pkgName === selfName)
-            continue;
-        // Only consider packages that declare pi extensions.
-        if (!pkg.pi || !Array.isArray(pkg.pi.extensions) || pkg.pi.extensions.length === 0)
-            continue;
-        scanned.push(pkgName);
-        const memHits = matchMarkers(pkgDir, MARKERS.memory);
-        const compHits = matchMarkers(pkgDir, MARKERS.compaction);
-        const toolHits = matchMarkers(pkgDir, MARKERS.toolOutput);
-        if (compHits.length > 0) {
-            conflicts.push({
-                package: pkgName,
-                severity: "high",
-                kind: "compaction",
-                evidence: compHits,
-                recommendation: "Disabling recommended — competes with pi-mega-compact's conversation compaction.",
-            });
-            continue; // compaction is the dominant conflict; don't double-flag.
-        }
-        if (memHits.length > 0) {
-            conflicts.push({
-                package: pkgName,
-                severity: "high",
-                kind: "memory",
-                evidence: memHits,
-                recommendation: "pi-mega-compact now owns save-to-memory (/mega-memory, its own SQLite). Disable this to avoid duplicate memory stores.",
-            });
-            continue;
-        }
-        if (toolHits.length > 0) {
-            conflicts.push({
-                package: pkgName,
-                severity: "info",
-                kind: "tool-output",
-                evidence: toolHits,
-                recommendation: "Shapes tool output (summarize/compact tool results). Generally compatible; no action needed.",
-            });
+        for (const name of entries) {
+            const pkgDir = join(root, name);
+            let st;
+            try {
+                st = statSync(pkgDir);
+            }
+            catch {
+                continue;
+            }
+            if (!st.isDirectory())
+                continue;
+            // A candidate is either a real code extension (declares pi.extensions) or a
+            // data-only memory store (MEMORY.md / sessions.db at its root).
+            const pkgJson = join(pkgDir, "package.json");
+            let pkg = null;
+            if (existsSync(pkgJson)) {
+                try {
+                    pkg = JSON.parse(readFileSync(pkgJson, "utf-8"));
+                }
+                catch {
+                    pkg = null;
+                }
+            }
+            const isCodeExt = !!pkg &&
+                !!pkg.pi &&
+                Array.isArray(pkg.pi.extensions) &&
+                pkg.pi.extensions.length > 0;
+            const isMemoryStore = isMemoryStoreDir(pkgDir);
+            if (!isCodeExt && !isMemoryStore)
+                continue;
+            const pkgName = pkg?.name ?? name;
+            if (pkgName === selfName)
+                continue;
+            scanned.push(`${pkgName} (${name})`);
+            if (isCodeExt) {
+                const memHits = matchMarkers(pkgDir, MARKERS.memory);
+                const compHits = matchMarkers(pkgDir, MARKERS.compaction);
+                const toolHits = matchMarkers(pkgDir, MARKERS.toolOutput);
+                if (compHits.length > 0) {
+                    conflicts.push({
+                        package: pkgName,
+                        severity: "high",
+                        kind: "compaction",
+                        evidence: compHits,
+                        recommendation: "Disabling recommended — competes with pi-mega-compact's conversation compaction.",
+                    });
+                    continue; // compaction is the dominant conflict; don't double-flag.
+                }
+                if (memHits.length > 0) {
+                    conflicts.push({
+                        package: pkgName,
+                        severity: "high",
+                        kind: "memory",
+                        evidence: memHits,
+                        recommendation: "pi-mega-compact now owns save-to-memory (/mega-memory, its own SQLite). Disable this to avoid duplicate memory stores.",
+                    });
+                    continue;
+                }
+                if (toolHits.length > 0) {
+                    conflicts.push({
+                        package: pkgName,
+                        severity: "info",
+                        kind: "tool-output",
+                        evidence: toolHits,
+                        recommendation: "Shapes tool output (summarize/compact tool results). Generally compatible; no action needed.",
+                    });
+                }
+            }
+            // Data-only memory store: no source to grep, but the on-disk signature
+            // (sessions.db / MEMORY.md) is a conflict with our SQLite memory store.
+            if (isMemoryStore) {
+                const evidence = [];
+                if (existsSync(join(pkgDir, "sessions.db")))
+                    evidence.push("sessions.db");
+                if (existsSync(join(pkgDir, "MEMORY.md")))
+                    evidence.push("MEMORY.md");
+                conflicts.push({
+                    package: pkgName,
+                    severity: "high",
+                    kind: "memory",
+                    evidence,
+                    recommendation: "pi-mega-compact now owns save-to-memory (its own SQLite). This data-only memory store competes with it — disable to avoid a duplicate / capped memory buffer.",
+                });
+            }
         }
     }
     return { scanned, conflicts };

package/dist/extensions/conflict-scan.test.js

@@ -0,0 +1,115 @@
+/**
+ * conflict-scan.test.ts — unit tests for the extension-conflict scanner.
+ *
+ * Fixture trees are written under a temp dir and scanned via
+ * MEGACOMPACT_EXT_SCAN_DIR (which makes collectScanRoots() return that
+ * single root). This covers the S24 follow-up fix:
+ *
+ *   1. node_modules-style code extensions (package.json + pi.extensions) are
+ *      still detected by source-marker grep (regression).
+ *   2. USER-LEVEL extensions installed outside npm (e.g. pi-hermes-memory)
+ *      now get scanned too — previously only `node_modules` was walked, so a
+ *      data-only memory store (MEMORY.md + sessions.db, no package.json)
+ *      was never flagged (the 5000-char file-buffer error slipped through).
+ *   3. The data-only memory-store signature is detected even with no source.
+ *   4. pi-mega-compact (selfName) is always skipped.
+ */
+import { test, after } from "node:test";
+import assert from "node:assert/strict";
+import { mkdtempSync, rmSync, mkdirSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { detectConflicts, collectScanRoots } from "./conflict-scan.js";
+const base = mkdtempSync(join(tmpdir(), "mc-scan-"));
+let n = 0;
+/** Make a fixture root containing one or more fake extensions, return its path. */
+function fixture(build) {
+    const root = join(base, `case-${n++}`);
+    mkdirSync(root, { recursive: true });
+    build(root);
+    return root;
+}
+after(() => {
+    rmSync(base, { recursive: true, force: true });
+});
+test("scans a user-level, data-only memory store (no package.json)", () => {
+    const root = fixture((r) => {
+        const ext = join(r, "pi-hermes-memory");
+        mkdirSync(ext, { recursive: true });
+        // No package.json, no source — just pi's memory-store signature.
+        writeFileSync(join(ext, "MEMORY.md"), "# memory\n");
+        writeFileSync(join(ext, "sessions.db"), "");
+    });
+    process.env.MEGACOMPACT_EXT_SCAN_DIR = root;
+    try {
+        const { conflicts } = detectConflicts();
+        assert.ok(conflicts.length >= 1, "expected a memory conflict");
+        const hit = conflicts.find((c) => c.kind === "memory");
+        assert.ok(hit, "expected a memory-kind conflict");
+        assert.equal(hit.severity, "high");
+        assert.ok(hit.evidence.includes("MEMORY.md") ||
+            hit.evidence.includes("sessions.db"), "evidence should name the on-disk memory signature");
+    }
+    finally {
+        delete process.env.MEGACOMPACT_EXT_SCAN_DIR;
+    }
+});
+test("still detects a code extension by source marker (regression)", () => {
+    const root = fixture((r) => {
+        // A code extension is a DIRECT child of the scan root (mirrors the
+        // node_modules layout: packages live one level under the root).
+        const ext = join(r, "some-memory-ext");
+        mkdirSync(ext, { recursive: true });
+        writeFileSync(join(ext, "package.json"), JSON.stringify({ name: "some-memory-ext", pi: { extensions: ["x.ts"] } }));
+        writeFileSync(join(ext, "index.ts"), "export const MEMORY_TOOL = true;");
+    });
+    process.env.MEGACOMPACT_EXT_SCAN_DIR = root;
+    try {
+        const { conflicts } = detectConflicts();
+        const hit = conflicts.find((c) => c.package === "some-memory-ext");
+        assert.ok(hit, "expected some-memory-ext to be flagged");
+        assert.equal(hit.kind, "memory");
+    }
+    finally {
+        delete process.env.MEGACOMPACT_EXT_SCAN_DIR;
+    }
+});
+test("skips pi-mega-compact (selfName) and non-extension dirs", () => {
+    const root = fixture((r) => {
+        // selfName dir with a memory signature — must be ignored.
+        const me = join(r, "node_modules", "pi-mega-compact");
+        mkdirSync(me, { recursive: true });
+        writeFileSync(join(me, "sessions.db"), "");
+        // unrelated dir with no pi.extensions and no memory signature.
+        mkdirSync(join(r, "node_modules", "totally-fine"), { recursive: true });
+    });
+    process.env.MEGACOMPACT_EXT_SCAN_DIR = root;
+    try {
+        const { scanned, conflicts } = detectConflicts();
+        assert.equal(conflicts.length, 0, "no conflicts expected");
+        assert.ok(!scanned.some((s) => s.includes("pi-mega-compact")), "selfName should not appear in scanned");
+    }
+    finally {
+        delete process.env.MEGACOMPACT_EXT_SCAN_DIR;
+    }
+});
+test("collectScanRoots honors MEGACOMPACT_EXT_SCAN_DIR override", () => {
+    const root = fixture(() => { });
+    process.env.MEGACOMPACT_EXT_SCAN_DIR = root;
+    try {
+        const roots = collectScanRoots();
+        assert.deepEqual(roots, [root], "override replaces the whole root list");
+    }
+    finally {
+        delete process.env.MEGACOMPACT_EXT_SCAN_DIR;
+    }
+});
+test("collectScanRoots falls back to node_modules + user dir when no override", () => {
+    delete process.env.MEGACOMPACT_EXT_SCAN_DIR;
+    delete process.env.MEGACOMPACT_EXT_USER_DIR;
+    // No override set and this test file lives under extensions/, so node_modules
+    // resolution walks up from here; the user dir (~/.pi/agent) may or may
+    // not exist in CI. We only assert the call returns a non-throwing array.
+    const roots = collectScanRoots();
+    assert.ok(Array.isArray(roots), "collectScanRoots must return an array");
+});