pi-mega-compact 0.11.9 → 0.11.11

package/README.md

@@ -6,15 +6,19 @@ A local context compressor for the [pi coding agent](https://github.com/earendil
 
 - **Auto-compaction** — the store watches context pressure and compacts quietly in the background. You'll notice when a long session just stays long while the token gauge rests comfortably far from the ceiling.
 - **Two-layer compaction** — every LLM call sees a live trim of the context window, and every trim is checkpointed to SQLite so a crash or a `/clear` never loses the work.
-- **Semantic dedup, three layers deep** — exact hash → MinHash/LSH → cosine over trigram embeddings. You rarely notice it; that's the point.
-- **RAPTOR memory hierarchy** — decisions you made an hour ago don't scroll off; they get packed up as hierarchical checkpoints and re-inlined the moment your next session asks for them. Multi-level retrieval (leaves + summary clusters) is on by default and tunes itself off the build history.
+- **Semantic dedup, three layers deep** — exact hash (L0) -> MinHash/LSH (L1) -> cosine over trigram embeddings (L2). The dedup audit log records per-tier decisions with similarity scores for tuning.
+- **RAPTOR memory hierarchy** — decisions you made an hour ago don't scroll off; they get packed up as hierarchical checkpoints and re-inlined the moment your next session asks for them. Multi-level retrieval (leaves + summary clusters) is on by default. Since v0.11.10, RAPTOR tree updates are incremental (no full rebuild) — enabled by default.
 - **Per-turn tracking + rewind.** Every turn, checkpoint, and recall hit lands as a row in an isolated `turns.db` — `turns`, `turn_recall`, `conversation_forks`. The dashboard **Turns tab** shows turn-by-turn memory: context pressure, the compact epoch that superseded each turn, and the exact checkpoints recalled into it. A **fork** action branches a conversation at any turn (carrying its recall set); a **rewind** action queues an intent the host consumes at the next `before_agent_start`. The `TurnStore` is contract-first (capability-gated reader/writer/admin views) so the same spine backs the dashboard, the TUI, or an API gateway.
-- **Cross-repo recall** — doors I close in one repo don't reopen when I move to another. A decision stored while hacking repo A is a recall hit the next time I'm in repo B.
-- **Durable memory** — every ten turns the store auto-reviews and safe-keeps decisions, facts, and preferences as first-class RAG memories, so long-running projects remember what mattered.
+- **Cross-repo recall** — doors you close in one repo don't reopen when you move to another. A decision stored while hacking repo A is a recall hit the next time you're in repo B.
+- **Durable memory** — on a cadence the store auto-reviews and safe-keeps decisions, facts, and preferences as first-class RAG memories, so long-running projects remember what mattered.
+- **Prompt-cache optimization** — message separation + cache striping (default OFF, opt-in). Targets 82-90% cache hit rate by structuring context around provider cache boundaries. Enables with `MEGACOMPACT_MESSAGE_SEPARATION` and `MEGACOMPACT_CACHE_STRIPING`.
+- **RAG suite** — query reformulation (TF-IDF + RRF), tiered routing (L0 cache -> L1 FTS5 -> L2 PGlite), and recall-quality metrics (CRAG). All default OFF, opt-in via flags `MEGACOMPACT_QUERY_REFORMULATION`, `MEGACOMPACT_TIERED_ROUTER`, `MEGACOMPACT_RECALL_METRICS`.
+- **Stacked memory graph** — the dashboard shows memory composition over time from 3 content sources (turns, durable memories, wiki) with a 9-gate validation system and a graph-health indicator. Per-model provider cache breakdown in the Cache tab.
+- **Debug bundle** — Maintenance tab in the dashboard has a **Gather Debug Logs** button that collects events, config, and store state into a shareable archive for bug reports.
 - **Fully local** — node:sqlite + trigram embeddings by default. Bring your own localhost embedder (ONNX, Ollama, TEI) for better semantic matches. Zero calls off your machine except the optional, localhost-only dashboard.
 - **Team-run aware** — fine-grained durable trim fires at agent settle during sub-agent runs, so long multi-agent work doesn't just collapse at the end.
-- **Multi-pi dashboard** — one dashboard tab per active pi process with the context stack, per-repo stats, and a live SSE feed across all of them. The React SPA now has lazy-loaded tabs: Overview, Repos, Events, Config, Metrics, Cache, Game, Achievements, Sessions, Topics (wiki), and Turns (per-turn memory + recall + rewind).
-- **Auto-categorizing wiki.** Every ten compactions, the store clusters your real memory embeddings (k-means) and labels each cluster with its most discriminative terms (TF-IDF) — no LLM, no Ollama, fully local. The dashboard **Wiki tab** browses topics, searches by label or term, and drills down into the member memories of each cluster.
+- **Multi-pi dashboard** — one dashboard tab per active pi process with the context stack, per-repo stats, and a live SSE feed across all of them. The React SPA has lazy-loaded tabs: Overview, Repos, Events, Config, Metrics, Cache, Game, Achievements, Sessions, Topics (wiki), Turns (per-turn memory + recall + rewind), and Maintenance (debug bundle).
+- **Auto-categorizing wiki.** Every 3 compactions (seeds from turns before that), the store clusters your real memory embeddings (k-means) and labels each cluster with its most discriminative terms (TF-IDF) — no LLM, no Ollama, fully local. The dashboard **Wiki tab** browses topics, searches by label or term, and drills down into the member memories of each cluster.
 
 ## Install
 
@@ -22,7 +26,7 @@ A local context compressor for the [pi coding agent](https://github.com/earendil
 pi install npm:pi-mega-compact
 ```
 
-That's it. `pi update --extensions` pulls updates going forward.
+That's it. **`pi update --extensions`** pulls updates going forward. npm is the only distribution path — never use `.tgz` tarballs or symlinks for shipping.
 
 <details>
 <summary>From source (development)</summary>
@@ -48,6 +52,7 @@ Key commands:
 - `/mega-recall [query]` — semantic search the store, `--cross-repo` for all repos
 - `/mega-memory save|list|search|forget` — manage durable memories
 - `/mega-dashboard` — start the localhost dashboard
+- `/mega-setup` — embedding wizard: detects Ollama/llama.cpp, suggests upgrades when recall quality is low
 
 Full command reference: [`docs/COMMANDS.md`](docs/COMMANDS.md)
 
@@ -62,7 +67,15 @@ Set env vars before starting pi. Defaults are in `src/config/dedup.ts`.
 | `MEGACOMPACT_DEDUP_SIM` | `0.90` | Cosine threshold for near-dup collapse |
 | `MEGACOMPACT_CROSSREPO_ENABLED` | `true` | Cross-repo recall on resume |
 | `MEGACOMPACT_EMBEDDING_URL` | _(unset)_ | BYO localhost embedder endpoint |
-| `MEGACOMPACT_TUI_WIDGET` | `true` | Render the above-editor panel. Set to `0` to suppress it — useful when pi runs inside an editor terminal (e.g. Neovim `:terminal`) where you drive scrollback yourself and the panel's repaints fight it. Compaction is unaffected. |
+| `MEGACOMPACT_TUI_WIDGET` | `true` | Render the above-editor panel |
+| `MEGACOMPACT_MESSAGE_SEPARATION` | `false` | Opt-in: separate messages at provider cache boundaries |
+| `MEGACOMPACT_CACHE_STRIPING` | `false` | Opt-in: stripe cache stripes across conversation |
+| `MEGACOMPACT_QUERY_REFORMULATION` | `false` | Opt-in: TF-IDF + RRF query reformulation |
+| `MEGACOMPACT_TIERED_ROUTER` | `false` | Opt-in: L0->L1->L2 recall routing |
+| `MEGACOMPACT_RECALL_METRICS` | `false` | Opt-in: CRAG recall quality metrics |
+| `MEGACOMPACT_MEMORY_GRAPH_SEED_TURNS` | `true` | Seed memory graph from turns (default ON) |
+| `MEGACOMPACT_WIKI_SEED_FROM_TURNS` | `true` | Seed wiki from turns (default ON) |
+| `MEGACOMPACT_RAPTOR_INCREMENTAL` | `true` | Incremental RAPTOR updates (default ON) |
 
 Full config reference: [`docs/CONFIGURATION.md`](docs/CONFIGURATION.md)
 
@@ -70,11 +83,11 @@ Full config reference: [`docs/CONFIGURATION.md`](docs/CONFIGURATION.md)
 
 ```
 extensions/    Pi entry points (mega-compact, mega-trim, dashboard)
-src/engine.ts  Trident pipeline (supersede → collapse → cluster)
+src/engine.ts  Trident pipeline (supersede -> collapse -> cluster)
 src/vectorStore.ts   Local vector DB (add/search/dedupe)
 src/compact.ts       Summarize / merge / auto-compact
 src/memory.ts        Durable memories + auto-review
-src/store/sqlite.ts   node:sqlite store (Node ≥22.13)
+src/store/sqlite.ts   node:sqlite store (Node >=22.13)
 src/store/vectorIndex.ts  PGlite/HNSW cross-repo index
 ```
 
@@ -84,12 +97,22 @@ Detailed architecture: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md)
 
 ```bash
 npm run build     # TypeScript compile
-npm test          # Build + 769 tests
+npm test          # Build + 1112 tests
 npm run lint      # Type check + guardrails scan
 ```
 
 Testing guide: [`TESTER_GUIDE.md`](TESTER_GUIDE.md)
 
+## Tester requests
+
+These areas benefit from real-world usage data. The automated test suite covers correctness, but tuning requires diverse sessions.
+
+- **Cosine threshold validation** — the 0.90 dedup threshold may need adjustment per content type (try 0.93 for code, 0.87 for prose). Run with `MEGACOMPACT_DEDUP_SIM` set to different values and report false positives via the debug bundle (Maintenance -> Gather Debug Logs). The dedup audit log (events.log) records `similarityScore` + `matchedId` per decision.
+- **Cross-repo recall quality** — enable cross-repo recall and report the relevance-vs-noise ratio from real multi-repo sessions. The Turns tab shows recall hits per turn.
+- **Dedup layer audit** — is MinHash/LSH (L1) catching enough over exact-hash (L0) + cosine (L2) to justify the complexity? If L1 catches <5% additional in your sessions, we may simplify to 2-layer. The dedup audit log has per-tier decisions.
+- **Compaction death-spiral** — v0.11.9 fixed an "Already compacted" loop that made sessions unrecoverable (critical-over escape hatch forces a trim + durable compact at >=90% context). If you see this recur, report it immediately with the debug bundle.
+- **Share debug bundles** — when reporting any issue, first gather from Maintenance -> Gather Debug Logs and include the archive.
+
 ## Troubleshooting
 
 ### npm lifecycle scripts disabled
@@ -107,7 +130,7 @@ npm install --install-strategy=linked
 
 ### node:sqlite not found / experimental flag required
 
-pi-mega-compact requires Node ≥22.13 for the built-in `node:sqlite` module. If you're on an older version or your Node build doesn't include it by default:
+pi-mega-compact requires Node >=22.13 for the built-in `node:sqlite` module. If you're on an older version or your Node build doesn't include it by default:
 
 ```bash
 export NODE_OPTIONS="--experimental-sqlite"
@@ -120,7 +143,7 @@ Add the export to your shell profile (`.bashrc`, `.zshrc`, etc.) to make it perm
 
 BSD 3-Clause
 
-## ☕ Support
+## Support
 
 If this project helped you, consider buying me a coffee:
 

package/dist/extensions/mega-events/context-handler.js

@@ -240,7 +240,7 @@ export function registerContextHandler(pi, runtime, config) {
             if (runtime.trimCache &&
                 runtime.trimCache.checkpointId === runtime.rt.lastCheckpointId &&
                 runtime.trimCache.cut <= messages.length) {
-                const recent = messages.slice(runtime.trimCache.cut); // guardrails-allow PREVENT-PI-002
+                const recent = messages.slice(runtime.trimCache.cut); // guardrails-allow PREVENT-PI-002: cached `cut` was sanitized by computeLiveTrimCut (src/boundary.ts); replayed verbatim, transcript only grows within an epoch.
                 runtime.diagLiveTrimFires++;
                 runtime.diagLiveTrimReplays++;
                 runtime.snapshot(ctx);

package/dist/src/config/dedup.js

@@ -64,6 +64,7 @@ export function loadDedupConfig() {
         RAPTOR_MULTILEVEL_ENABLED: envBool("MEGACOMPACT_RAPTOR_MULTILEVEL", true),
         RAPTOR_LEVEL_WEIGHTS: envNumArray("MEGACOMPACT_RAPTOR_LEVEL_WEIGHTS", [1.0, 0.9, 0.8, 0.7, 0.5]),
         RAPTOR_LEAF_EXPANSION: envBool("MEGACOMPACT_RAPTOR_LEAF_EXPANSION", true),
+        RAPTOR_INCREMENTAL: envBool("MEGACOMPACT_RAPTOR_INCREMENTAL", true),
         RAPTOR_MAX_LEAF_EXPANSION: envNum("MEGACOMPACT_RAPTOR_MAX_LEAF_EXP", 10),
         RAPTOR_FRESHNESS_HOURS: envNum("MEGACOMPACT_RAPTOR_FRESHNESS_HOURS", 4),
         RAPTOR_INJECT_SUMMARIES: envBool("MEGACOMPACT_RAPTOR_INJECT_SUMMARIES", true),

package/dist/src/dedup/raptor/incremental.js

@@ -0,0 +1,276 @@
+/**
+ * incremental.ts — incremental RAPTOR tree update (Sprint 26, #7).
+ *
+ * Instead of rebuilding the full tree (~2s per compaction), insert only the new
+ * checkpoints since the last build and recompute only the affected cluster
+ * assignments up the tree. Falls back to a full rebuild when the tree is
+ * missing, corrupted, or >50% of nodes are new.
+ *
+ * Non-fatal: every failure logs and returns null (caller falls back to full
+ * rebuild). PREVENT-PI-004: zero network; all operations are local SQLite + CPU.
+ */
+import { cosineSimilarity } from "../../embedder.js";
+import { defaultEmbedder as getDefaultEmbedder } from "../../embedder.js";
+import { buildRaptorTree } from "./tree.js";
+import { meanVector } from "./kmeans.js";
+import { summarizeCluster } from "./summarizer.js";
+import { saveRaptorTree, listRaptorNodes, } from "../../store/sqlite.js";
+/** When >FRESH_THRESHOLD of existing leaves are new, a full rebuild is cheaper. */
+const FRESH_THRESHOLD = 0.5;
+/**
+ * Incrementally update a persisted RAPTOR tree with new leaves.
+ *
+ * 1. Reads the existing tree from the store.
+ * 2. Filters `newLeaves` to those NOT already covered by the tree.
+ * 3. If no new leaves → early-return (existing tree).
+ * 4. If no existing tree or >50% new → full rebuild.
+ * 5. For each new leaf, finds the best-matching Level-0 cluster node,
+ *    reassigns (inserts into its children), updates that node's summary
+ *    and centroid embedding.
+ * 6. Propagates changes up through parent levels to the root.
+ * 7. Persists the updated tree (full delete + reinsert in a tx).
+ *
+ * Never throws. Returns the updated tree, or null on failure/fallback.
+ */
+export function incrementRaptorTree(existingLeaves, newLeaves, opts) {
+    const logger = opts.logger;
+    const builtAt = opts.builtAt ?? Date.now();
+    try {
+        // 1. Read existing tree.
+        const storedNodes = listRaptorNodes(opts.sessionId, opts.stateDir);
+        if (storedNodes.length === 0) {
+            logger?.info("raptor_incremental_no_tree", { sessionId: opts.sessionId });
+            return null; // no existing tree — caller falls back to full buildRaptorTree
+        }
+        // 2. Find new leaf ids not already in the tree.
+        const coveredLeafIds = new Set();
+        for (const n of storedNodes) {
+            for (const cid of n.children)
+                coveredLeafIds.add(cid);
+        }
+        const reallyNewLeaves = newLeaves.filter((l) => !coveredLeafIds.has(l.id));
+        const totalCovered = coveredLeafIds.size;
+        if (reallyNewLeaves.length === 0) {
+            logger?.info("raptor_incremental_no_new", { sessionId: opts.sessionId });
+            return rehydrateFromStored(storedNodes);
+        }
+        // 4. Fallback guard: >50% new → full rebuild.
+        if (totalCovered > 0 &&
+            reallyNewLeaves.length / totalCovered > FRESH_THRESHOLD) {
+            logger?.info("raptor_incremental_fallback_ratio", {
+                sessionId: opts.sessionId,
+                coveredNodes: totalCovered,
+                newLeaves: reallyNewLeaves.length,
+                threshold: FRESH_THRESHOLD,
+            });
+            return fullRebuild(existingLeaves, reallyNewLeaves, opts, logger);
+        }
+        // 5. Build the in-memory tree from stored nodes.
+        const tree = rehydrateFromStored(storedNodes);
+        if (!tree || tree.nodes.size === 0) {
+            logger?.info("raptor_incremental_rehydrate_failed", {
+                sessionId: opts.sessionId,
+            });
+            return fullRebuild(existingLeaves, reallyNewLeaves, opts, logger);
+        }
+        // Group stored nodes by level.
+        const levelGroups = new Map();
+        for (const n of storedNodes) {
+            const g = levelGroups.get(n.level);
+            if (g)
+                g.push(n);
+            else
+                levelGroups.set(n.level, [n]);
+        }
+        // Level-0 covers leaf ids directly.
+        const level0Nodes = levelGroups.get(0) ??
+            storedNodes.filter((n) => {
+                // A node is level 0 if its level is the minimum.
+                const minLevel = Math.min(...storedNodes.map((x) => x.level));
+                return n.level === minLevel;
+            });
+        if (level0Nodes.length === 0) {
+            // Single-root tree with no hierarchy: cannot increment, fall back.
+            return fullRebuild(existingLeaves, reallyNewLeaves, opts, logger);
+        }
+        // For each new leaf, assign to the nearest Level-0 cluster node.
+        const leafEmbeddings = new Map();
+        for (const l of reallyNewLeaves)
+            leafEmbeddings.set(l.id, l.embedding);
+        // Track which Level-0 nodes changed (by their stored id).
+        const affectedNodeIds = new Set();
+        for (const leaf of reallyNewLeaves) {
+            const emb = leafEmbeddings.get(leaf.id);
+            if (!emb || emb.length === 0)
+                continue;
+            // Find best matching Level-0 node.
+            let bestNode = null;
+            let bestSim = -Infinity;
+            for (const cn of level0Nodes) {
+                const sim = cosineSimilarity(emb, cn.embedding);
+                if (sim > bestSim) {
+                    bestSim = sim;
+                    bestNode = cn;
+                }
+            }
+            if (!bestNode)
+                continue;
+            // Add the leaf id to this node's children.
+            const updated = tree.nodes.get(bestNode.id);
+            if (updated) {
+                if (!updated.children.includes(leaf.id)) {
+                    updated.children.push(leaf.id);
+                }
+                affectedNodeIds.add(bestNode.id);
+            }
+        }
+        if (affectedNodeIds.size === 0) {
+            // No matches found — fall back to full rebuild.
+            logger?.info("raptor_incremental_no_matches", {
+                sessionId: opts.sessionId,
+            });
+            return fullRebuild(existingLeaves, reallyNewLeaves, opts, logger);
+        }
+        // 6. Recompute embeddings and summaries for affected nodes, bottom-up.
+        // Build a map: leafId → leaf info for summarization.
+        const leafMap = new Map();
+        for (const l of existingLeaves)
+            leafMap.set(l.id, l);
+        for (const l of reallyNewLeaves)
+            leafMap.set(l.id, l);
+        // Recompute affected Level-0 nodes.
+        for (const nid of affectedNodeIds) {
+            const node = tree.nodes.get(nid);
+            if (!node)
+                continue;
+            const coveredLeaves = node.children
+                .map((cid) => leafMap.get(cid))
+                .filter((l) => !!l);
+            if (coveredLeaves.length > 0) {
+                node.embedding = meanVector(coveredLeaves.map((l) => l.embedding));
+                // Summarize: flatten messages from all covered leaves.
+                const messages = coveredLeaves.flatMap((l) => l.messages);
+                if (messages.length > 0) {
+                    const cs = summarizeCluster(messages);
+                    node.summary = cs.summary;
+                    node.tokenEstimate = cs.tokenEstimate;
+                }
+            }
+        }
+        // Propagate up: for each higher level, recompute nodes whose leaf set
+        // includes any newly inserted leaf. Since `children` stores flattened leaf
+        // ids (not node ids), we check intersection with the new leaf id set.
+        const newLeafIds = new Set(reallyNewLeaves.map((l) => l.id));
+        const sortedLevels = [...levelGroups.keys()].sort((a, b) => a - b);
+        for (const level of sortedLevels.slice(1)) {
+            const levelNodes = levelGroups.get(level) ?? [];
+            for (const stored of levelNodes) {
+                const node = tree.nodes.get(stored.id);
+                if (!node)
+                    continue;
+                if (node.children.some((cid) => newLeafIds.has(cid))) {
+                    // Recompute embedding as centroid of covered leaves.
+                    const covered = node.children
+                        .map((cid) => leafMap.get(cid))
+                        .filter((l) => !!l);
+                    if (covered.length > 0) {
+                        node.embedding = meanVector(covered.map((l) => l.embedding));
+                        const messages = covered.flatMap((l) => l.messages);
+                        if (messages.length > 0) {
+                            const cs = summarizeCluster(messages);
+                            node.summary = cs.summary;
+                            node.tokenEstimate = cs.tokenEstimate;
+                        }
+                    }
+                }
+            }
+        }
+        // 7. Persist the full updated tree (delete + reinsert in a tx).
+        tree.builtAt = builtAt;
+        saveRaptorTree(opts.sessionId, tree, builtAt, opts.stateDir);
+        logger?.info("raptor_incremental_success", {
+            sessionId: opts.sessionId,
+            nodeCount: tree.nodes.size,
+            newLeaves: reallyNewLeaves.length,
+            affectedNodes: affectedNodeIds.size,
+        });
+        return tree;
+    }
+    catch (e) {
+        logger?.error("raptor_incremental_failed", {
+            sessionId: opts.sessionId,
+            error: String(e instanceof Error ? e.message : e),
+        });
+        return null;
+    }
+}
+// ---------------------------------------------------------------------------
+// Helpers
+// ---------------------------------------------------------------------------
+/**
+ * Full rebuild fallback: merges existingLeaves + newLeaves, builds from scratch.
+ */
+function fullRebuild(existingLeaves, newLeaves, opts, logger) {
+    logger?.info("raptor_incremental_full_rebuild", {
+        sessionId: opts.sessionId,
+    });
+    try {
+        // Merge: deduplicate by leaf id.
+        const seen = new Set();
+        const allLeaves = [];
+        for (const l of existingLeaves) {
+            if (!seen.has(l.id)) {
+                seen.add(l.id);
+                allLeaves.push(l);
+            }
+        }
+        for (const l of newLeaves) {
+            if (!seen.has(l.id)) {
+                seen.add(l.id);
+                allLeaves.push(l);
+            }
+        }
+        const tree = buildRaptorTree(allLeaves, {
+            embedder: opts.embedder ?? getDefaultEmbedder(),
+            budgetMs: opts.budgetMs,
+            clustersPerLevel: opts.clustersPerLevel,
+            consistencyThreshold: opts.consistencyThreshold,
+        });
+        const builtAt = opts.builtAt ?? Date.now();
+        saveRaptorTree(opts.sessionId, tree, builtAt, opts.stateDir);
+        return tree;
+    }
+    catch (e2) {
+        logger?.error("raptor_incremental_full_rebuild_failed", {
+            sessionId: opts.sessionId,
+            error: String(e2 instanceof Error ? e2.message : e2),
+        });
+        return null;
+    }
+}
+/** Rebuild the in-memory RaptorTree from stored nodes. */
+function rehydrateFromStored(nodes) {
+    if (nodes.length === 0)
+        return null;
+    const root = nodes.reduce((best, n) => (!best || n.level > (best?.level ?? -1) ? n : best), null);
+    const tree = {
+        nodes: new Map(nodes.map((n) => [
+            n.id,
+            {
+                id: n.id,
+                level: n.level,
+                parentId: n.parentId,
+                children: n.children,
+                summary: n.summary,
+                embedding: n.embedding,
+                qualityMarker: n.qualityMarker,
+                tokenEstimate: n.tokenEstimate,
+            },
+        ])),
+        rootId: root?.id ?? null,
+        levels: Math.max(1, ...nodes.map((n) => n.level + 1)),
+        timedOut: root != null && root.level >= 99,
+        builtAt: nodes.reduce((max, n) => Math.max(max, n.builtAt), 0),
+    };
+    return tree;
+}

package/dist/src/dedup/raptor/index.js

@@ -11,9 +11,11 @@
  */
 import { defaultEmbedder } from "../../embedder.js";
 import { buildRaptorTree } from "./tree.js";
+import { incrementRaptorTree } from "./incremental.js";
 import { stagedExpansion } from "./retrieval.js";
 import { saveRaptorTree, listRaptorNodes } from "../../store/sqlite.js";
 import { insertBuildHistory, computeCoherenceScore } from "./buildHistory.js";
+import { loadDedupConfig } from "../../config/dedup.js";
 /** Shadow mode is on by default; set RAPTOR_SHADOW_MODE=false to serve live. */
 export function isShadowMode() {
     return process.env.RAPTOR_SHADOW_MODE !== "false";
@@ -30,15 +32,37 @@ export function runRaptor(leaves, opts) {
     const embedder = opts.embedder ?? defaultEmbedder();
     const logger = opts.logger;
     const startedAt = opts.builtAt ?? Date.now();
+    const builtAt = startedAt;
     try {
-        const tree = buildRaptorTree(leaves, {
-            embedder,
-            budgetMs: opts.budgetMs,
-            clustersPerLevel: opts.clustersPerLevel,
-            consistencyThreshold: opts.consistencyThreshold,
-        });
-        const builtAt = opts.builtAt ?? Date.now();
-        saveRaptorTree(opts.sessionId, tree, builtAt, opts.stateDir);
+        // Try incremental update first when the flag is ON (Sprint 26, #7).
+        // The incremental path reads the existing tree, diffs by leaf id, and
+        // inserts only new leaves into the nearest clusters. Falls back to a full
+        // rebuild when the tree is missing, corrupted, or >50% of nodes are new.
+        const cfg = loadDedupConfig();
+        let tree = null;
+        if (cfg.RAPTOR_INCREMENTAL) {
+            tree = incrementRaptorTree(leaves, leaves, {
+                embedder,
+                stateDir: opts.stateDir,
+                sessionId: opts.sessionId,
+                budgetMs: opts.budgetMs,
+                clustersPerLevel: opts.clustersPerLevel,
+                consistencyThreshold: opts.consistencyThreshold,
+                logger,
+                builtAt,
+            });
+        }
+        // Fall back to full rebuild when incremental returned null (not attempted,
+        // tree missing, ratio exceeded, or internal error).
+        if (!tree) {
+            tree = buildRaptorTree(leaves, {
+                embedder,
+                budgetMs: opts.budgetMs,
+                clustersPerLevel: opts.clustersPerLevel,
+                consistencyThreshold: opts.consistencyThreshold,
+            });
+            saveRaptorTree(opts.sessionId, tree, builtAt, opts.stateDir);
+        }
         logger?.info("raptor_build", {
             sessionId: opts.sessionId,
             nodes: tree.nodes.size,

package/dist/src/dedup/raptor/summarizer.js

@@ -17,7 +17,7 @@ function ollamaEndpoint() {
     const model = process.env.MEGACOMPACT_RAPTOR_MODEL;
     if (!model)
         return null;
-    const base = process.env.MEGACOMPACT_RAPTOR_URL ?? "http://127.0.0.1:11434";
+    const base = process.env.MEGACOMPACT_RAPTOR_URL ?? "http://127.0.0.1:11434"; // guardrails-allow PREVENT-PI-004: loopback-only Ollama endpoint for local RAPTOR summarization
     // Guard: only loopback is permitted (remote Ollama would violate PREVENT-PI-004).
     if (!/^https?:\/\/(localhost|127\.0\.0\.1)([:/]|$)/.test(base)) {
         throw new Error(`MEGACOMPACT_RAPTOR_URL must be localhost/127.0.0.1 (got ${base}). ` +

package/dist/src/httpEmbedder.js

@@ -26,7 +26,7 @@
  */
 import { l2Normalize } from "./embedder.js";
 import { spawnSync } from "node:child_process"; // guardrails-allow PREVENT-PI-004: localhost-only user-spawned embedding server (BYO backend, never remote)
-import { isIP } from "node:net";
+import { isIP } from "node:net"; // guardrails-allow PREVENT-PI-004: localhost-only loopback address validation in BYO embedding server
 // Inline worker script: resolves a hostname via dns.lookup in a child process
 // (dns.lookup is callback-async; the child has its own event loop). Used to
 // verify that a hostname in the embedding URL resolves to loopback ONLY.

package/extensions/mega-events/context-handler.ts

@@ -293,7 +293,7 @@ export function registerContextHandler(
 				runtime.trimCache.checkpointId === runtime.rt.lastCheckpointId &&
 				runtime.trimCache.cut <= messages.length
 			) {
-				const recent = messages.slice(runtime.trimCache.cut); // guardrails-allow PREVENT-PI-002
+				const recent = messages.slice(runtime.trimCache.cut); // guardrails-allow PREVENT-PI-002: cached `cut` was sanitized by computeLiveTrimCut (src/boundary.ts); replayed verbatim, transcript only grows within an epoch.
 				runtime.diagLiveTrimFires++;
 				runtime.diagLiveTrimReplays++;
 				runtime.snapshot(ctx);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-mega-compact",
-  "version": "0.11.9",
+  "version": "0.11.11",
   "description": "Layered, local, vector-backed context compressor for pi — supersede/collapse/cluster compaction with deduped inline recall.",
   "type": "module",
   "license": "BSD-3-Clause",

package/src/config/dedup.ts

@@ -66,6 +66,7 @@ export interface DedupConfigShape {
   RAPTOR_MULTILEVEL_ENABLED: boolean;
   RAPTOR_LEVEL_WEIGHTS: number[]; // per-level weights, index 0 = leaves (uncalibrated)
   RAPTOR_LEAF_EXPANSION: boolean;
+  RAPTOR_INCREMENTAL: boolean;
   RAPTOR_MAX_LEAF_EXPANSION: number; // uncalibrated
   RAPTOR_FRESHNESS_HOURS: number; // S42D: skip rebuild when tree is fresh (uncalibrated)
   /** S25 Phase-2: inject top-level RAPTOR summary nodes (root + level-1
@@ -109,6 +110,7 @@ export function loadDedupConfig(): DedupConfigShape {
     RAPTOR_MULTILEVEL_ENABLED: envBool("MEGACOMPACT_RAPTOR_MULTILEVEL", true),
     RAPTOR_LEVEL_WEIGHTS: envNumArray("MEGACOMPACT_RAPTOR_LEVEL_WEIGHTS", [1.0, 0.9, 0.8, 0.7, 0.5]),
     RAPTOR_LEAF_EXPANSION: envBool("MEGACOMPACT_RAPTOR_LEAF_EXPANSION", true),
+    RAPTOR_INCREMENTAL: envBool("MEGACOMPACT_RAPTOR_INCREMENTAL", true),
     RAPTOR_MAX_LEAF_EXPANSION: envNum("MEGACOMPACT_RAPTOR_MAX_LEAF_EXP", 10),
     RAPTOR_FRESHNESS_HOURS: envNum("MEGACOMPACT_RAPTOR_FRESHNESS_HOURS", 4),
     RAPTOR_INJECT_SUMMARIES: envBool("MEGACOMPACT_RAPTOR_INJECT_SUMMARIES", true),

package/src/dedup/raptor/incremental.ts

@@ -0,0 +1,339 @@
+/**
+ * incremental.ts — incremental RAPTOR tree update (Sprint 26, #7).
+ *
+ * Instead of rebuilding the full tree (~2s per compaction), insert only the new
+ * checkpoints since the last build and recompute only the affected cluster
+ * assignments up the tree. Falls back to a full rebuild when the tree is
+ * missing, corrupted, or >50% of nodes are new.
+ *
+ * Non-fatal: every failure logs and returns null (caller falls back to full
+ * rebuild). PREVENT-PI-004: zero network; all operations are local SQLite + CPU.
+ */
+
+import type { Embedder, Vector } from "../../embedder.js";
+import { cosineSimilarity } from "../../embedder.js";
+import { defaultEmbedder as getDefaultEmbedder } from "../../embedder.js";
+import { buildRaptorTree, type Leaf, type RaptorTree } from "./tree.js";
+import type { QualityMarker } from "./guardrails.js";
+import { meanVector } from "./kmeans.js";
+import { summarizeCluster } from "./summarizer.js";
+import type { Logger } from "../../log.js";
+import {
+  saveRaptorTree,
+  listRaptorNodes,
+  type StoredRaptorNode,
+} from "../../store/sqlite.js";
+
+/** When >FRESH_THRESHOLD of existing leaves are new, a full rebuild is cheaper. */
+const FRESH_THRESHOLD = 0.5;
+
+/**
+ * Incrementally update a persisted RAPTOR tree with new leaves.
+ *
+ * 1. Reads the existing tree from the store.
+ * 2. Filters `newLeaves` to those NOT already covered by the tree.
+ * 3. If no new leaves → early-return (existing tree).
+ * 4. If no existing tree or >50% new → full rebuild.
+ * 5. For each new leaf, finds the best-matching Level-0 cluster node,
+ *    reassigns (inserts into its children), updates that node's summary
+ *    and centroid embedding.
+ * 6. Propagates changes up through parent levels to the root.
+ * 7. Persists the updated tree (full delete + reinsert in a tx).
+ *
+ * Never throws. Returns the updated tree, or null on failure/fallback.
+ */
+export function incrementRaptorTree(
+  existingLeaves: Leaf[],
+  newLeaves: Leaf[],
+  opts: {
+    embedder?: Embedder;
+    stateDir: string;
+    sessionId: string;
+    budgetMs?: number;
+    clustersPerLevel?: number;
+    consistencyThreshold?: number;
+    logger?: Logger;
+    builtAt?: number;
+  },
+): RaptorTree | null {
+  const logger = opts.logger;
+  const builtAt = opts.builtAt ?? Date.now();
+
+  try {
+    // 1. Read existing tree.
+    const storedNodes = listRaptorNodes(opts.sessionId, opts.stateDir);
+    if (storedNodes.length === 0) {
+      logger?.info("raptor_incremental_no_tree", { sessionId: opts.sessionId });
+      return null; // no existing tree — caller falls back to full buildRaptorTree
+    }
+
+    // 2. Find new leaf ids not already in the tree.
+    const coveredLeafIds = new Set<string>();
+    for (const n of storedNodes) {
+      for (const cid of n.children) coveredLeafIds.add(cid);
+    }
+    const reallyNewLeaves = newLeaves.filter(
+      (l) => !coveredLeafIds.has(l.id),
+    );
+    const totalCovered = coveredLeafIds.size;
+
+    if (reallyNewLeaves.length === 0) {
+      logger?.info("raptor_incremental_no_new", { sessionId: opts.sessionId });
+      return rehydrateFromStored(storedNodes);
+    }
+
+    // 4. Fallback guard: >50% new → full rebuild.
+    if (
+      totalCovered > 0 &&
+      reallyNewLeaves.length / totalCovered > FRESH_THRESHOLD
+    ) {
+      logger?.info("raptor_incremental_fallback_ratio", {
+        sessionId: opts.sessionId,
+        coveredNodes: totalCovered,
+        newLeaves: reallyNewLeaves.length,
+        threshold: FRESH_THRESHOLD,
+      });
+      return fullRebuild(existingLeaves, reallyNewLeaves, opts, logger);
+    }
+
+    // 5. Build the in-memory tree from stored nodes.
+    const tree = rehydrateFromStored(storedNodes);
+    if (!tree || tree.nodes.size === 0) {
+      logger?.info("raptor_incremental_rehydrate_failed", {
+        sessionId: opts.sessionId,
+      });
+      return fullRebuild(existingLeaves, reallyNewLeaves, opts, logger);
+    }
+
+    // Group stored nodes by level.
+    const levelGroups = new Map<number, StoredRaptorNode[]>();
+    for (const n of storedNodes) {
+      const g = levelGroups.get(n.level);
+      if (g) g.push(n);
+      else levelGroups.set(n.level, [n]);
+    }
+    // Level-0 covers leaf ids directly.
+    const level0Nodes =
+      levelGroups.get(0) ??
+      storedNodes.filter((n) => {
+        // A node is level 0 if its level is the minimum.
+        const minLevel = Math.min(...storedNodes.map((x) => x.level));
+        return n.level === minLevel;
+      });
+    if (level0Nodes.length === 0) {
+      // Single-root tree with no hierarchy: cannot increment, fall back.
+      return fullRebuild(existingLeaves, reallyNewLeaves, opts, logger);
+    }
+
+    // For each new leaf, assign to the nearest Level-0 cluster node.
+    const leafEmbeddings = new Map<string, Vector>();
+    for (const l of reallyNewLeaves) leafEmbeddings.set(l.id, l.embedding);
+
+    // Track which Level-0 nodes changed (by their stored id).
+    const affectedNodeIds = new Set<string>();
+
+    for (const leaf of reallyNewLeaves) {
+      const emb = leafEmbeddings.get(leaf.id);
+      if (!emb || emb.length === 0) continue;
+
+      // Find best matching Level-0 node.
+      let bestNode: StoredRaptorNode | null = null;
+      let bestSim = -Infinity;
+      for (const cn of level0Nodes) {
+        const sim = cosineSimilarity(emb, cn.embedding);
+        if (sim > bestSim) {
+          bestSim = sim;
+          bestNode = cn;
+        }
+      }
+      if (!bestNode) continue;
+
+      // Add the leaf id to this node's children.
+      const updated = tree.nodes.get(bestNode.id);
+      if (updated) {
+        if (!updated.children.includes(leaf.id)) {
+          updated.children.push(leaf.id);
+        }
+        affectedNodeIds.add(bestNode.id);
+      }
+    }
+
+    if (affectedNodeIds.size === 0) {
+      // No matches found — fall back to full rebuild.
+      logger?.info("raptor_incremental_no_matches", {
+        sessionId: opts.sessionId,
+      });
+      return fullRebuild(existingLeaves, reallyNewLeaves, opts, logger);
+    }
+
+    // 6. Recompute embeddings and summaries for affected nodes, bottom-up.
+    // Build a map: leafId → leaf info for summarization.
+    const leafMap = new Map<string, Leaf>();
+    for (const l of existingLeaves) leafMap.set(l.id, l);
+    for (const l of reallyNewLeaves) leafMap.set(l.id, l);
+
+    // Recompute affected Level-0 nodes.
+    for (const nid of affectedNodeIds) {
+      const node = tree.nodes.get(nid);
+      if (!node) continue;
+      const coveredLeaves = node.children
+        .map((cid) => leafMap.get(cid))
+        .filter((l): l is Leaf => !!l);
+      if (coveredLeaves.length > 0) {
+        node.embedding = meanVector(
+          coveredLeaves.map((l) => l.embedding),
+        );
+        // Summarize: flatten messages from all covered leaves.
+        const messages = coveredLeaves.flatMap((l) => l.messages);
+        if (messages.length > 0) {
+          const cs = summarizeCluster(messages);
+          node.summary = cs.summary;
+          node.tokenEstimate = cs.tokenEstimate;
+        }
+      }
+    }
+
+    // Propagate up: for each higher level, recompute nodes whose leaf set
+    // includes any newly inserted leaf. Since `children` stores flattened leaf
+    // ids (not node ids), we check intersection with the new leaf id set.
+    const newLeafIds = new Set(reallyNewLeaves.map((l) => l.id));
+    const sortedLevels = [...levelGroups.keys()].sort((a, b) => a - b);
+    for (const level of sortedLevels.slice(1)) {
+      const levelNodes = levelGroups.get(level) ?? [];
+      for (const stored of levelNodes) {
+        const node = tree.nodes.get(stored.id);
+        if (!node) continue;
+
+        if (node.children.some((cid) => newLeafIds.has(cid))) {
+          // Recompute embedding as centroid of covered leaves.
+          const covered = node.children
+            .map((cid) => leafMap.get(cid))
+            .filter((l): l is Leaf => !!l);
+          if (covered.length > 0) {
+            node.embedding = meanVector(
+              covered.map((l) => l.embedding),
+            );
+            const messages = covered.flatMap((l) => l.messages);
+            if (messages.length > 0) {
+              const cs = summarizeCluster(messages);
+              node.summary = cs.summary;
+              node.tokenEstimate = cs.tokenEstimate;
+            }
+          }
+        }
+      }
+    }
+
+    // 7. Persist the full updated tree (delete + reinsert in a tx).
+    tree.builtAt = builtAt;
+    saveRaptorTree(opts.sessionId, tree, builtAt, opts.stateDir);
+
+    logger?.info("raptor_incremental_success", {
+      sessionId: opts.sessionId,
+      nodeCount: tree.nodes.size,
+      newLeaves: reallyNewLeaves.length,
+      affectedNodes: affectedNodeIds.size,
+    });
+
+    return tree;
+  } catch (e) {
+    logger?.error("raptor_incremental_failed", {
+      sessionId: opts.sessionId,
+      error: String(e instanceof Error ? e.message : e),
+    });
+    return null;
+  }
+}
+
+// ---------------------------------------------------------------------------
+// Helpers
+// ---------------------------------------------------------------------------
+
+/**
+ * Full rebuild fallback: merges existingLeaves + newLeaves, builds from scratch.
+ */
+function fullRebuild(
+  existingLeaves: Leaf[],
+  newLeaves: Leaf[],
+  opts: {
+    embedder?: Embedder;
+    budgetMs?: number;
+    clustersPerLevel?: number;
+    consistencyThreshold?: number;
+    stateDir: string;
+    sessionId: string;
+    logger?: Logger;
+    builtAt?: number;
+  },
+  logger?: Logger,
+): RaptorTree | null {
+  logger?.info("raptor_incremental_full_rebuild", {
+    sessionId: opts.sessionId,
+  });
+  try {
+    // Merge: deduplicate by leaf id.
+    const seen = new Set<string>();
+    const allLeaves: Leaf[] = [];
+    for (const l of existingLeaves) {
+      if (!seen.has(l.id)) {
+        seen.add(l.id);
+        allLeaves.push(l);
+      }
+    }
+    for (const l of newLeaves) {
+      if (!seen.has(l.id)) {
+        seen.add(l.id);
+        allLeaves.push(l);
+      }
+    }
+    const tree = buildRaptorTree(allLeaves, {
+      embedder: opts.embedder ?? getDefaultEmbedder(),
+      budgetMs: opts.budgetMs,
+      clustersPerLevel: opts.clustersPerLevel,
+      consistencyThreshold: opts.consistencyThreshold,
+    });
+    const builtAt = opts.builtAt ?? Date.now();
+    saveRaptorTree(opts.sessionId, tree, builtAt, opts.stateDir);
+    return tree;
+  } catch (e2) {
+    logger?.error("raptor_incremental_full_rebuild_failed", {
+      sessionId: opts.sessionId,
+      error: String(e2 instanceof Error ? e2.message : e2),
+    });
+    return null;
+  }
+}
+
+/** Rebuild the in-memory RaptorTree from stored nodes. */
+function rehydrateFromStored(
+  nodes: StoredRaptorNode[],
+): RaptorTree | null {
+  if (nodes.length === 0) return null;
+  const root = nodes.reduce<StoredRaptorNode | null>(
+    (best, n) => (!best || n.level > (best?.level ?? -1) ? n : best),
+    null,
+  );
+  const tree: RaptorTree = {
+    nodes: new Map(
+      nodes.map((n) => [
+        n.id,
+        {
+          id: n.id,
+          level: n.level,
+          parentId: n.parentId,
+          children: n.children,
+          summary: n.summary,
+          embedding: n.embedding,
+          qualityMarker: n.qualityMarker as QualityMarker,
+          tokenEstimate: n.tokenEstimate,
+        },
+      ]),
+    ),
+    rootId: root?.id ?? null,
+    levels: Math.max(1, ...nodes.map((n) => n.level + 1)),
+    timedOut: root != null && root.level >= 99,
+    builtAt: nodes.reduce((max, n) => Math.max(max, n.builtAt), 0),
+  };
+  return tree;
+}
+

package/src/dedup/raptor/index.ts

@@ -13,10 +13,12 @@
 import type { Embedder } from "../../embedder.js";
 import { defaultEmbedder } from "../../embedder.js";
 import { buildRaptorTree, type Leaf, type RaptorTree } from "./tree.js";
+import { incrementRaptorTree } from "./incremental.js";
 import { stagedExpansion } from "./retrieval.js";
 import { Logger } from "../../log.js";
 import { saveRaptorTree, listRaptorNodes } from "../../store/sqlite.js";
 import { insertBuildHistory, computeCoherenceScore } from "./buildHistory.js";
+import { loadDedupConfig } from "../../config/dedup.js";
 
 /** Shadow mode is on by default; set RAPTOR_SHADOW_MODE=false to serve live. */
 export function isShadowMode(): boolean {
@@ -51,15 +53,38 @@ export function runRaptor(
   const embedder = opts.embedder ?? defaultEmbedder();
   const logger = opts.logger;
   const startedAt = opts.builtAt ?? Date.now();
+  const builtAt = startedAt;
   try {
-    const tree = buildRaptorTree(leaves, {
-      embedder,
-      budgetMs: opts.budgetMs,
-      clustersPerLevel: opts.clustersPerLevel,
-      consistencyThreshold: opts.consistencyThreshold,
-    });
-    const builtAt = opts.builtAt ?? Date.now();
-    saveRaptorTree(opts.sessionId, tree, builtAt, opts.stateDir);
+    // Try incremental update first when the flag is ON (Sprint 26, #7).
+    // The incremental path reads the existing tree, diffs by leaf id, and
+    // inserts only new leaves into the nearest clusters. Falls back to a full
+    // rebuild when the tree is missing, corrupted, or >50% of nodes are new.
+    const cfg = loadDedupConfig();
+    let tree: RaptorTree | null = null;
+    if (cfg.RAPTOR_INCREMENTAL) {
+      tree = incrementRaptorTree(leaves, leaves, {
+        embedder,
+        stateDir: opts.stateDir,
+        sessionId: opts.sessionId,
+        budgetMs: opts.budgetMs,
+        clustersPerLevel: opts.clustersPerLevel,
+        consistencyThreshold: opts.consistencyThreshold,
+        logger,
+        builtAt,
+      });
+    }
+
+    // Fall back to full rebuild when incremental returned null (not attempted,
+    // tree missing, ratio exceeded, or internal error).
+    if (!tree) {
+      tree = buildRaptorTree(leaves, {
+        embedder,
+        budgetMs: opts.budgetMs,
+        clustersPerLevel: opts.clustersPerLevel,
+        consistencyThreshold: opts.consistencyThreshold,
+      });
+      saveRaptorTree(opts.sessionId, tree, builtAt, opts.stateDir);
+    }
     logger?.info("raptor_build", {
       sessionId: opts.sessionId,
       nodes: tree.nodes.size,

package/src/dedup/raptor/summarizer.ts

@@ -24,7 +24,7 @@ export interface ClusterSummary {
 function ollamaEndpoint(): { url: string; model: string } | null {
   const model = process.env.MEGACOMPACT_RAPTOR_MODEL;
   if (!model) return null;
-  const base = process.env.MEGACOMPACT_RAPTOR_URL ?? "http://127.0.0.1:11434";
+  const base = process.env.MEGACOMPACT_RAPTOR_URL ?? "http://127.0.0.1:11434"; // guardrails-allow PREVENT-PI-004: loopback-only Ollama endpoint for local RAPTOR summarization
   // Guard: only loopback is permitted (remote Ollama would violate PREVENT-PI-004).
   if (!/^https?:\/\/(localhost|127\.0\.0\.1)([:/]|$)/.test(base)) {
     throw new Error(

package/src/httpEmbedder.ts

@@ -28,7 +28,7 @@
 import type { Embedder, Vector } from "./embedder.js";
 import { l2Normalize } from "./embedder.js";
 import { spawnSync } from "node:child_process"; // guardrails-allow PREVENT-PI-004: localhost-only user-spawned embedding server (BYO backend, never remote)
-import { isIP } from "node:net";
+import { isIP } from "node:net"; // guardrails-allow PREVENT-PI-004: localhost-only loopback address validation in BYO embedding server
 
 export interface HttpEmbedderOptions {
   url: string;