pi-mega-compact 0.7.7 → 0.7.9

package/README.md

@@ -6,7 +6,7 @@ sessions into a **local SQLite store** and offers **deduped inline recall** —
 running **locally inside the extension**, with **no remote MCP server** and
 **zero network calls at runtime** (PREVENT-PI-004).
 
-> **Status - v0.7.7.** Storage uses the built-in `node:sqlite` backend
+> **Status - v0.7.9.** Storage uses the built-in `node:sqlite` backend
 > (`DatabaseSync`, Node >=22.13): zero native build step, fully local, and zero
 > network at runtime. See [RELEASE_NOTES.md](RELEASE_NOTES.md) for the full
 > changelog.
@@ -96,7 +96,7 @@ Marker    insert compact-marker; dedupe so repeated triggers cost ~0 tokens
 **One store, three ways to read it back — one dedup engine:**
 
 | Entry point | Trigger | Behavior |
-|---|---|---|
+| --- | --- | --- |
 | **Auto-inline** (Layer 5) | `session_start` / `session_tree` | Resume → `recallAndInline(source:"resume")` prepends the most relevant checkpoints, deduped against current context. |
 | **On-demand recall** | `/mega-recall [query]` | Semantic search the store, dedupe, and inline the top-K. |
 | **Dedup sentinel** | every compact | A lightweight `mega-compact-marker` entry lets auto-inline and recall skip re-injecting / re-vectorizing already-present regions. |
@@ -243,7 +243,7 @@ The commands (slash commands inside pi):
 ### Commands
 
 | Command | Description |
-|---|---|
+| --- | --- |
 | `/mega-compact [summary...]` | Manually compact the current session. A summary arg is used verbatim; otherwise the COLLAPSE heuristics build one. Persists a `chkpt_xxx`. |
 | `/mega-compact off` | Disable auto-compaction for this session. |
 | `/mega-status` | Show config + current context usage + store stats (checkpoint count, dedup rate, tokens saved) + the **installed version**. |
@@ -267,13 +267,12 @@ The commands (slash commands inside pi):
 
 The **tier** you see in the toolbar and dashboard is a *live pressure band* (`low` → `medium` → `high` → `ultra` → `mega`) that climbs automatically as your context window fills and falls back as it's relieved — it is driven by `currentTokens / effectiveThreshold`, not a manual setting. The base compaction *threshold* is set by `MEGACOMPACT_TIER` at startup as a **% of the model context window** (`low` 50% · `medium` 60% · `high` 70% · `ultra` 70% · `mega` 75%; default `low`) — the fire point is `tierPct × contextWindow`, so it always lands below pi's native ~80% auto-compaction (any model size). The old static token amounts (50k/100k/200k/1M/10M) are now only the boot fallback used before the first context event reports a window. `/mega-tier` was removed in v0.7.6. Higher pressure also deepens the live trim and reviews durable memory more often — the whole system reacts as one.
 
-
 ### Live stats widget
 
 Above the pi editor the extension shows a compact widget:
 
 ```
- ⚡ high·low v0.7.7 │ 142k/200k tokens (71%) │ 3 chkpts │ 🤖 2 agents │ turn 5
+ ⚡ high·low v0.7.9 │ 142k/200k tokens (71%) │ 3 chkpts │ 🤖 2 agents │ turn 5
    ◐ armed │ dedup: 92% │ saved: 45k tok
 ```
 
@@ -304,17 +303,17 @@ before starting pi.
 ### Core settings
 
 | Variable | Default | Meaning |
-|---|---|---|
+| --- | --- | --- |
 | `MEGACOMPACT_FAST_GATE_PCT` | `70` | Context-usage % that arms the auto-trigger. Defaults to the tier's % of window (`tierPct*100`): low 50 · med 60 · high 70 · ultra 70 · mega 75. Override raises the arming floor. |
 | `MEGACOMPACT_TIER` | `low` | Named trigger preset — sets the compaction threshold as a **% of the model context window**: `low`(50%) `medium`(60%) `high`(70%) `ultra`(70%) `mega`(75%). Fire point = `tierPct × contextWindow`, so it always fires below pi's native ~80% auto-compaction (any model size). The old static token amounts (50k/100k/200k/1M/10M) are now only the **boot fallback** used before the first context event reports a window. Default `low`. |
-| `MEGACOMPACT_THRESHOLD_TOKENS` | _(tier default)_ | Explicit **absolute** token budget (the `custom` tier). Overrides `MEGACOMPACT_TIER` when set and is **never percent-scaled** — use this to pin an exact token fire point regardless of model window. |
+| `MEGACOMPACT_THRESHOLD_TOKENS` | *(tier default)* | Explicit **absolute** token budget (the `custom` tier). Overrides `MEGACOMPACT_TIER` when set and is **never percent-scaled** — use this to pin an exact token fire point regardless of model window. |
 | `MEGACOMPACT_ANCHOR_USER_MESSAGES` | `3` | Never drop the most recent N user messages (anchor floor). |
 | `MEGACOMPACT_PRESERVE_RECENT` | `4` | Preserve the most recent N messages verbatim. |
 | `MEGACOMPACT_AUTO` | `true` | Enable the auto-trigger. |
 | `MEGACOMPACT_AUTO_INLINE` | `true` | Auto-inline on resume / branch. |
 | `MEGACOMPACT_AUTO_INLINE_K` | `3` | Top-K checkpoints to auto-inline. |
 | `MEGACOMPACT_DEDUP_SIM` | `0.90` | Cosine threshold to collapse near-dupes. |
-| `MEGACOMPACT_STATE_DIR` | _(none — per-repo default)_ | Override the store location. By default state is per-repo at `<repo>/.pi/mega-compact/`; this env var forces a single explicit dir (used as the fallback for non-git cwds). |
+| `MEGACOMPACT_STATE_DIR` | *(none — per-repo default)* | Override the store location. By default state is per-repo at `<repo>/.pi/mega-compact/`; this env var forces a single explicit dir (used as the fallback for non-git cwds). |
 
 ### Dedup pipeline flags
 
@@ -323,7 +322,7 @@ behavior. `MARK_ONLY_*` tiers run + record their decision but never
 collapse (safe partial-rollout / auto-degrade state).
 
 | Variable | Default | Meaning |
-|---|---|---|
+| --- | --- | --- |
 | `MEGACOMPACT_L0_ENABLED` | `true` | L0 exact content-hash dedup. |
 | `MEGACOMPACT_L1_ENABLED` | `true` | L1 MinHash/LSH near-dup verification. |
 | `MEGACOMPACT_L2_ENABLED` | `true` | L2 semantic cosine dedup + MMR retrieval diversity. |
@@ -332,7 +331,7 @@ collapse (safe partial-rollout / auto-degrade state).
 | `MEGACOMPACT_MARK_ONLY_L1` | `false` | L1: record, don't collapse. |
 | `MEGACOMPACT_MARK_ONLY_L2` | `false` | L2: record, don't collapse. |
 | `MEGACOMPACT_MINILM` | `false` | MiniLM embedder flag — **off; not shipped** (see Embedding). BYO via `MEGACOMPACT_EMBEDDING_URL`. |
-| `MEGACOMPACT_EMBEDDING_URL` | _(unset)_ | BYO localhost embedder endpoint (loopback-only; enables `HttpEmbedder`). |
+| `MEGACOMPACT_EMBEDDING_URL` | *(unset)* | BYO localhost embedder endpoint (loopback-only; enables `HttpEmbedder`). |
 | `MEGACOMPACT_L2_THRESHOLD` | `0.85` | L2 cosine firing point (trigram-honest; set higher for semantic backends). |
 | `MEGACOMPACT_L1_JACCARD` | `0.8` | L1 MinHash/LSH near-dup Jaccard threshold. |
 | `MEGACOMPACT_MMR_LAMBDA` | `0.5` | MMR retrieval-diversity weight (λ·relevance − (1−λ)·maxSim). |
@@ -349,13 +348,13 @@ checklist, MARK_ONLY degrade) and `docs/RETENTION_POLICY.md` for TTL / soft-dele
 ### Continuity & memory
 
 | Variable | Default | Meaning |
-|---|---|---|
+| --- | --- | --- |
 | `MEGACOMPACT_LEGACY_DURABLE_TRIM` | `false` | Restore the legacy auto-trigger (`ctx.compact()` stops the agent). One-release rollback; default uses live context-event trim + pi native auto-compaction (compact-and-continue). |
 | `MEGACOMPACT_CROSSREPO_ENABLED` | `true` | Cross-repo recall on resume + `/mega-recall --cross-repo` (HNSW index over every repo). |
 | `MEGACOMPACT_CROSSREPO_COSINE` | `0.90` | Stricter cosine floor for cross-repo hits (vs `0.85` same-repo). |
 | `MEGACOMPACT_MEMORY_AUTO_REVIEW` | `true` | Auto-review the conversation every `MEGACOMPACT_MEMORY_REVIEW_INTERVAL` turns → durable memories. |
 | `MEGACOMPACT_MEMORY_REVIEW_INTERVAL` | `10` | Turns between auto-review cycles. |
-| `MEGACOMPACT_PGLITE_DISABLED` | _(unset — index on)_ | Kill-switch for the PGlite/HNSW cross-repo index; set `1`/`true` to disable (falls back to sync per-session scan). |
+| `MEGACOMPACT_PGLITE_DISABLED` | *(unset — index on)* | Kill-switch for the PGlite/HNSW cross-repo index; set `1`/`true` to disable (falls back to sync per-session scan). |
 
 ## Dashboard
 

package/dist/extensions/dashboard-server/helpers.js

@@ -0,0 +1,37 @@
+/**
+ * File-reading helper functions for the dashboard server.
+ */
+import { readFileSync } from "node:fs";
+export function readSnapshot(snapshotPath) {
+    try {
+        const raw = readFileSync(snapshotPath, "utf-8");
+        return JSON.parse(raw);
+    }
+    catch {
+        return {
+            version: 1,
+            updatedAt: null,
+            tier: "unknown",
+            config: { fastGatePct: 80, thresholdTokens: 100_000, anchorUserMessages: 1, preserveRecent: 2, auto: true, autoInlineK: 3 },
+            session: { id: null, state: null, persistedThisSession: false, lastCheckpointId: null, lastCompactedFrom: 0 },
+            context: { tokens: null, percent: null, contextWindow: 0 },
+            trigger: { armed: false, ready: false, currentTokens: null, thresholdTokens: 100_000, fastGatePct: 80 },
+            store: { checkpointCount: 0, totalTokenEstimate: 0, originalTokens: 0, tokensSaved: 0, injectedCount: 0, dedupHitRate: 0, storageDedupRate: 0, dedupCollapsed: 0 },
+            crew: { activeAgents: 0, currentTurn: 0 },
+            repo: { checkpointCount: 0, totalTokenEstimate: 0, originalTokens: 0, tokensSaved: 0, sessionCount: 0, dedupAttempts: 0, dedupCollapsed: 0, storageDedupRate: 0 },
+            integrity: { regionsRetained: 0, compressedOriginalBytes: 0, duplicatesCollapsed: 0, bytesPermanentlyDeleted: 0 },
+            model: undefined,
+        };
+    }
+}
+export function readFrom(path, charOffset) {
+    try {
+        const content = readFileSync(path, "utf-8");
+        if (content.length <= charOffset)
+            return { data: "", offset: charOffset };
+        return { data: content.slice(charOffset), offset: content.length };
+    }
+    catch {
+        return { data: "", offset: charOffset };
+    }
+}

package/dist/extensions/dashboard-server/html/all-repos-tab.js

@@ -0,0 +1,26 @@
+/**
+ * "All repos" tab panel — machine-wide registry table (index.sqlite).
+ *
+ * Rows are populated by the index poller in `script.ts` (shared with the
+ * in-panel table on the Current-repo tab via the same render call).
+ */
+export function allReposTab() {
+    return `<!-- All repos (machine-wide registry from index.sqlite) -->
+<div class="tab-panel" id="panel-all">
+  <table class="repos">
+    <thead>
+      <tr>
+        <th>Repo</th><th>Model</th>
+        <th style="text-align:right">Checkpoints</th>
+        <th style="text-align:right">Tokens Saved</th>
+        <th style="text-align:right">Retained</th>
+        <th style="text-align:right">Last Compacted</th>
+      </tr>
+    </thead>
+    <tbody id="all-rows"><tr><td colspan="6" class="repo-none">loading…</td></tr></tbody>
+  </table>
+  <div class="updated" id="all-updated"></div>
+</div>
+
+`;
+}

package/dist/extensions/dashboard-server/html/body-open.js

@@ -0,0 +1,23 @@
+/**
+ * Body opening: </head><body>, offline banner, page heading, and the tab nav.
+ *
+ * `tierName` is interpolated into the heading tier badge.
+ * Includes the new "High Score" future-tab button (next project).
+ */
+export function bodyOpen(tierName) {
+    return `</head>
+<body>
+
+<div class="offline-banner" id="offline-banner">Dashboard data unavailable — waiting for a pi session to write snapshot...</div>
+
+<h1><span>mega-compact</span><span class="tier">${tierName}</span><span class="model-pill" id="hdr-model">—</span></h1>
+
+<nav class="tabs">
+  <button class="tab active" data-tab="current">Current repo</button>
+  <button class="tab" data-tab="all">All repos</button>
+  <button class="tab" data-tab="summary">Summary</button>
+  <button class="tab future" data-tab="highscore">High Score<span class="soon">soon</span></button>
+</nav>
+
+`;
+}

package/dist/extensions/dashboard-server/html/current-repo-tab.js

@@ -0,0 +1,130 @@
+/**
+ * "Current repo" tab panel — the per-repo dashboard view.
+ *
+ * Contains: context-window meter, trigger status, vector-store stats,
+ * repo-wide stats, data-safety card, configuration, model & cost savings,
+ * crew/agents, a legend, the live event stream, and the in-panel all-repos
+ * table (shared rows with the All-repos tab via the index poller).
+ *
+ * `tierName` is interpolated into the Configuration card.
+ */
+export function currentRepoTab(tierName) {
+    return `<!-- Current repo (existing single-repo view) -->
+<div class="tab-panel" id="panel-current">
+<div class="grid">
+  <div class="card">
+    <h2>Context Window</h2>
+    <div class="meter-label" id="ctx-pct">—</div>
+    <div class="meter-track"><div class="meter-fill" id="ctx-bar" style="width:0%"></div></div>
+    <div class="meter-sub" id="ctx-sub">waiting for data</div>
+  </div>
+  <div class="card">
+    <h2>Trigger Status</h2>
+    <div class="status-row"><div class="bullet" id="tr-armed"></div><span>Armed (context ≥ fast gate)</span></div>
+    <div class="status-row"><div class="bullet" id="tr-ready"></div><span>Ready (tokens ≥ threshold)</span></div>
+    <div class="state-text" id="tr-state">waiting</div>
+  </div>
+  <div class="card">
+    <h2>Vector Store</h2>
+    <div class="stat-grid">
+      <span class="label" title="A saved summary of a chunk of your conversation that was compacted to free up space.">Checkpoints</span><span class="value" id="st-count">0</span>
+      <span class="label" title="How much conversation we are currently holding as compact summaries (the 'memory' this extension keeps). Smaller is better.">Tokens Stored</span><span class="value" id="st-tokens">0</span>
+      <span class="label" title="Total size of the original conversation text before it was compacted.">Original Tokens</span><span class="value" id="st-orig">0</span>
+      <span class="label" title="How much conversation space we have freed up for you (original size minus the compact summary we kept).">Tokens Saved</span><span class="value" id="st-saved">0</span>
+      <span class="label" title="How many times old context was automatically brought back into the conversation because it was relevant to what you were doing.">Injected</span><span class="value" id="st-injected">0</span>
+      <span class="label" title="Of the times we recalled old context, how often it was actually on-topic.">Recall Relevance</span><span class="value" id="st-dedup">0%</span>
+      <span class="label" title="How often new content matched something we already had, so we skipped storing a duplicate copy. Higher = less wasted space.">Storage Dedup</span><span class="value" id="st-sdedup">0%</span>
+      <span class="label" title="How many duplicate chunks we collapsed into one instead of storing separately.">Collapsed</span><span class="value" id="st-collapsed">0</span>
+      <span class="label" title="The ID of the most recent saved checkpoint.">Last ID</span><span class="value" id="st-lastid">—</span>
+    </div>
+  </div>
+  <div class="card">
+    <h2>Repo (all sessions)</h2>
+    <div class="stat-grid">
+      <span class="label">Checkpoints</span><span class="value" id="rp-count">0</span>
+      <span class="label">Tokens Stored</span><span class="value" id="rp-tokens">0</span>
+      <span class="label">Original Tokens</span><span class="value" id="rp-orig">0</span>
+      <span class="label">Tokens Saved</span><span class="value" id="rp-saved">0</span>
+      <span class="label">Sessions</span><span class="value" id="rp-sessions">0</span>
+      <span class="label">Collapsed</span><span class="value" id="rp-collapsed">0</span>
+      <span class="label">Storage Dedup</span><span class="value" id="rp-sdedup">0%</span>
+    </div>
+  </div>
+  <div class="card safe">
+    <h2>🛡 Data Safety</h2>
+    <div class="stat-grid">
+      <span class="label">Regions Retained</span><span class="value" id="ig-retained">0</span>
+      <span class="label">Compressed-Original</span><span class="value" id="ig-bytes">0 B</span>
+      <span class="label">Dedup Duplicates</span><span class="value" id="ig-dupes">0</span>
+      <span class="label">Permanently Deleted</span><span class="value ok" id="ig-deleted">0 B</span>
+    </div>
+    <p class="safe-note">Every compacted region is kept verbatim (compressed). "Drop" = removed from the live window only. We never delete your data.</p>
+  </div>
+  <div class="card">
+    <h2>Configuration</h2>
+    <div class="conf-grid">
+      <span class="label">Tier</span><span class="value" id="cf-tier">${tierName}</span>
+      <span class="label">Threshold</span><span class="value" id="cf-threshold">—</span>
+      <span class="label">Fast Gate</span><span class="value" id="cf-gate">—</span>
+      <span class="label">Auto</span><span class="value" id="cf-auto">—</span>
+      <span class="label">Anchor</span><span class="value" id="cf-anchor">—</span>
+    </div>
+  </div>
+  <div class="card cost">
+    <h2>💰 Model &amp; Cost Savings</h2>
+    <div class="cost-usd" id="cost-usd">≈ $0.00 saved</div>
+    <div class="cost-sub" id="cost-windows">0 context-windows extended</div>
+    <div class="stat-grid" style="margin-top:12px">
+      <span class="label" title="The model pi is currently using — its pricing drives the cost figure.">Model</span><span class="value" id="md-name">—</span>
+      <span class="label" title="The provider serving the model.">Provider</span><span class="value" id="md-provider">—</span>
+      <span class="label" title="USD per input token, from the model's pricing.">Input Rate</span><span class="value" id="md-input">—</span>
+      <span class="label" title="USD per output token, from the model's pricing.">Output Rate</span><span class="value" id="md-output">—</span>
+    </div>
+  </div>
+  <div class="card">
+    <h2>Crew / Agents</h2>
+    <div class="stat-grid">
+      <span class="label">Active Agents</span><span class="value" id="cr-agents">0</span>
+      <span class="label">Current Turn</span><span class="value" id="cr-turn">0</span>
+      <span class="label">Status</span><span class="value" id="cr-status">idle</span>
+    </div>
+  </div>
+  <div class="card legend">
+    <h2>What these numbers mean</h2>
+    <ul class="legend-list">
+      <li><b>Tokens saved</b> — conversation space this extension has freed up for you (it compacted old text into short summaries).</li>
+      <li><b>Tokens stored</b> — how much "memory" (compact summaries) the extension is currently holding for this repo.</li>
+      <li><b>Injected</b> — times old context was automatically pasted back in because it was relevant to your current task.</li>
+      <li><b>Recall relevance</b> — of those, how often the recalled context was actually on-topic.</li>
+      <li><b>Storage dedup</b> — how often new content matched something already saved, so a duplicate copy was skipped (saves space).</li>
+      <li><b>Data safety</b> — every compacted region is kept verbatim (compressed). Nothing is permanently deleted; you can restore any of it.</li>
+    </ul>
+    <p class="legend-note">Hover any label above for a quick explanation.</p>
+  </div>
+</div>
+
+<div class="events">
+  <h2>Event Stream</h2>
+  <div class="events-wrap" id="events"><div class="empty">connecting…</div></div>
+</div>
+
+<h2 style="margin-top:24px;font-size:13px;color:#8b949e;text-transform:uppercase;letter-spacing:.5px">All Repositories</h2>
+<table class="repos">
+  <thead>
+    <tr>
+      <th>Repo</th><th>Model</th>
+      <th style="text-align:right">Checkpoints</th>
+      <th style="text-align:right">Tokens Saved</th>
+      <th style="text-align:right">Retained</th>
+      <th style="text-align:right">Last Compacted</th>
+    </tr>
+  </thead>
+  <tbody id="cur-rows"><tr><td colspan="6" class="repo-none">loading…</td></tr></tbody>
+</table>
+<div class="updated" id="cur-updated"></div>
+
+<div class="updated" id="updated"></div>
+</div><!-- /panel-current -->
+
+`;
+}

package/dist/extensions/dashboard-server/html/head-open.js

@@ -0,0 +1,16 @@
+/**
+ * Document opening: DOCTYPE, <html>, <head> opening + meta/title.
+ *
+ * Styles are injected between this fragment and `bodyOpen()` by the
+ * `dashboardHtml` composer in `../html-template.ts`.
+ */
+export function headOpen() {
+    return `<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>mega-compact dashboard</title>
+`;
+    // trailing \n preserves the original blank-line separation to <style> in the composer
+}

package/dist/extensions/dashboard-server/html/high-score-tab.js

@@ -0,0 +1,25 @@
+/**
+ * "High Score" tab — FUTURE / PLACEHOLDER.
+ *
+ * Reserved for the next project milestone: a per-repo + global
+ * compaction-savings leaderboard ("high score" — most tokens saved, best
+ * recall relevance, longest context-window extension streak, etc.).
+ *
+ * Ships as a visible-but-inert stub so the tab nav, routing, and styles are
+ * in place; the data model + scoring logic land in a follow-up. The composer
+ * in `../html-template.ts` includes this panel; `script.ts` wires
+ * `highscore` into the tab-switch `panels` map so clicking the tab reveals
+ * this placeholder.
+ */
+export function highScoreTab() {
+    return `<!-- High Score (FUTURE / placeholder for the next project milestone) -->
+<div class="tab-panel" id="panel-highscore">
+  <div class="placeholder">
+    <div class="em">🏆</div>
+    <h2>High Score</h2>
+    <p>A compaction-savings leaderboard is coming in the next project milestone — per-repo and global rankings for tokens saved, recall relevance, and context-window extension streaks. Stay tuned.</p>
+  </div>
+</div>
+
+`;
+}

package/dist/extensions/dashboard-server/html/repo-detail-modal.js

@@ -0,0 +1,26 @@
+/**
+ * Per-repo detail modal — fixed overlay opened when a repo row is clicked.
+ *
+ * Position-independent in the DOM; the modal is toggled via the `.open`
+ * class by `script.ts`. Sits between the tab panels and the script so it
+ * overlays whichever panel is active.
+ */
+export function repoDetailModal() {
+    return `<!-- Per-repo detail modal -->
+<div class="repo-detail" id="repo-detail">
+  <div class="repo-detail-box">
+    <h2><span id="rd-name">Repo</span><button class="repo-close" id="rd-close" title="Close">×</button></h2>
+    <div class="repo-path" id="rd-path"></div>
+    <div class="stat-grid">
+      <span class="label">Model</span><span class="value" id="rd-model">—</span>
+      <span class="label">Checkpoints</span><span class="value" id="rd-cp">0</span>
+      <span class="label">Tokens Saved</span><span class="value" id="rd-saved">0</span>
+      <span class="label">Compressed-Original</span><span class="value" id="rd-bytes">0 B</span>
+      <span class="label">Last Compacted</span><span class="value" id="rd-when">—</span>
+      <span class="label">Provider</span><span class="value" id="rd-provider">—</span>
+    </div>
+  </div>
+</div>
+
+`;
+}