pi-mega-compact 0.6.7 → 0.7.0

package/README.md

@@ -6,7 +6,7 @@ sessions into a **local SQLite store** and offers **deduped inline recall** —
 running **locally inside the extension**, with **no remote MCP server** and
 **zero network calls at runtime** (PREVENT-PI-004).
 
-> **Current version:** `v0.6.2` — storage backend is **`node:sqlite`**
+> **Current version:** `v0.6.9` — storage backend is **`node:sqlite`**
 > (`DatabaseSync`, a Node ≥22.13 built-in), replacing the old `better-sqlite3`
 > native addon and the per-session gzipped JSON checkpoint files. **Zero native
 > build step, fully local, zero network at runtime.** Legacy
@@ -23,10 +23,12 @@ running **locally inside the extension**, with **no remote MCP server** and
 pi's context window is finite. When a session gets long — especially a team run
 with sub-agents — pi-mega-compact keeps it going without overflowing:
 
-1. **Watches one signal.** A single live `pressure = currentTokens / thresholdTokens`
+1. **Watches one signal.** A single live `pressure = currentTokens / effectiveThreshold`
    drives everything — the tier label, how aggressively the live trim drops
-   context, and how often durable memory is reviewed. As context fills, the whole
-   system reacts together; as it's relieved, it backs off.
+   context, and how often durable memory is reviewed. `effectiveThreshold` is
+   `tierPct × contextWindow` (a **% of the model's context window**), so the
+   trim fires below pi's native ~80% auto-compaction for any model size. As
+   context fills, the whole system reacts together; as it's relieved, it backs off.
 2. **Compacts in two layers.** On every LLM call it returns a **live, compacted
    view** (the model sees a summary + recent anchor, non-destructively). And it
    persists a durable **checkpoint** — and, at each agent settle during a team
@@ -233,7 +235,7 @@ The commands (slash commands inside pi):
 | `/mega-help` | Explain the toolbar widget terms (live tier, gate, dedup, tokens saved). |
 | `/mega-compat-check` | Detect extension conflicts (duplicate commands / overlapping handlers) across installed pi extensions. |
 
-The **tier** you see in the toolbar and dashboard is a *live pressure band* (`low` → `medium` → `high` → `ultra` → `mega`) that climbs automatically as your context window fills and falls back as it's relieved — it is driven by `currentTokens / thresholdTokens`, not a manual setting. The base compaction *threshold* (token budget) is still chosen by the `MEGACOMPACT_TIER` env var at startup (`low`/`medium`/`high`/`ultra`/`mega`, default `low`); `/mega-tier` was removed in v0.6.0. Higher pressure also deepens the live trim and reviews durable memory more often — the whole system reacts as one.
+The **tier** you see in the toolbar and dashboard is a *live pressure band* (`low` → `medium` → `high` → `ultra` → `mega`) that climbs automatically as your context window fills and falls back as it's relieved — it is driven by `currentTokens / effectiveThreshold`, not a manual setting. The base compaction *threshold* is set by `MEGACOMPACT_TIER` at startup as a **% of the model context window** (`low` 50% · `medium` 60% · `high` 70% · `ultra` 70% · `mega` 75%; default `low`) — the fire point is `tierPct × contextWindow`, so it always lands below pi's native ~80% auto-compaction (any model size). The old static token amounts (50k/100k/200k/1M/10M) are now only the boot fallback used before the first context event reports a window. `/mega-tier` was removed in v0.6.0. Higher pressure also deepens the live trim and reviews durable memory more often — the whole system reacts as one.
 | `/mega-dashboard` | Start the **localhost-only** live dashboard and open it in a browser (token gauge, store stats, live event stream, per-repo + cross-repo drift). |
 | `/mega-dashboard-status` | Report dashboard server status. |
 | `/mega-dashboard-stop` | Stop the dashboard server. |
@@ -266,9 +268,9 @@ before starting pi.
 
 | Variable | Default | Meaning |
 |---|---|---|
-| `MEGACOMPACT_FAST_GATE_PCT` | `70` | Context-usage % that arms the auto-trigger. |
-| `MEGACOMPACT_TIER` | `low` | Named trigger preset — sets the token threshold. `low`(50k) `medium`(100k) `high`(200k) `ultra`(1M) `mega`(10M). |
-| `MEGACOMPACT_THRESHOLD_TOKENS` | _(tier default)_ | Explicit token budget confirming compaction. Overrides `MEGACOMPACT_TIER` when set. |
+| `MEGACOMPACT_FAST_GATE_PCT` | `70` | Context-usage % that arms the auto-trigger. Defaults to the tier's % of window (`tierPct*100`): low 50 · med 60 · high 70 · ultra 70 · mega 75. Override raises the arming floor. |
+| `MEGACOMPACT_TIER` | `low` | Named trigger preset — sets the compaction threshold as a **% of the model context window**: `low`(50%) `medium`(60%) `high`(70%) `ultra`(70%) `mega`(75%). Fire point = `tierPct × contextWindow`, so it always fires below pi's native ~80% auto-compaction (any model size). The old static token amounts (50k/100k/200k/1M/10M) are now only the **boot fallback** used before the first context event reports a window. Default `low`. |
+| `MEGACOMPACT_THRESHOLD_TOKENS` | _(tier default)_ | Explicit **absolute** token budget (the `custom` tier). Overrides `MEGACOMPACT_TIER` when set and is **never percent-scaled** — use this to pin an exact token fire point regardless of model window. |
 | `MEGACOMPACT_ANCHOR_USER_MESSAGES` | `3` | Never drop the most recent N user messages (anchor floor). |
 | `MEGACOMPACT_PRESERVE_RECENT` | `4` | Preserve the most recent N messages verbatim. |
 | `MEGACOMPACT_AUTO` | `true` | Enable the auto-trigger. |

package/dist/extensions/dashboard-server.js

@@ -83,7 +83,52 @@ function readIndex() {
             inputRate: r.input_rate ?? null,
             outputRate: r.output_rate ?? null,
             lastSeen: Number(r.last_seen ?? 0),
+            // Defaults — enriched below from each repo's own store.
+            tokensKept: 0,
+            tokensDropped: 0,
+            sessions: 0,
+            contextWindow: null,
+            maxTokens: null,
+            reasoning: null,
         }));
+        // Enrich each repo with per-store token + model detail read directly via
+        // node:sqlite (same zero-dependency invariant as readIndex; no store graph
+        // import). Best-effort: a missing/corrupt store degrades to the defaults
+        // above so the dashboard never fails to render.
+        for (const repo of mapped) {
+            try {
+                const storePath = join(repo.stateDir, "sqlite.db");
+                if (existsSync(storePath)) {
+                    const sdb = new DatabaseSync(storePath, { readOnly: true });
+                    try {
+                        const tok = sdb
+                            .prepare(`SELECT COALESCE(SUM(token_estimate),0) AS kept,
+                        COALESCE(SUM(original_token_estimate),0) AS dropped,
+                        COUNT(DISTINCT session_id) AS sess
+                 FROM context_chunks WHERE dedup_status != 'removed'`)
+                            .get();
+                        repo.tokensKept = Number(tok.kept ?? 0);
+                        repo.tokensDropped = Number(tok.dropped ?? 0);
+                        repo.sessions = Number(tok.sess ?? 0);
+                        const mrow = sdb
+                            .prepare(`SELECT context_window, max_tokens, reasoning
+                 FROM model_snapshots ORDER BY captured_at DESC LIMIT 1`)
+                            .get();
+                        if (mrow) {
+                            repo.contextWindow = Number(mrow.context_window ?? 0) || null;
+                            repo.maxTokens = Number(mrow.max_tokens ?? 0) || null;
+                            repo.reasoning = Number(mrow.reasoning ?? 0) === 1;
+                        }
+                    }
+                    finally {
+                        sdb.close();
+                    }
+                }
+            }
+            catch {
+                /* best-effort — keep the defaults */
+            }
+        }
         // Defensive display hygiene (belt-and-suspenders — the real fix is that
         // tests now isolate via MEGACOMPACT_INDEX_DIR): drop transient test/temp
         // paths that should never have been real repos, and collapse duplicate
@@ -139,14 +184,15 @@ function readSnapshot(snapshotPath) {
             tier: "unknown",
             presetTier: "unknown",
             pressure: 0,
-            config: { fastGatePct: 80, thresholdTokens: 100_000, anchorUserMessages: 1, preserveRecent: 2, auto: true, autoInlineK: 3 },
+            config: { fastGatePct: 80, thresholdTokens: 100_000, tierPct: null, effectiveThresholdPct: null, anchorUserMessages: 1, preserveRecent: 2, auto: true, autoInlineK: 3 },
             session: { id: null, state: null, persistedThisSession: false, lastCheckpointId: null, lastCompactedFrom: 0 },
             context: { tokens: null, percent: null, contextWindow: 0 },
-            trigger: { armed: false, ready: false, currentTokens: null, thresholdTokens: 100_000, fastGatePct: 80 },
+            trigger: { armed: false, ready: false, currentTokens: null, thresholdTokens: 100_000, fastGatePct: 80, tierPct: null, effectiveThresholdPct: null },
             store: { checkpointCount: 0, totalTokenEstimate: 0, originalTokens: 0, tokensSaved: 0, injectedCount: 0, dedupHitRate: 0, storageDedupRate: 0, dedupCollapsed: 0 },
             crew: { activeAgents: 0, currentTurn: 0 },
             repo: { checkpointCount: 0, totalTokenEstimate: 0, originalTokens: 0, tokensSaved: 0, sessionCount: 0, dedupAttempts: 0, dedupCollapsed: 0, storageDedupRate: 0 },
             integrity: { regionsRetained: 0, compressedOriginalBytes: 0, duplicatesCollapsed: 0, bytesPermanentlyDeleted: 0 },
+            compression: { session: { tokensIn: 0, tokensOut: 0, tokensFreed: 0, compressionPct: 0, dedupPct: 0 }, repo: { tokensIn: 0, tokensOut: 0, tokensFreed: 0, compressionPct: 0, dedupPct: 0 } },
             model: undefined,
         };
     }
@@ -332,9 +378,9 @@ function dashboardHtml(tierName) {
     <div class="conf-grid">
       <span class="label" title="Live pressure band — climbs low→mega as context fills the window.">Tier (live)</span><span class="value" id="cf-tier">${tierName}</span>
       <span class="label" title="The env-resolved base compaction preset (low/medium/high/ultra/mega) that set the token threshold.">Preset</span><span class="value" id="cf-preset">—</span>
-      <span class="label" title="Live pressure = currentTokens / thresholdTokens (0–100%).">Pressure</span><span class="value" id="cf-pressure">—</span>
-      <span class="label">Threshold</span><span class="value" id="cf-threshold">—</span>
-      <span class="label">Fast Gate</span><span class="value" id="cf-gate">—</span>
+      <span class="label" title="Live pressure = currentTokens / threshold — % of the model context window (threshold fires at the tier's % of window).">Pressure</span><span class="value" id="cf-pressure">—</span>
+      <span class="label" title="Compaction threshold = tierPct × model context window — mega-compact trims BELOW pi's native ~80% auto-compact for any model size.">Threshold</span><span class="value" id="cf-threshold">—</span>
+      <span class="label" title="Fast-gate arming floor — the live trim arms once context passes this % of the window.">Fast Gate</span><span class="value" id="cf-gate">—</span>
       <span class="label">Auto</span><span class="value" id="cf-auto">—</span>
       <span class="label">Anchor</span><span class="value" id="cf-anchor">—</span>
     </div>
@@ -443,15 +489,23 @@ function dashboardHtml(tierName) {
     <thead>
       <tr>
         <th>Model</th><th>Provider</th>
-        <th style="text-align:right">Repos</th>
+        <th style="text-align:right" title="Tokens dropped from context by compaction (the input reclaimed)">Tokens In</th>
+        <th style="text-align:right" title="Tokens kept as compacted summaries still in context (the output retained)">Tokens Out</th>
+        <th style="text-align:right">Freed</th>
+        <th style="text-align:right" title="Model context window (max input tokens the model accepts)">Ctx Window</th>
+        <th style="text-align:right" title="Model max output tokens per turn">Max Out</th>
+        <th style="text-align:right" title="Reasoning-capable model">Reas.</th>
+        <th style="text-align:right" title="Distinct sessions with at least one checkpoint">Sessions</th>
         <th style="text-align:right">Checkpoints</th>
-        <th style="text-align:right">Tokens Saved</th>
+        <th style="text-align:right" title="USD per input token">In $/tok</th>
+        <th style="text-align:right" title="USD per output token">Out $/tok</th>
         <th style="text-align:right">$ Saved</th>
         <th style="text-align:right">Last Used</th>
       </tr>
     </thead>
-    <tbody id="bm-rows"><tr><td colspan="7" class="repo-none">loading…</td></tr></tbody>
+    <tbody id="bm-rows"><tr><td colspan="14" class="repo-none">loading…</td></tr></tbody>
   </table>
+  <p class="legend-note" style="margin-top:8px">Tokens In = Σ original region tokens dropped by compaction. Tokens Out = Σ compacted summary tokens still retained in context. Freed = Tokens In − Tokens Out (net context reclaimed). Ctx Window / Max Out / Reas. come from the latest captured model snapshot for each repo.</p>
 
   <div class="updated" id="sm-updated"></div>
 </div>
@@ -547,7 +601,17 @@ function dashboardHtml(tierName) {
     document.getElementById('cf-tier').textContent = d.tier + ' (live)';
     document.getElementById('cf-preset').textContent = d.presetTier;
     document.getElementById('cf-pressure').textContent = Math.round((d.pressure || 0) * 100) + '%';
-    document.getElementById('cf-threshold').textContent = d.config.thresholdTokens.toLocaleString();
+    // (b) Threshold: show the effective token threshold AND the % of the model
+    // context window it represents (percentage-based tiers). d.config.tierPct
+    // is present on the live snapshot written by the runtime (Phase-1/2a).
+    var cfgPct = d.config.tierPct;
+    var cw = d.context.contextWindow || 0;
+    var thresholdTxt = d.config.thresholdTokens.toLocaleString();
+    if (cfgPct != null && cw > 0) {
+      thresholdTxt += ' (' + Math.round(cfgPct * 100) + '% of ' + cw.toLocaleString() + ')';
+    }
+    document.getElementById('cf-threshold').textContent = thresholdTxt;
+    // (c) Fast Gate: arming floor — live trim arms once context passes this %.
     document.getElementById('cf-gate').textContent = d.config.fastGatePct + '%';
     document.getElementById('cf-auto').textContent = d.config.auto ? 'enabled' : 'disabled';
     document.getElementById('cf-anchor').textContent = d.config.anchorUserMessages;
@@ -673,33 +737,67 @@ function dashboardHtml(tierName) {
     var rows = document.getElementById('bm-rows');
     if (!rows) return;
     if (!repos || !repos.length) {
-      rows.innerHTML = '<tr><td colspan="7" class="repo-none">No repositories registered yet.</td></tr>';
+      rows.innerHTML = '<tr><td colspan="14" class="repo-none">No repositories registered yet.</td></tr>';
       return;
     }
     var groups = {};
     for (var i = 0; i < repos.length; i++) {
       var r = repos[i];
       var key = (r.modelName && String(r.modelName).trim()) || '(unknown)';
-      if (!groups[key]) groups[key] = { model: key, provider: r.providerName || r.provider || '—', repos: 0, checkpoints: 0, tokensSaved: 0, usd: 0, lastAt: 0, rates: [] };
+      if (!groups[key]) groups[key] = {
+        model: key, provider: r.providerName || r.provider || '—', repos: 0, checkpoints: 0,
+        tokensSaved: 0, tokensIn: 0, tokensOut: 0, sessions: 0, usd: 0, lastAt: 0,
+        inRates: [], outRates: [], ctxWindows: [], maxTokens: [], reasoning: null,
+      };
       var g = groups[key];
       g.repos++;
       g.checkpoints += (r.checkpointCount || 0);
       g.tokensSaved += (r.tokensSaved || 0);
-      if (r.inputRate) { g.usd += (r.tokensSaved || 0) * r.inputRate; g.rates.push(r.inputRate); }
+      g.tokensIn += (r.tokensDropped || 0);
+      g.tokensOut += (r.tokensKept || 0);
+      g.sessions += (r.sessions || 0);
+      if (r.inputRate) { g.usd += (r.tokensSaved || 0) * r.inputRate; g.inRates.push(r.inputRate); }
+      if (r.outputRate) g.outRates.push(r.outputRate);
+      if (r.contextWindow) g.ctxWindows.push(r.contextWindow);
+      if (r.maxTokens) g.maxTokens.push(r.maxTokens);
+      if (r.reasoning != null) g.reasoning = r.reasoning;
       if (r.lastCompactedAt && r.lastCompactedAt > g.lastAt) g.lastAt = r.lastCompactedAt;
     }
     var arr = [];
     for (var k in groups) { if (Object.prototype.hasOwnProperty.call(groups, k)) arr.push(groups[k]); }
     arr.sort(function(a, b) { return b.tokensSaved - a.tokensSaved; });
+    // Helper: a set of numeric samples collapses to a single value when all
+    // repos in the group agree, otherwise shows the range (min–max) so the
+    // user can see mixed-config model groups at a glance.
+    function collapseNum(samples) {
+      if (!samples || !samples.length) return '—';
+      var lo = Math.min.apply(null, samples), hi = Math.max.apply(null, samples);
+      return lo === hi ? lo.toLocaleString() : lo.toLocaleString() + '–' + hi.toLocaleString();
+    }
+    function collapseRate(samples) {
+      if (!samples || !samples.length) return '—';
+      var lo = Math.min.apply(null, samples), hi = Math.max.apply(null, samples);
+      var fmt = function(v) { return '$' + v.toFixed(6); };
+      return lo === hi ? fmt(lo) : fmt(lo) + '–' + fmt(hi);
+    }
     rows.innerHTML = arr.map(function(g) {
+      var freed = (g.tokensIn || 0) - (g.tokensOut || 0);
       var usd = g.usd > 0 ? '$' + g.usd.toFixed(4) : '—';
       var when = g.lastAt ? new Date(g.lastAt).toLocaleString() : '—';
+      var reas = g.reasoning == null ? '—' : (g.reasoning ? 'yes' : 'no');
       return '<tr>' +
         '<td><span class="repo-model">' + sanitize(g.model) + '</span></td>' +
         '<td>' + sanitize(g.provider) + '</td>' +
-        '<td class="num">' + g.repos.toLocaleString() + '</td>' +
+        '<td class="num">' + (g.tokensIn || 0).toLocaleString() + '</td>' +
+        '<td class="num">' + (g.tokensOut || 0).toLocaleString() + '</td>' +
+        '<td class="num">' + freed.toLocaleString() + '</td>' +
+        '<td class="num">' + collapseNum(g.ctxWindows) + '</td>' +
+        '<td class="num">' + collapseNum(g.maxTokens) + '</td>' +
+        '<td class="num">' + reas + '</td>' +
+        '<td class="num">' + g.sessions.toLocaleString() + '</td>' +
         '<td class="num">' + g.checkpoints.toLocaleString() + '</td>' +
-        '<td class="num">' + g.tokensSaved.toLocaleString() + '</td>' +
+        '<td class="num">' + collapseRate(g.inRates) + '</td>' +
+        '<td class="num">' + collapseRate(g.outRates) + '</td>' +
         '<td class="num">' + sanitize(usd) + '</td>' +
         '<td class="num">' + sanitize(when) + '</td>' +
       '</tr>';

package/dist/extensions/mega-commands.js

@@ -116,9 +116,20 @@ export function registerCommands(pi, runtime, config) {
             }
             catch { /* non-fatal */ }
             const crossRepoStr = `${crossRepoInjections} cross-repo injections recorded · ${repoCount} repos indexed`;
+            // Effective compaction threshold = tierPct × model context window (kept
+            // BELOW pi's native ~80% auto-compact for any model size). Falls back to
+            // the boot token value when the window is unknown (custom tier / pre-
+            // model-select). Display matches the dashboard's percentage-based view.
+            const effThreshold = config.tierPct != null && ctxWindow > 0
+                ? Math.round(config.tierPct * ctxWindow)
+                : config.thresholdTokens;
+            const winStr = ctxWindow > 0
+                ? (ctxWindow >= 1_000_000 ? `${Math.round(ctxWindow / 1_000_000)}M` : `${Math.round(ctxWindow / 1_000)}k`)
+                : "?";
+            const tierPctStr = config.tierPct != null ? `${Math.round(config.tierPct * 100)}%` : "n/a";
             ctx.ui.notify(`[mega-compact] pct=${pct} tokens=${tokens} tier=${runtime.pressureBand} (live) preset=${config.tier} ` +
                 `pressure=${Math.round(runtime.pressure * 100)}% fastGate=${config.fastGatePct}% ` +
-                `threshold=${config.thresholdTokens} auto=${config.auto} autoInline=${config.autoInline}\n` +
+                `threshold=${effThreshold.toLocaleString()} (${tierPctStr} of ${winStr} window) tierPct=${config.tierPct != null ? config.tierPct.toFixed(2) : "n/a"} auto=${config.auto} autoInline=${config.autoInline}\n` +
                 `[mega-compact] store: ${st.checkpointCount} chkpt · ` +
                 `${st.totalTokenEstimate} tok · last=${st.lastCheckpointId ?? "—"} · ` +
                 `injected=${st.injectedCount} · dedup=${(st.dedupHitRate * 100).toFixed(0)}%\n` +