pi-mega-compact 0.8.14 → 0.8.16

package/README.md

@@ -1,503 +1,91 @@
 # pi-mega-compact
 
-A **layered, local, vector-backed context compressor** for the
-[pi coding agent](https://github.com/earendil-works/pi). It compacts long
-sessions into a **local SQLite store** and offers **deduped inline recall** — all
-running **locally inside the extension**, with **no remote MCP server** and
-**zero network calls at runtime** (PREVENT-PI-004).
-
-> **Status - v0.7.9.** Storage uses the built-in `node:sqlite` backend
-> (`DatabaseSync`, Node >=22.13): zero native build step, fully local, and zero
-> network at runtime. See [RELEASE_NOTES.md](RELEASE_NOTES.md) for the full
-> changelog.
-
----
+A local context compressor for the [pi coding agent](https://github.com/earendil-works/pi). Keeps long sessions running without overflowing the context window. All on your machine — no cloud, no API calls, no telemetry.
 
 ## Features
 
-- **Local & private** - everything stays on your disk. No telemetry, no API key, no MCP server, no cloud. The only network surface is an optional localhost dashboard you open yourself.
-- **Two-layer compaction** - a non-destructive live summary every LLM call, plus durable checkpoints that relieve context mid-run (not just at the end).
-- **Vector store + dedup** - each checkpoint is embedded and stored locally; an L0->L2 + RAPTOR cascade collapses duplicate work so storage and recall stay lean.
-- **Automatic recall** - the most relevant checkpoints are re-inlined on resume or branch switch; cross-repo memory-RAG augments a thin store with decisions from other repos.
-- **Live dashboard** - a localhost-only view of token usage, store stats, savings, per-repo activity, and an **Active Repos tab** that shows every currently-open session (last 30 minutes) side by side — so running multiple sessions at once is visible in one place.
-- **Database maintenance** - `/mega-db-*` commands plus best-effort auto-maintenance on session start.
-
-## Table of contents
-
-- [Overview](#overview)
-- [How it works](#how-it-works)
-- [Installation](#installation)
-- [Usage](#usage)
-- [Configuration](#configuration)
-- [Dashboard](#dashboard)
-- [Architecture](#architecture)
-- [Development](#development)
-- [Testing & bug reports](#testing--bug-reports)
-- [Acknowledgements](#acknowledgements)
-- [License](#license)
-
-## Overview
-
-pi's context window is finite. When a session gets long — especially a team run
-with sub-agents — pi-mega-compact keeps it going without overflowing:
-
-1. **Watches one signal.** A single live `pressure = currentTokens / effectiveThreshold`
-   drives everything — the tier label, how aggressively the live trim drops
-   context, and how often durable memory is reviewed. `effectiveThreshold` is
-   `tierPct × contextWindow` (a **% of the model's context window**), so the
-   trim fires below pi's native ~80% auto-compaction for any model size. As
-   context fills, the whole system reacts together; as it's relieved, it backs off.
-2. **Compacts in two layers.** On every LLM call it returns a **live, compacted
-   view** (the model sees a summary + recent anchor, non-destructively). And it
-   persists a durable **checkpoint** — and, at each agent settle during a team
-   run, fires pi's **native durable trim** so the on-disk transcript is actually
-   truncated (context relieves mid-run, and resume reloads the trimmed transcript
-   instead of a 150k window).
-3. **Stores** each checkpoint in a **local vector database** (SQLite) with an
-   embedding, so similar regions are found later and **duplicate work is never
-   stored twice**.
-4. **Recalls** the right context automatically — same-repo checkpoints on resume,
-   plus **cross-repo memory-RAG**: decisions you saved in one repo are inlined as
-   context when you start a session in another.
-
-Everything lives on **your disk**. No telemetry, no API, no MCP server, no cloud.
-The only optional network surface is a **user-triggered localhost dashboard** you
-open yourself.
-
-### Why "mega"?
-
-The compaction pipeline is a **Trident** — three deterministic stages
-(supersede → collapse → cluster) that run over your conversation before anything
-is persisted. The checkpoint it produces is small (a summary + key decisions +
-next steps + files touched), so the same session that would otherwise overflow
-its window keeps going on a fraction of the tokens. On top of the Trident, a
-single pressure signal orchestrates the live trim, the durable trim, and memory
-review as one coherent system rather than four independent triggers.
-
----
-
-## How it works
-
-```
-Layer 5  Recall / Inline      ONE local vector store → 3 entry points, 1 dedup engine
-Layer 4  Persist / Checkpoint  compactSession()  → embed + store in SQLite (chkpt_xxx)
-Layer 3  Cluster (vectorize)  local vector index → semantic dedup + recall
-Layer 2  Collapse (summarize)  summarizeMessages() heuristic + agent summary on /mega-compact
-Layer 1  Supersede (prune)     drop obsolete file-reads / superseded turns (zero cost)
-─────────────────────────────────────────────────────────────────────────
-Trigger   context → token fast-gate → autoCompactCheck → live trim (per call)
-Durable    agent_end (idle + over threshold) → ctx.compact() → session_before_compact
-          supplies the summary; pi truncates the transcript (relieves context)
-Live      context handler returns { messages:[summary, …recent] } — model sees a
-          compacted window every LLM call; the on-disk transcript is untouched
-Marker    insert compact-marker; dedupe so repeated triggers cost ~0 tokens
-```
-
-**One store, three ways to read it back — one dedup engine:**
-
-| Entry point | Trigger | Behavior |
-| --- | --- | --- |
-| **Auto-inline** (Layer 5) | `session_start` / `session_tree` | Resume → `recallAndInline(source:"resume")` prepends the most relevant checkpoints, deduped against current context. |
-| **On-demand recall** | `/mega-recall [query]` | Semantic search the store, dedupe, and inline the top-K. |
-| **Dedup sentinel** | every compact | A lightweight `mega-compact-marker` entry lets auto-inline and recall skip re-injecting / re-vectorizing already-present regions. |
-
-**The dedup cascade** (shared across all entry points) collapses redundant work
-so storage and recall stay lean:
-
-- **L0 — exact:** SHA-256 content hash + region hash + summary hash. Identical (or
-  whitespace/casing-normalized) regions collapse to one row.
-- **L1 — near-dup:** MinHash signatures + LSH bucketing + trigram verification
-  catch one-word rewordings that L0 misses.
-- **L2 — semantic:** cosine over the embedding collapses paraphrases; MMR
-  diversifies retrieval so a cluster of near-hits yields distinct results.
-- **RAPTOR — pre-compression tree** (shadow mode by default): a hierarchical
-  summary tree over checkpoints, built + logged but not served until promoted.
-
-Every tier is gated by its own feature flag (see [Configuration](#configuration)).
-A tier can be put in `MARK_ONLY` (record the decision, don't collapse) as a safe
-partial-rollout or auto-degrade state.
-
-### Embedding (two modes, both local)
+- **Auto-compaction** — watches context pressure and compacts in the background before you hit the ceiling
+- **Two-layer compaction** — live trim on every LLM call (model sees a smaller window) + durable checkpoints persisted to SQLite
+- **Semantic dedup** — three-stage pipeline (exact hash → MinHash/LSH → cosine) collapses redundant work so nothing is stored twice
+- **Cross-repo recall** — decisions saved in one repo surface when you start a session in another
+- **Durable memory** — auto-reviews conversation every 10 turns, writes decisions/facts/preferences to SQLite, injects them as RAG context on recall
+- **Fully local** — SQLite + trigram embedder by default. Bring your own localhost embedder (ONNX, Ollama, TEI) for better semantic matching
+- **Team-run aware** — fires native durable trim at agent settle during sub-agent runs, so context relieves mid-run not just at the end
+- **Dashboard** — optional localhost-only live dashboard with token gauge, store stats, and event stream
 
-The default embedder is **`TrigramEmbedder`** — a deterministic hashed trigram
-bag (512-dim, L2-normalized), **zero dependencies, instant, fully offline**. It
-is heuristic-strength, which is the right bar for "inline the right checkpoint,"
-not production RAG.
-
-**Optional: bring-your-own (BYO) localhost embedder.** Set
-`MEGACOMPACT_EMBEDDING_URL` to a **localhost/127.0.0.1** endpoint you run
-yourself (local ONNX/TEI/llamafile/Ollama-embeddings). The extension talks to it
-from `src/httpEmbedder.ts` (loopback-only — a remote host is rejected at config
-time, preserving PREVENT-PI-004). Compacted content never leaves the machine and
-no model ships with the extension. See `src/httpEmbedder.ts` for the
-OpenAI-style contract and the `MEGACOMPACT_EMBEDDING_KEY` / `MEGACOMPACT_EMBEDDING_HEADERS`
-/ `MEGACOMPACT_EMBEDDING_DIM` options.
-
-> **Note on MiniLM:** a `MEGACOMPACT_MINILM` flag exists in `src/config/dedup.ts`
-> but defaults to **off**, and the MiniLM (all-MiniLM-L6-v2) ONNX embedder was
-> prototyped then deliberately **not shipped** (async-vs-sync conflict with the
-> synchronous VectorStore, second native dep, no free semantic win without a
-> network call). The `Embedder` interface remains the seam — inject a local
-> embedder (e.g. via your own `MEGACOMPACT_EMBEDDING_URL`) instead.
-
----
-
-## Installation
-
-> **Full step-by-step guide** (pi + OpenClaw + every command + troubleshooting):
-> [`docs/INSTALL_AND_USAGE.md`](docs/INSTALL_AND_USAGE.md).
-
-### Requirements
-
-- **Node >= 22.13** (the synchronous `node:sqlite` backend requires it; see
-  `engines.node`). No native module is compiled — the store is a Node built-in.
-- No network call and no API key are needed at runtime (PREVENT-PI-004).
-- A pi coding agent install with package support (`pi install` / `pi update
-  --extensions`). npm-installed packages are auto-discovered via the package's
-  `pi` manifest entry; local checkouts load from `~/.pi/agent/extensions/`.
-
-### Install from npm (recommended)
-
-pi installs extensions as **packages**. `pi install npm:<pkg>` writes an
-`npm:` source into your pi config's `packages` array; pi then auto-discovers the
-extension from the package's own `"pi": { "extensions": [...] }` manifest entry
-— **no manual `settings.json` edit is needed**.
+## Install
 
 ```bash
-pi install npm:pi-mega-compact        # first time: adds to packages + installs
-pi update --extensions                # thereafter: pulls the latest published version
+pi install npm:pi-mega-compact
 ```
 
-`pi update --extensions` refreshes every `npm:` entry in `packages` (including on
-other devices that share this config). The package ships both the TypeScript
-source (which pi loads directly) and the compiled `dist/`, so nothing else needs
-building.
-
-> **Tip — keep the spec unpinned.** Use `npm:pi-mega-compact`, not
-> `npm:pi-mega-compact@0.5.1`. Version-pinned specs are *skipped* by
-> `pi update --extensions`, so a pin would freeze you on that release. The
-> installed version is always visible in the toolbar widget (`⚡ <tier> vX.Y.Z`)
-> and via `/mega-status`.
-
-> **From a git checkout (development only).** To hack on the extension, clone and
-> build locally, then symlink it into pi's extensions dir — but this bypasses the
-> package manager, so it is NOT updated by `pi update --extensions`. Convert to the
-> npm package (above) before shipping. The bundled `./install.sh` helper does the
-> symlink + config edit (needs `jq`).
->
-> ```bash
-> git clone https://github.com/TheArchitectit/pi-mega-compact.git \
->   ~/.pi/agent/extensions/pi-mega-compact
-> cd ~/.pi/agent/extensions/pi-mega-compact
-> npm install && npm run build
-> ```
+That's it. `pi update --extensions` pulls updates going forward.
 
-> **No tarballs — ever.** Distribution and updates go through `npm publish` +
-> `pi update --extensions` **only**. Never build or rely on a `.tgz` (`npm pack`):
-> a tarball bypasses pi's package manager and does not propagate to other devices.
-> To validate a real install, bump the version, `npm publish`, then
-> `pi update --extensions` on the device. (`.gitignore` rejects `*.tgz` so one can't
-> be committed by accident.)
-
-### Storage
-
-pi-mega-compact uses a dual local backend — **zero network, no native build step**:
-
-- **`node:sqlite`** (`DatabaseSync`, Node ≥22.13 built-in) — the synchronous source of truth for checkpoints, session state, and the dedup index. No dependency, no install script, survives pi's `install-scripts` block.
-- **PGlite + `@electric-sql/pglite-pgvector`** (WASM Postgres + HNSW `vector_cosine_ops`) — an optional, best-effort async vector index for **cross-repo recall** at `~/.pi/mega-compact-vector`. It holds both checkpoint embeddings and durable-memory embeddings, so decisions saved in one repo are findable from another. The sync store stays authoritative; the index degrades to the sync per-session scan on any failure.
-
-Kill-switch: `MEGACOMPACT_PGLITE_DISABLED=1` fully disables the PGlite index (falls back to sync scan). Requires Node ≥22.13 (`engines.node`).
-
-### Cross-repo recall
-
-On resume, recall augments from other repos' checkpoints when this repo's store is thin; `/mega-recall --cross-repo` searches all repos via the HNSW index. Cross-repo hits use a stricter cosine floor (`MEGACOMPACT_CROSSREPO_COSINE`, default 0.90) and are labeled with their source repo. A machine-wide injected-set (`~/.mega-compact-index/index.sqlite`) prevents re-injecting the same foreign checkpoint.
-
-### Memory
-
-pi-mega-compact auto-reviews the conversation every 10 turns (the cadence shortens as pressure climbs) and writes durable `decision`/`fact`/`preference` memories to SQLite (local, hallucination-guarded). Relevant memories are injected as RAG context on recall (capped, deduped). **Cross-repo memory-RAG (S24):** every memory write is mirrored into the PGlite/HNSW index, so when same-repo recall is thin the system augments with the nearest memories from *other* repos (stricter `MEGACOMPACT_CROSSREPO_COSINE` floor, deduped against what's already in view). Manual: `/mega-memory save|list|forget` (or `/m`).
-
-### Uninstall
+<details>
+<summary>From source (development)</summary>
 
 ```bash
-pi uninstall npm:pi-mega-compact   # removes from settings.packages + the npm tree
-# or, to keep the config entry but drop the package:
-npm uninstall pi-mega-compact
+git clone https://github.com/TheArchitectit/pi-mega-compact.git \
+  ~/.pi/agent/extensions/pi-mega-compact
+cd ~/.pi/agent/extensions/pi-mega-compact
+npm install && npm run build
 ```
 
-If you symlinked it into pi's extensions dir (dev only), also remove that link:
-
-```bash
-rm -f ~/.pi/agent/extensions/pi-mega-compact
-```
-
----
+The bundled `./install.sh` helper does the symlink + config edit (needs `jq`).
+</details>
 
 ## Usage
 
-Once installed and registered, pi-mega-compact runs **automatically** — you don't
-have to drive it. Past the context threshold it compacts in the background and
-drops a checkpoint; on resume it re-inlines the relevant ones silently.
-
-The commands (slash commands inside pi):
-
-### Commands
-
-| Command | Description |
-| --- | --- |
-| `/mega-compact [summary...]` | Manually compact the current session. A summary arg is used verbatim; otherwise the COLLAPSE heuristics build one. Persists a `chkpt_xxx`. |
-| `/mega-compact off` | Disable auto-compaction for this session. |
-| `/mega-status` | Show config + current context usage + store stats (checkpoint count, dedup rate, tokens saved) + the **installed version**. |
-| `/mega-recall [query]` | Semantic-search the local store, dedupe against the current window, and inline the top-K relevant checkpoints. No query → uses your latest message. `--cross-repo` searches all repos. |
-| `/mega-memory save <text>` / `save <category> <text>` / `list` / `search <query>` / `forget <text>` / `consolidate` | Manage durable memories (decisions, facts, preferences) written by auto-review and recalled as RAG context. Also `/m` shortform. |
-| `/mega-restore <chkpt\|recent>` | Re-inject a checkpoint's verbatim original region into context. |
-| `/mega-history` | List this session's checkpoints (id, date, files, tokens). |
-| `/mega-view <chkpt\|recent>` | Show a checkpoint's verbatim original region. |
-| `/mega-help` | Explain the toolbar widget terms (live tier, gate, dedup, tokens saved). |
-| `/mega-compat-check` | Detect extension conflicts (duplicate commands / overlapping handlers) across installed pi extensions. |
-| `/mega-db-stats` | Show mega-compact SQLite DB stats: table row counts, disk footprint (db + WAL + SHM), page count, freelist %, WAL frames. Read-only; safe any time. |
-| `/mega-db-prune [days]` | DELETE `raw_transcript` + `checkpoint_epochs` rows older than N days (default 30) + orphan `dedup_mirror` rows. Reports deleted counts + reclaimed bytes. |
-| `/mega-db-vacuum` | `VACUUM` the DB (rebuild pages, reclaim freelist). Heavy: briefly doubles disk usage. |
-| `/mega-db-check` | `PRAGMA integrity_check` + `wal_checkpoint(TRUNCATE)`. Fold the WAL into the main file and verify DB health. Use after a crash. |
-| `/mega-db-reconcile` | Fix `dedup_mirror.ref_count` drift vs actual `raw_transcript` refs, delete orphan dedup rows, backfill missing `content_ref`. Run after `/mega-db-prune` or a crash. |
-| `/mega-dashboard [open]` | Start the **localhost-only** live dashboard and open it in a browser (token gauge, store stats, live event stream, per-repo + All-repos/Summary views, cross-repo drift). |
-| `/mega-dashboard-status` | Report dashboard server status (port / url / live). |
-| `/mega-dashboard-stop` | Stop the dashboard server. |
-
-### The tier system
-
-The **tier** you see in the toolbar and dashboard is a *live pressure band* (`low` → `medium` → `high` → `ultra` → `mega`) that climbs automatically as your context window fills and falls back as it's relieved — it is driven by `currentTokens / effectiveThreshold`, not a manual setting. The base compaction *threshold* is set by `MEGACOMPACT_TIER` at startup as a **% of the model context window** (`low` 50% · `medium` 60% · `high` 70% · `ultra` 70% · `mega` 75%; default `low`) — the fire point is `tierPct × contextWindow`, so it always lands below pi's native ~80% auto-compaction (any model size). The old static token amounts (50k/100k/200k/1M/10M) are now only the boot fallback used before the first context event reports a window. `/mega-tier` was removed in v0.7.6. Higher pressure also deepens the live trim and reviews durable memory more often — the whole system reacts as one.
-
-### Live stats widget
-
-Above the pi editor the extension shows a compact widget:
-
-```
- ⚡ high·low v0.7.9 │ 142k/200k tokens (71%) │ 3 chkpts │ 🤖 2 agents │ turn 5
-   ◐ armed │ dedup: 92% │ saved: 45k tok
-```
-
-- **Version** — the installed npm version (read from `package.json` at runtime),
-  so the widget always reflects what `pi update --extensions` last pulled. If
-  this looks stale after an update, restart the dashboard server / pi session.
-- **Tier** — active compaction tier (low/medium/high/ultra/mega)
-- **Token usage** — current / max context window and %
-- **Checkpoints** — persisted checkpoints for the session
-- **Trigger state** — ○ idle, ◐ armed, ● ready
-- **Dedup hit rate** — % of checkpoints collapsed as duplicates
-- **Active agents / turn** — sub-agent count and conversation turn (when > 0)
-
-> **DB housekeeping** — `/mega-db-stats` / `prune` / `vacuum` / `check` / `reconcile`
-> give you manual control over the SQLite store. In addition, a best-effort
-> **auto-maintenance** pass runs on `session_start`: it prunes rows older than
-> 30d, checkpoints the WAL if it's over 10 MB, and VACUUMs if the DB is over
-> 100 MB AND the freelist is >20% of pages. It never blocks session start and
-> logs a one-line summary to the diagnostic log. (v0.7.6+)
-
----
-
-## Configuration (env-backed)
-
-All defaults are in `src/config/dedup.ts` (single source of truth). Set env vars
-before starting pi.
-
-### Core settings
-
-| Variable | Default | Meaning |
-| --- | --- | --- |
-| `MEGACOMPACT_FAST_GATE_PCT` | `70` | Context-usage % that arms the auto-trigger. Defaults to the tier's % of window (`tierPct*100`): low 50 · med 60 · high 70 · ultra 70 · mega 75. Override raises the arming floor. |
-| `MEGACOMPACT_TIER` | `low` | Named trigger preset — sets the compaction threshold as a **% of the model context window**: `low`(50%) `medium`(60%) `high`(70%) `ultra`(70%) `mega`(75%). Fire point = `tierPct × contextWindow`, so it always fires below pi's native ~80% auto-compaction (any model size). The old static token amounts (50k/100k/200k/1M/10M) are now only the **boot fallback** used before the first context event reports a window. Default `low`. |
-| `MEGACOMPACT_THRESHOLD_TOKENS` | *(tier default)* | Explicit **absolute** token budget (the `custom` tier). Overrides `MEGACOMPACT_TIER` when set and is **never percent-scaled** — use this to pin an exact token fire point regardless of model window. |
-| `MEGACOMPACT_ANCHOR_USER_MESSAGES` | `3` | Never drop the most recent N user messages (anchor floor). |
-| `MEGACOMPACT_PRESERVE_RECENT` | `4` | Preserve the most recent N messages verbatim. |
-| `MEGACOMPACT_AUTO` | `true` | Enable the auto-trigger. |
-| `MEGACOMPACT_AUTO_INLINE` | `true` | Auto-inline on resume / branch. |
-| `MEGACOMPACT_AUTO_INLINE_K` | `3` | Top-K checkpoints to auto-inline. |
-| `MEGACOMPACT_DEDUP_SIM` | `0.90` | Cosine threshold to collapse near-dupes. |
-| `MEGACOMPACT_STATE_DIR` | *(none — per-repo default)* | Override the store location. By default state is per-repo at `<repo>/.pi/mega-compact/`; this env var forces a single explicit dir (used as the fallback for non-git cwds). |
-
-### Dedup pipeline flags
+Once installed, it runs automatically. Past the context threshold it compacts in the background. On resume, it re-inlines relevant checkpoints silently.
 
-These gate the L0/L1/L2/RAPTOR dedup tiers. Defaults reproduce the all-active
-behavior. `MARK_ONLY_*` tiers run + record their decision but never
-collapse (safe partial-rollout / auto-degrade state).
+Key commands:
 
-| Variable | Default | Meaning |
-| --- | --- | --- |
-| `MEGACOMPACT_L0_ENABLED` | `true` | L0 exact content-hash dedup. |
-| `MEGACOMPACT_L1_ENABLED` | `true` | L1 MinHash/LSH near-dup verification. |
-| `MEGACOMPACT_L2_ENABLED` | `true` | L2 semantic cosine dedup + MMR retrieval diversity. |
-| `MEGACOMPACT_RAPTOR_ENABLED` | `false` | RAPTOR pre-compression tree (**shadow mode by default** — builds + logs, does not serve retrieval). |
-| `MEGACOMPACT_MARK_ONLY_L0` | `false` | L0: record, don't collapse. |
-| `MEGACOMPACT_MARK_ONLY_L1` | `false` | L1: record, don't collapse. |
-| `MEGACOMPACT_MARK_ONLY_L2` | `false` | L2: record, don't collapse. |
-| `MEGACOMPACT_MINILM` | `false` | MiniLM embedder flag — **off; not shipped** (see Embedding). BYO via `MEGACOMPACT_EMBEDDING_URL`. |
-| `MEGACOMPACT_EMBEDDING_URL` | *(unset)* | BYO localhost embedder endpoint (loopback-only; enables `HttpEmbedder`). |
-| `MEGACOMPACT_L2_THRESHOLD` | `0.85` | L2 cosine firing point (trigram-honest; set higher for semantic backends). |
-| `MEGACOMPACT_L1_JACCARD` | `0.8` | L1 MinHash/LSH near-dup Jaccard threshold. |
-| `MEGACOMPACT_MMR_LAMBDA` | `0.5` | MMR retrieval-diversity weight (λ·relevance − (1−λ)·maxSim). |
-| `MEGACOMPACT_SEMDEDUP_COSINE` | `0.95` | Offline SemDeDup pair threshold → `dedup_status='removed'`. |
-| `MEGACOMPACT_FP_RATE_L0` | `0.01` | L0 false-positive alert threshold (auto → MARK_ONLY). |
-| `MEGACOMPACT_FP_RATE_L1L2` | `0.05` | L1/L2 false-positive alert threshold (auto → MARK_ONLY). |
-| `MEGACOMPACT_ALERT_WINDOW_MS` | `600000` | FP-rate rolling window (10 min). |
-| `MEGACOMPACT_P95_BUDGET_MS` | `100` | Per-tier p95 latency budget; canary auto-disables on breach. |
+- `/mega-compact` — manually compact the current session
+- `/mega-status` — show context usage, store stats, version
+- `/mega-recall [query]` — semantic search the store, `--cross-repo` for all repos
+- `/mega-memory save|list|search|forget` — manage durable memories
+- `/mega-dashboard` — start the localhost dashboard
 
-See `docs/DEDUP_RUNBOOK.md` for incident response (SEV tiers, first-15-min
-checklist, MARK_ONLY degrade) and `docs/RETENTION_POLICY.md` for TTL / soft-delete
-/ VACUUM.
+Full command reference: [`docs/COMMANDS.md`](docs/COMMANDS.md)
 
-### Continuity & memory
+## Configuration
 
-| Variable | Default | Meaning |
-| --- | --- | --- |
-| `MEGACOMPACT_LEGACY_DURABLE_TRIM` | `false` | Restore the legacy auto-trigger (`ctx.compact()` stops the agent). One-release rollback; default uses live context-event trim + pi native auto-compaction (compact-and-continue). |
-| `MEGACOMPACT_CROSSREPO_ENABLED` | `true` | Cross-repo recall on resume + `/mega-recall --cross-repo` (HNSW index over every repo). |
-| `MEGACOMPACT_CROSSREPO_COSINE` | `0.90` | Stricter cosine floor for cross-repo hits (vs `0.85` same-repo). |
-| `MEGACOMPACT_MEMORY_AUTO_REVIEW` | `true` | Auto-review the conversation every `MEGACOMPACT_MEMORY_REVIEW_INTERVAL` turns → durable memories. |
-| `MEGACOMPACT_MEMORY_REVIEW_INTERVAL` | `10` | Turns between auto-review cycles. |
-| `MEGACOMPACT_PGLITE_DISABLED` | *(unset — index on)* | Kill-switch for the PGlite/HNSW cross-repo index; set `1`/`true` to disable (falls back to sync per-session scan). |
+Set env vars before starting pi. Defaults are in `src/config/dedup.ts`.
 
-## Dashboard
+| Variable | Default | Description |
+|---|---|---|
+| `MEGACOMPACT_TIER` | `low` | Threshold as % of context window (low=50%, medium=60%, high=70%) |
+| `MEGACOMPACT_AUTO` | `true` | Enable auto-compaction |
+| `MEGACOMPACT_DEDUP_SIM` | `0.90` | Cosine threshold for near-dup collapse |
+| `MEGACOMPACT_CROSSREPO_ENABLED` | `true` | Cross-repo recall on resume |
+| `MEGACOMPACT_EMBEDDING_URL` | _(unset)_ | BYO localhost embedder endpoint |
 
-The localhost-only dashboard (started with `/mega-dashboard`) is a single-page
-app served from a detached child process on `127.0.0.1` (random port in
-9320–9329). Every API is read-only — the server never writes the index or your
-store. It reads the machine-wide `repo_registry`
-(`~/.mega-compact-index/index.sqlite`) plus the current repo's own `node:sqlite`
-store.
-
-### Tabs
-
-- **Current repo** — the live single-session view: context-window gauge, trigger
-  status, the Vector Store (checkpoints / dropped / kept / freed / injected /
-  dedup / collapsed), the repo-wide aggregate, the Data Safety card, the live
-  Model & Cost Savings card, and crew/agent activity.
-- **All repos** — a machine-wide table aggregated from `repo_registry`
-  (`GET /api/index`): one row per repo with checkpoints, tokens saved,
-  compressed-originals, last-compacted, and active model. Each row opens a
-  per-repo detail modal. Currently-open sessions are badged **active** (see
-  below).
-- **Summary** — machine-wide header tiles plus a **savings-by-model** table
-  (tokens in/out/freed, context window, $ saved) grouped by the model you were
-  running.
-
-### Active Repos tab
-
-The dashboard has a dedicated **Active Repos** tab that lists every server /
-session seen within the **last 30 minutes**, each with its live tier,
-context %, and session state. It is backed by `GET /api/servers`, which walks
-the machine-wide `repo_registry`, reads each repo's per-process
-`dashboard.json` snapshot, and returns one row per currently-open session —
-so 1–6 sessions running at once are visible together in a single table instead
-of only the single current-repo view. Each row shows that repo's live
-cache-hit / compaction / time-saved totals (see below).
-
-The older **All repos** view and `GET /api/summary` still surface an
-`activeRepos` count, and `GET /api/repos?active=Nh` filters to repos seen
-within the last *N* hours (e.g. `?active=24h`, hour-granular) for the
-longer-window cross-repo table.
-
-### Metrics (DB-backed, durable)
-
-The dashboard's cumulative **Cache hits**, **Compactions**, and **Estimated
-time saved** cards are backed by **SQLite `meta` counters**
-(`compact_count`, `recall_injected`, `cache_hit_tokens_saved`, plus the
-existing `tokens_saved` / `deduped`), not the per-process `dashboard.json`
-snapshot. Because they live in the repo's `node:sqlite` store, the totals are
-**durable across session restarts** and travel with the repo's state dir —
-`dashboard.json` is now just the live per-process view that feeds the Active
-Repos rows. The cards show:
-
-- **Cache hits** = dedup collapses (`deduped`) + recall re-injections
-  (`recall_injected`) — as **current session** and **repo-wide total**.
-- **Compactions** = current session (`checkpointCount`) + repo-wide total
-  (`compact_count` from `meta`).
-- **Estimated time saved** = compact time saved + cache-hit time saved, derived
-  from tokens ÷ ~2k tok/s and labeled `est.`, as current session + total.
-
-### Localhost API
-
-`GET /api/snapshot` (current-repo live state),
-`/api/servers` (**Active Repos** — sessions active in the last 30 min, with
-tier / context % / state / live cache-hit & compaction totals),
-`/api/index` (all repos), `/api/repos` (with `?active=Nh` filter), `/api/summary`
-(header tiles + `activeRepos`), `/api/drift` (cross-repo drift: stale /
-compaction-lag / model-churn — read-only), `/api/events` (SSE live event stream),
-and `/api/version`.
-
-### Data safety
-
-Every compacted region is kept verbatim (compressed);
-the Data Safety card shows regions retained, compressed-originals bytes, dedup
-duplicates, and permanently-deleted bytes (**always 0**). Nothing is permanently
-deleted — any region is restorable.
-
----
+Full config reference: [`docs/CONFIGURATION.md`](docs/CONFIGURATION.md)
 
 ## Architecture
 
 ```
-extensions/mega-compact.ts   pi extension entry; wires src/ into pi lifecycle
-extensions/mega-trim.ts      live context-event trim (compact-and-continue, no abort)
-extensions/mega-conflict-cmds.ts  extension-conflict detector (/mega-compat-check)
-extensions/dashboard-server.ts     localhost dashboard (HTML + snapshot/version/drift APIs)
-src/adapt.ts                  the single pi↔engine message adapter (index-aligned)
-src/engine.ts                 Layer 4: compactSession() Trident pipeline + recall()
-src/vectorStore.ts            Layer 3: local vector DB (add/search/dedupe + near-dup)
-src/embedder.ts               default TrigramEmbedder (deterministic, 512-dim)
-src/httpEmbedder.ts           BYO localhost embedder seam (MEGACOMPACT_EMBEDDING_URL)
-src/store/sqlite.ts           the "one store" — node:sqlite context_chunks + session_state (FTS5 trigram)
-src/store/vectorIndex.ts      async PGlite/HNSW cross-repo vector index (redundant, best-effort)
-src/store/migrate.ts          JSON → SQLite migration (legacy .checkpoints.json.gz retained)
-src/store/backfill.ts         resumable backfill orchestrator (L0/L1/L2/RAPTOR)
-src/memory.ts                 durable memories (decision/fact/preference) + auto-review
-src/memoryOps.ts              memory apply/consolidate ops
-src/memoryRecall.ts           memory recall + auto-inline (RAG context)
-src/driftDetection.ts         cross-repo drift report (stale/idle/compaction-lag/model-churn)
-src/monitoring.ts             local events.log + dashboard.json metrics + FP alerts
-src/canary.ts                 sequential L0→L1→L2→RAPTOR rollout, auto-disable on p95 breach
-src/config/dedup.ts           single source of truth for ALL dedup tier flags + thresholds
-src/store.ts                  state dir + JSON DR helpers + compression re-exports
-src/compact.ts                Layer 2: summarize / merge / autoCompactCheck
-src/supersede.ts              Layer 1: obsolete file-read pruning
-src/boundary.ts               drop-boundary guards (anchor floor + tool-pair)
-src/tokens.ts                 deterministic token estimator
-src/types.ts                  engine-internal types
+extensions/    Pi entry points (mega-compact, mega-trim, dashboard)
+src/engine.ts  Trident pipeline (supersede → collapse → cluster)
+src/vectorStore.ts   Local vector DB (add/search/dedupe)
+src/compact.ts       Summarize / merge / auto-compact
+src/memory.ts        Durable memories + auto-review
+src/store/sqlite.ts   node:sqlite store (Node ≥22.13)
+src/store/vectorIndex.ts  PGlite/HNSW cross-repo index
 ```
 
-The `src/` directory is **pi-agnostic** and fully unit-tested (`node --test`).
-The extension entry adapts between the engine and pi's runtime types.
-
----
+Detailed architecture: [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md)
 
 ## Development
 
 ```bash
-npm run build      # tsc
-npm test           # build + node --test on dist/**/*.test.js (407 tests)
-npm run lint       # tsc --noEmit + guardrails-scan
-npm run guardrails # regression_check + guardrails-scan
+npm run build     # TypeScript compile
+npm test          # Build + 353 tests
+npm run lint      # Type check + guardrails scan
 ```
 
-The agent-guardrails suite (Four Laws, scope, secrets, regression) gates every
-change.
-
----
-
-## Testing & bug reports
-
-Full QA instructions — environment setup, the manual test checklist, what to
-include in a bug report, and known limitations — live in
-[`TESTER_GUIDE.md`](TESTER_GUIDE.md). Open issues at
-[github.com/TheArchitectit/pi-mega-compact/issues](https://github.com/TheArchitectit/pi-mega-compact/issues).
-
----
-
-## Acknowledgements
-
-Algorithmic reference (reimplemented in TypeScript, not vendored): memory-mcp
-(`MemoryCompactor` / `compact.py`), claw-code (`trident.rs` / `compact.rs`), and
-neuralwatt-mcr (pi-extension mechanics). Attribution as design sources only.
+Testing guide: [`TESTER_GUIDE.md`](TESTER_GUIDE.md)
 
 ## License
 
-[BSD-2-Clause](./LICENSE)
+BSD-2-Clause

package/dist/extensions/dashboard-client/src/hooks/useApi.js

@@ -0,0 +1,51 @@
+/**
+ * dashboard-client/src/hooks/useApi.ts — Generic data fetching hook.
+ *
+ * Provides typed fetch with retry, stale detection, and error handling.
+ * SPRINT-B1: basic fetch. SPRINT-D1: retry + stale.
+ */
+import { useState, useEffect, useCallback, useRef } from 'react';
+// SPRINT-D1-REMAINING: integrate retryWithBackoff from utils/retry.ts.
+// SPRINT-D1-REMAINING: integrate staleness detection from utils/staleness.ts.
+// SPRINT-T1-REMAINING: add Authorization header when auth token present.
+export function useApi(fetchFn, options = {}) {
+    const { pollInterval = 0, maxRetries = 0 } = options;
+    const [data, setData] = useState(null);
+    const [error, setError] = useState(null);
+    const [loading, setLoading] = useState(true);
+    const [lastFetchedAt, setLastFetchedAt] = useState(null);
+    const mountedRef = useRef(true);
+    const doFetch = useCallback(async () => {
+        setLoading(true);
+        setError(null);
+        try {
+            const result = await fetchFn();
+            if (mountedRef.current) {
+                setData(result);
+                setLastFetchedAt(Date.now());
+            }
+        }
+        catch (err) {
+            if (mountedRef.current) {
+                setError(err instanceof Error ? err : new Error(String(err)));
+            }
+        }
+        finally {
+            if (mountedRef.current) {
+                setLoading(false);
+            }
+        }
+    }, [fetchFn]);
+    useEffect(() => {
+        mountedRef.current = true;
+        doFetch();
+        return () => { mountedRef.current = false; };
+    }, [doFetch]);
+    useEffect(() => {
+        if (pollInterval <= 0)
+            return;
+        const timer = setInterval(doFetch, pollInterval);
+        return () => clearInterval(timer);
+    }, [doFetch, pollInterval]);
+    return { data, error, loading, refetch: doFetch, lastFetchedAt };
+}