pi-mega-compact 0.5.0 → 0.5.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +57 -107
- package/dist/extensions/dashboard-server.js +8 -1
- package/dist/extensions/mega-runtime.js +25 -3
- package/dist/extensions/openclaw-mega-compact.js +291 -0
- package/dist/src/minilm.js +92 -0
- package/dist/src/wordpiece.js +129 -0
- package/extensions/dashboard-server.ts +6 -2
- package/extensions/mega-runtime.ts +24 -3
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -6,10 +6,13 @@ sessions into a **local SQLite store** and offers **deduped inline recall** —
|
|
|
6
6
|
running **locally inside the extension**, with **no remote MCP server** and
|
|
7
7
|
**zero network calls at runtime** (PREVENT-PI-004).
|
|
8
8
|
|
|
9
|
-
> **v0.
|
|
10
|
-
>
|
|
11
|
-
> JSON checkpoint files.
|
|
12
|
-
>
|
|
9
|
+
> **Current version:** `v0.5.1` — storage backend is **`node:sqlite`**
|
|
10
|
+
> (`DatabaseSync`, a Node ≥22.13 built-in), replacing the old `better-sqlite3`
|
|
11
|
+
> native addon and the per-session gzipped JSON checkpoint files. **Zero native
|
|
12
|
+
> build step, fully local, zero network at runtime.** Legacy
|
|
13
|
+
> `.checkpoints.json.gz` snapshots are retained as disaster-recovery fallbacks
|
|
14
|
+
> and auto-imported on first run. Cross-repo recall, durable memory, and a
|
|
15
|
+
> localhost dashboard round out the continuity story.
|
|
13
16
|
|
|
14
17
|
---
|
|
15
18
|
|
|
@@ -110,9 +113,9 @@ OpenAI-style contract and the `MEGACOMPACT_EMBEDDING_KEY` / `MEGACOMPACT_EMBEDDI
|
|
|
110
113
|
|
|
111
114
|
### Requirements
|
|
112
115
|
|
|
113
|
-
- **Node >=
|
|
114
|
-
|
|
115
|
-
|
|
116
|
+
- **Node >= 22.13** (the synchronous `node:sqlite` backend requires it; see
|
|
117
|
+
`engines.node`). No native module is compiled — the store is a Node built-in.
|
|
118
|
+
- No network call and no API key are needed at runtime (PREVENT-PI-004).
|
|
116
119
|
- A pi coding agent install with package support (`pi install` / `pi update
|
|
117
120
|
--extensions`). npm-installed packages are auto-discovered via the package's
|
|
118
121
|
`pi` manifest entry; local checkouts load from `~/.pi/agent/extensions/`.
|
|
@@ -135,8 +138,10 @@ source (which pi loads directly) and the compiled `dist/`, so nothing else needs
|
|
|
135
138
|
building.
|
|
136
139
|
|
|
137
140
|
> **Tip — keep the spec unpinned.** Use `npm:pi-mega-compact`, not
|
|
138
|
-
> `npm:pi-mega-compact@0.
|
|
139
|
-
> `pi update --extensions`, so a pin would freeze you on that release.
|
|
141
|
+
> `npm:pi-mega-compact@0.5.1`. Version-pinned specs are *skipped* by
|
|
142
|
+
> `pi update --extensions`, so a pin would freeze you on that release. The
|
|
143
|
+
> installed version is always visible in the toolbar widget (`⚡ <tier> vX.Y.Z`)
|
|
144
|
+
> and via `/mega-status`.
|
|
140
145
|
|
|
141
146
|
> **From a git checkout (development only).** To hack on the extension, clone and
|
|
142
147
|
> build locally, then symlink it into pi's extensions dir — but this bypasses the
|
|
@@ -158,7 +163,7 @@ building.
|
|
|
158
163
|
> `pi update --extensions` on the device. (`.gitignore` rejects `*.tgz` so one can't
|
|
159
164
|
> be committed by accident.)
|
|
160
165
|
|
|
161
|
-
### Storage
|
|
166
|
+
### Storage
|
|
162
167
|
|
|
163
168
|
pi-mega-compact uses a dual local backend — **zero network, no native build step**:
|
|
164
169
|
|
|
@@ -167,22 +172,14 @@ pi-mega-compact uses a dual local backend — **zero network, no native build st
|
|
|
167
172
|
|
|
168
173
|
Kill-switch: `MEGACOMPACT_PGLITE_DISABLED=1` fully disables the PGlite index (falls back to sync scan). Requires Node ≥22.13 (`engines.node`).
|
|
169
174
|
|
|
170
|
-
### Cross-repo recall
|
|
175
|
+
### Cross-repo recall
|
|
171
176
|
|
|
172
177
|
On resume, recall augments from other repos' checkpoints when this repo's store is thin; `/mega-recall --cross-repo` searches all repos via the HNSW index. Cross-repo hits use a stricter cosine floor (`MEGACOMPACT_CROSSREPO_COSINE`, default 0.90) and are labeled with their source repo. A machine-wide injected-set (`~/.mega-compact-index/index.sqlite`) prevents re-injecting the same foreign checkpoint.
|
|
173
178
|
|
|
174
|
-
### Memory
|
|
179
|
+
### Memory
|
|
175
180
|
|
|
176
181
|
pi-mega-compact auto-reviews the conversation every 10 turns and writes durable `decision`/`fact`/`preference` memories to SQLite (local, hallucination-guarded). Relevant memories are injected as RAG context on recall (capped, deduped). Manual: `/mega-memory save|list|forget`.
|
|
177
182
|
|
|
178
|
-
### Verify
|
|
179
|
-
|
|
180
|
-
```bash
|
|
181
|
-
npm test # all unit/integration tests pass (346 as of v0.5.0)
|
|
182
|
-
npm run lint # tsc --noEmit + guardrails scan clean
|
|
183
|
-
python3 scripts/regression_check.py --all # spec/plan regression gate
|
|
184
|
-
```
|
|
185
|
-
|
|
186
183
|
### Uninstall
|
|
187
184
|
|
|
188
185
|
```bash
|
|
@@ -211,11 +208,16 @@ The commands (slash commands inside pi):
|
|
|
211
208
|
|---|---|
|
|
212
209
|
| `/mega-compact [summary...]` | Manually compact the current session. A summary arg is used verbatim; otherwise the COLLAPSE heuristics build one. Persists a `chkpt_xxx`. |
|
|
213
210
|
| `/mega-compact off` | Disable auto-compaction for this session. |
|
|
214
|
-
| `/mega-status` | Show config + current context usage + store stats (checkpoint count, dedup rate, tokens saved)
|
|
211
|
+
| `/mega-status` | Show config + current context usage + store stats (checkpoint count, dedup rate, tokens saved) + the **installed version**. |
|
|
215
212
|
| `/mega-recall [query]` | Semantic-search the local store, dedupe against the current window, and inline the top-K relevant checkpoints. No query → uses your latest message. `--cross-repo` searches all repos. |
|
|
216
213
|
| `/mega-memory save <text>` / `save <category> <text>` / `list` / `search <query>` / `forget <text>` / `consolidate` | Manage durable memories (decisions, facts, preferences) written by auto-review and recalled as RAG context. Also `/m` shortform. |
|
|
214
|
+
| `/mega-restore <chkpt\|recent>` | Re-inject a checkpoint's verbatim original region into context. |
|
|
215
|
+
| `/mega-history` | List this session's checkpoints (id, date, files, tokens). |
|
|
216
|
+
| `/mega-view <chkpt\|recent>` | Show a checkpoint's verbatim original region. |
|
|
217
|
+
| `/mega-help` | Explain the toolbar widget terms (tier, gate, dedup, tokens saved). |
|
|
217
218
|
| `/mega-tier [name]` | Set the compaction tier (`low` / `medium` / `high` / `ultra` / `mega`). Shows current tier with no arg. |
|
|
218
|
-
| `/mega-
|
|
219
|
+
| `/mega-compat-check` | Detect extension conflicts (duplicate commands / overlapping handlers) across installed pi extensions. |
|
|
220
|
+
| `/mega-dashboard` | Start the **localhost-only** live dashboard and open it in a browser (token gauge, store stats, live event stream, per-repo + cross-repo drift). |
|
|
219
221
|
| `/mega-dashboard-status` | Report dashboard server status. |
|
|
220
222
|
| `/mega-dashboard-stop` | Stop the dashboard server. |
|
|
221
223
|
|
|
@@ -224,10 +226,13 @@ The commands (slash commands inside pi):
|
|
|
224
226
|
Above the pi editor the extension shows a compact widget:
|
|
225
227
|
|
|
226
228
|
```
|
|
227
|
-
⚡ medium │ 142k/200k tokens (71%) │ 3 chkpts │ 🤖 2 agents │ turn 5
|
|
229
|
+
⚡ medium v0.5.1 │ 142k/200k tokens (71%) │ 3 chkpts │ 🤖 2 agents │ turn 5
|
|
228
230
|
◐ armed │ dedup: 92% │ saved: 45k tok
|
|
229
231
|
```
|
|
230
232
|
|
|
233
|
+
- **Version** — the installed npm version (read from `package.json` at runtime),
|
|
234
|
+
so the widget always reflects what `pi update --extensions` last pulled. If
|
|
235
|
+
this looks stale after an update, restart the dashboard server / pi session.
|
|
231
236
|
- **Tier** — active compaction tier (low/medium/high/ultra/mega)
|
|
232
237
|
- **Token usage** — current / max context window and %
|
|
233
238
|
- **Checkpoints** — persisted checkpoints for the session
|
|
@@ -255,10 +260,10 @@ before starting pi.
|
|
|
255
260
|
| `MEGACOMPACT_DEDUP_SIM` | `0.90` | Cosine threshold to collapse near-dupes. |
|
|
256
261
|
| `MEGACOMPACT_STATE_DIR` | _(none — per-repo default)_ | Override the store location. By default state is per-repo at `<repo>/.pi/mega-compact/`; this env var forces a single explicit dir (used as the fallback for non-git cwds). |
|
|
257
262
|
|
|
258
|
-
#### Dedup pipeline flags (
|
|
263
|
+
#### Dedup pipeline flags (single source: `src/config/dedup.ts`)
|
|
259
264
|
|
|
260
265
|
These gate the L0/L1/L2/RAPTOR dedup tiers. Defaults reproduce the all-active
|
|
261
|
-
|
|
266
|
+
behavior. `MARK_ONLY_*` tiers run + record their decision but never
|
|
262
267
|
collapse (safe partial-rollout / auto-degrade state).
|
|
263
268
|
|
|
264
269
|
| Variable | Default | Meaning |
|
|
@@ -285,18 +290,18 @@ See `docs/DEDUP_RUNBOOK.md` for incident response (SEV tiers, first-15-min
|
|
|
285
290
|
checklist, MARK_ONLY degrade) and `docs/RETENTION_POLICY.md` for TTL / soft-delete
|
|
286
291
|
/ VACUUM.
|
|
287
292
|
|
|
288
|
-
#### Continuity + memory knobs
|
|
293
|
+
#### Continuity + memory knobs
|
|
289
294
|
|
|
290
295
|
| Variable | Default | Meaning |
|
|
291
296
|
|---|---|---|
|
|
292
|
-
| `MEGACOMPACT_LEGACY_DURABLE_TRIM` | `false` | Restore the
|
|
297
|
+
| `MEGACOMPACT_LEGACY_DURABLE_TRIM` | `false` | Restore the legacy auto-trigger (`ctx.compact()` stops the agent). One-release rollback; default uses live context-event trim + pi native auto-compaction (compact-and-continue). |
|
|
293
298
|
| `MEGACOMPACT_CROSSREPO_ENABLED` | `true` | Cross-repo recall on resume + `/mega-recall --cross-repo` (HNSW index over every repo). |
|
|
294
299
|
| `MEGACOMPACT_CROSSREPO_COSINE` | `0.90` | Stricter cosine floor for cross-repo hits (vs `0.85` same-repo). |
|
|
295
300
|
| `MEGACOMPACT_MEMORY_AUTO_REVIEW` | `true` | Auto-review the conversation every `MEGACOMPACT_MEMORY_REVIEW_INTERVAL` turns → durable memories. |
|
|
296
301
|
| `MEGACOMPACT_MEMORY_REVIEW_INTERVAL` | `10` | Turns between auto-review cycles. |
|
|
297
|
-
| `MEGACOMPACT_PGLITE_DISABLED` |
|
|
302
|
+
| `MEGACOMPACT_PGLITE_DISABLED` | _(unset — index on)_ | Kill-switch for the PGlite/HNSW cross-repo index; set `1`/`true` to disable (falls back to sync per-session scan). |
|
|
298
303
|
|
|
299
|
-
#### Dashboard
|
|
304
|
+
#### Dashboard
|
|
300
305
|
|
|
301
306
|
The localhost-only dashboard adds a **Summary** + **All-repos** view over the
|
|
302
307
|
machine-wide `repo_registry`, plus a **cross-repo drift** report (`GET /api/drift`)
|
|
@@ -306,78 +311,35 @@ All read-only — the report never writes the index.
|
|
|
306
311
|
|
|
307
312
|
---
|
|
308
313
|
|
|
309
|
-
## Reporting for testers (what to capture)
|
|
310
|
-
|
|
311
|
-
If you're testing pi-mega-compact, the maintainers need **local evidence**, not
|
|
312
|
-
guesswork. The store and logs are plain local files — never a network port.
|
|
313
|
-
|
|
314
|
-
1. **Install + run it** (see [Installation](#install)).
|
|
315
|
-
2. **Work a real session** until context fills past the gate (80%+) — you should
|
|
316
|
-
see the status chip flip to `● ready`, then `◐ armed`, a checkpoint persist,
|
|
317
|
-
and context visibly drop.
|
|
318
|
-
3. **Resume and confirm recall:** restart pi, ask about something you worked on
|
|
319
|
-
earlier; relevant checkpoints should auto-inline (or use
|
|
320
|
-
`/mega-recall <topic>`).
|
|
321
|
-
4. **Watch the live signal** while testing:
|
|
322
|
-
```bash
|
|
323
|
-
tail -f ~/.pi/agent/extensions/pi-mega-compact/events.log | jq .
|
|
324
|
-
```
|
|
325
|
-
Each line is `{ts, tier, result, latencyMs, falsePositive?}`.
|
|
326
|
-
5. **Run the dashboard** (`/mega-dashboard`) and check the token gauge, store
|
|
327
|
-
stats, and live event stream.
|
|
328
|
-
6. **Try `/mega-tier`** to see and switch compaction tiers.
|
|
329
|
-
|
|
330
|
-
### What to include in a bug report
|
|
331
|
-
|
|
332
|
-
- Output of `/mega-status` (config + store stats).
|
|
333
|
-
- Output of `/mega-dashboard-status`.
|
|
334
|
-
- Your pi version + OS + Node version (`node -v`).
|
|
335
|
-
- A slice of `events.log` around the problem (the `result`/`tier` lines).
|
|
336
|
-
- `dashboard.json` from the state dir (aggregate metrics: hit rate, FP rate,
|
|
337
|
-
per-tier p95, storage bytes).
|
|
338
|
-
- If you suspect data loss or duplication: the checkpoint count and the
|
|
339
|
-
`sqlite.db` size, plus the output of the DR drill (below).
|
|
340
|
-
|
|
341
|
-
**Disaster-recovery drill** (validates the store against its JSON snapshots and
|
|
342
|
-
rebuilds if corrupt — see `docs/RETENTION_POLICY.md` §5):
|
|
343
|
-
|
|
344
|
-
```bash
|
|
345
|
-
scripts/dedup-restore-drill.sh ~/.pi/agent/extensions/pi-mega-compact
|
|
346
|
-
```
|
|
347
|
-
|
|
348
|
-
**Benchmark** (dedup hit rate, compression ratio, per-tier p95, storage at
|
|
349
|
-
100 / 1K / 10K checkpoints):
|
|
350
|
-
|
|
351
|
-
```bash
|
|
352
|
-
npm run build
|
|
353
|
-
node scripts/dedup-benchmark.mjs 100 1000 10000
|
|
354
|
-
```
|
|
355
|
-
|
|
356
|
-
Open issues at: https://github.com/TheArchitectit/pi-mega-compact/issues
|
|
357
|
-
|
|
358
|
-
---
|
|
359
|
-
|
|
360
314
|
## Architecture & layout
|
|
361
315
|
|
|
362
316
|
```
|
|
363
317
|
extensions/mega-compact.ts pi extension entry; wires src/ into pi lifecycle
|
|
318
|
+
extensions/mega-trim.ts live context-event trim (compact-and-continue, no abort)
|
|
319
|
+
extensions/mega-conflict-cmds.ts extension-conflict detector (/mega-compat-check)
|
|
320
|
+
extensions/dashboard-server.ts localhost dashboard (HTML + snapshot/version/drift APIs)
|
|
364
321
|
src/adapt.ts the single pi↔engine message adapter (index-aligned)
|
|
365
322
|
src/engine.ts Layer 4: compactSession() Trident pipeline + recall()
|
|
366
323
|
src/vectorStore.ts Layer 3: local vector DB (add/search/dedupe + near-dup)
|
|
367
324
|
src/embedder.ts default TrigramEmbedder (deterministic, 512-dim)
|
|
368
325
|
src/httpEmbedder.ts BYO localhost embedder seam (MEGACOMPACT_EMBEDDING_URL)
|
|
369
|
-
src/store/sqlite.ts the "one store" —
|
|
326
|
+
src/store/sqlite.ts the "one store" — node:sqlite context_chunks + session_state (FTS5 trigram)
|
|
327
|
+
src/store/vectorIndex.ts async PGlite/HNSW cross-repo vector index (redundant, best-effort)
|
|
370
328
|
src/store/migrate.ts JSON → SQLite migration (legacy .checkpoints.json.gz retained)
|
|
371
329
|
src/store/backfill.ts resumable backfill orchestrator (L0/L1/L2/RAPTOR)
|
|
330
|
+
src/memory.ts durable memories (decision/fact/preference) + auto-review
|
|
331
|
+
src/memoryOps.ts memory apply/consolidate ops
|
|
332
|
+
src/memoryRecall.ts memory recall + auto-inline (RAG context)
|
|
333
|
+
src/driftDetection.ts cross-repo drift report (stale/idle/compaction-lag/model-churn)
|
|
372
334
|
src/monitoring.ts local events.log + dashboard.json metrics + FP alerts
|
|
373
335
|
src/canary.ts sequential L0→L1→L2→RAPTOR rollout, auto-disable on p95 breach
|
|
374
336
|
src/config/dedup.ts single source of truth for ALL dedup tier flags + thresholds
|
|
375
337
|
src/store.ts state dir + JSON DR helpers + compression re-exports
|
|
376
|
-
src/compact.ts
|
|
377
|
-
src/supersede.ts
|
|
378
|
-
src/boundary.ts
|
|
379
|
-
src/tokens.ts
|
|
380
|
-
src/types.ts
|
|
338
|
+
src/compact.ts Layer 2: summarize / merge / autoCompactCheck
|
|
339
|
+
src/supersede.ts Layer 1: obsolete file-read pruning
|
|
340
|
+
src/boundary.ts drop-boundary guards (anchor floor + tool-pair)
|
|
341
|
+
src/tokens.ts deterministic token estimator
|
|
342
|
+
src/types.ts engine-internal types
|
|
381
343
|
```
|
|
382
344
|
|
|
383
345
|
The `src/` directory is **pi-agnostic** and fully unit-tested (`node --test`).
|
|
@@ -389,34 +351,22 @@ The extension entry adapts between the engine and pi's runtime types.
|
|
|
389
351
|
|
|
390
352
|
```bash
|
|
391
353
|
npm run build # tsc
|
|
392
|
-
npm test # build + node --test on dist/**/*.test.js
|
|
354
|
+
npm test # build + node --test on dist/**/*.test.js (346 tests)
|
|
393
355
|
npm run lint # tsc --noEmit + guardrails-scan
|
|
394
356
|
npm run guardrails # regression_check + guardrails-scan
|
|
395
357
|
```
|
|
396
358
|
|
|
397
359
|
The agent-guardrails suite (Four Laws, scope, secrets, regression) gates every
|
|
398
|
-
|
|
360
|
+
change.
|
|
399
361
|
|
|
400
362
|
---
|
|
401
363
|
|
|
402
|
-
##
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
|
|
407
|
-
-
|
|
408
|
-
- ✅ Sprint 5 — commands / UX / config polish (status chip, store stats, debug log)
|
|
409
|
-
- ✅ Sprint 6 — hardening, docs, release (`install.sh`, CHANGELOG, `v0.1.0`)
|
|
410
|
-
- ✅ Sprint 8 — SQLite storage backbone (`better-sqlite3`, one store) + compression v2
|
|
411
|
-
- ✅ Sprints 9–11 — L0 exact-hash + L1 MinHash/LSH near-dup dedup tiers
|
|
412
|
-
- ✅ Sprint 12 — L2 semantic cosine + MMR; BYO localhost embedder (`HttpEmbedder`)
|
|
413
|
-
- ✅ Sprint 13 — RAPTOR hierarchical pre-compression (shadow mode)
|
|
414
|
-
- ✅ Sprint 14 — full pipeline: flags, backfill, monitoring, canary rollout
|
|
415
|
-
- ✅ Sprint 15 — benchmarks, DR drill, docs, `v0.2.0`
|
|
416
|
-
|
|
417
|
-
See `SPRINT_PLAN.md` for the full breakdown and `PLAN.md` for architecture,
|
|
418
|
-
`RESEARCH.md` for the pi-API constraints that shaped it, `CHANGELOG.md` for
|
|
419
|
-
release notes.
|
|
364
|
+
## Testing & bug reports
|
|
365
|
+
|
|
366
|
+
Full QA instructions — environment setup, the manual test checklist, what to
|
|
367
|
+
include in a bug report, and known limitations — live in
|
|
368
|
+
[`TESTER_GUIDE.md`](TESTER_GUIDE.md). Open issues at
|
|
369
|
+
[github.com/TheArchitectit/pi-mega-compact/issues](https://github.com/TheArchitectit/pi-mega-compact/issues).
|
|
420
370
|
|
|
421
371
|
---
|
|
422
372
|
|
|
@@ -428,4 +378,4 @@ neuralwatt-mcr (pi-extension mechanics). Attribution as design sources only.
|
|
|
428
378
|
|
|
429
379
|
## License
|
|
430
380
|
|
|
431
|
-
[
|
|
381
|
+
[BSD-2-Clause](./LICENSE)
|
|
@@ -119,6 +119,11 @@ function readIndex() {
|
|
|
119
119
|
}
|
|
120
120
|
}
|
|
121
121
|
// ---------------------------------------------------------------------------
|
|
122
|
+
// Types
|
|
123
|
+
// ---------------------------------------------------------------------------
|
|
124
|
+
/** Package version of this extension, surfaced in the dashboard header. */
|
|
125
|
+
let dashboardServerVersion = "0.0.0";
|
|
126
|
+
// ---------------------------------------------------------------------------
|
|
122
127
|
// Helpers
|
|
123
128
|
// ---------------------------------------------------------------------------
|
|
124
129
|
function readSnapshot(snapshotPath) {
|
|
@@ -169,6 +174,7 @@ function dashboardHtml(tierName) {
|
|
|
169
174
|
body { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif; background: #0d1117; color: #c9d1d9; padding: 24px; line-height: 1.5; }
|
|
170
175
|
h1 { font-size: 20px; font-weight: 600; margin-bottom: 20px; display: flex; align-items: center; gap: 10px; color: #f0f6fc; }
|
|
171
176
|
h1 .tier { background: #1f6feb; color: #fff; font-size: 11px; font-weight: 700; padding: 2px 8px; border-radius: 10px; text-transform: uppercase; letter-spacing: .5px; }
|
|
177
|
+
h1 .version-pill { background: #30363d; color: #8b949e; font-size: 11px; font-weight: 600; padding: 2px 8px; border-radius: 10px; font-family: ui-monospace, SFMono-Regular, Menlo, monospace; }
|
|
172
178
|
.grid { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; margin-bottom: 20px; }
|
|
173
179
|
.card { background: #161b22; border: 1px solid #30363d; border-radius: 8px; padding: 16px; }
|
|
174
180
|
.card.safe { border-color: #238636; }
|
|
@@ -255,7 +261,7 @@ function dashboardHtml(tierName) {
|
|
|
255
261
|
|
|
256
262
|
<div class="offline-banner" id="offline-banner">Dashboard data unavailable — waiting for a pi session to write snapshot...</div>
|
|
257
263
|
|
|
258
|
-
<h1><span>mega-compact</span><span class="tier">${tierName}</span><span class="model-pill" id="hdr-model">—</span></h1>
|
|
264
|
+
<h1><span>mega-compact</span><span class="tier">${tierName}</span><span class="version-pill">v${dashboardServerVersion}</span><span class="model-pill" id="hdr-model">—</span></h1>
|
|
259
265
|
|
|
260
266
|
<nav class="tabs">
|
|
261
267
|
<button class="tab active" data-tab="current">Current repo</button>
|
|
@@ -689,6 +695,7 @@ export async function launchDashboardServer(stateDir) {
|
|
|
689
695
|
const pkg = JSON.parse(readFileSync(p, "utf-8"));
|
|
690
696
|
if (pkg.version) {
|
|
691
697
|
SERVER_VERSION = pkg.version;
|
|
698
|
+
dashboardServerVersion = pkg.version;
|
|
692
699
|
break;
|
|
693
700
|
}
|
|
694
701
|
}
|
|
@@ -9,7 +9,9 @@
|
|
|
9
9
|
* original closure.
|
|
10
10
|
*/
|
|
11
11
|
import { sessionEntryToContextMessages } from "@earendil-works/pi-coding-agent";
|
|
12
|
-
import { join } from "node:path";
|
|
12
|
+
import { join, dirname } from "node:path";
|
|
13
|
+
import { fileURLToPath } from "node:url";
|
|
14
|
+
import { readFileSync } from "node:fs";
|
|
13
15
|
import { VectorStore } from "../src/vectorStore.js";
|
|
14
16
|
import { toEngineMessages } from "../src/adapt.js";
|
|
15
17
|
import { normalizeSessionId } from "../src/store.js";
|
|
@@ -20,6 +22,23 @@ import { Dashboard } from "./mega-dashboard.js";
|
|
|
20
22
|
export const STATUS_KEY = "mega-compact";
|
|
21
23
|
export const WIDGET_KEY = "mega-compact-stats";
|
|
22
24
|
export const MARKER_TYPE = "mega-compact-marker";
|
|
25
|
+
/** Cached npm version, read once from this extension's own package.json. */
|
|
26
|
+
let CACHED_VERSION = null;
|
|
27
|
+
function ownVersion() {
|
|
28
|
+
if (CACHED_VERSION !== null)
|
|
29
|
+
return CACHED_VERSION;
|
|
30
|
+
let v = "?";
|
|
31
|
+
try {
|
|
32
|
+
const here = dirname(fileURLToPath(import.meta.url)); // .../extensions
|
|
33
|
+
const pkg = JSON.parse(readFileSync(join(here, "..", "package.json"), "utf-8"));
|
|
34
|
+
v = pkg.version ?? "?";
|
|
35
|
+
}
|
|
36
|
+
catch {
|
|
37
|
+
v = "?";
|
|
38
|
+
}
|
|
39
|
+
CACHED_VERSION = v;
|
|
40
|
+
return v;
|
|
41
|
+
}
|
|
23
42
|
/** ANSI palette for the toolbar. The pi TUI's Text component preserves ANSI
|
|
24
43
|
* escape codes (see wrapTextWithAnsi), so raw escapes render as colors. No
|
|
25
44
|
* chalk dependency needed — these are just strings. */
|
|
@@ -239,7 +258,7 @@ export class MegaRuntime {
|
|
|
239
258
|
// Phase 3 — pulsing status glyph while a compaction is in flight.
|
|
240
259
|
const pulse = this.pulsing ? `${C.cyan}${PULSE[Math.floor(Date.now() / 250) % PULSE.length]}${C.reset} ` : "";
|
|
241
260
|
const lines = [
|
|
242
|
-
` ${C.amber}⚡ ${this.config.tier}${C.reset} │ ${tokStr}/${maxStr} tokens (${C.bold}${pctStr}${C.reset}) │ ${st.checkpointCount} saved${agentStr}${turnStr}`,
|
|
261
|
+
` ${C.amber}⚡ ${this.config.tier}${C.reset} v${C.bold}${ownVersion()}${C.reset} │ ${tokStr}/${maxStr} tokens (${C.bold}${pctStr}${C.reset}) │ ${st.checkpointCount} saved${agentStr}${turnStr}`,
|
|
243
262
|
` ${triggerLabel} │ ${C.magenta}repeat-skipped: ${dedupStr}${C.reset} │ ${C.gray}memory held:${C.reset} ${usedStr} │ ${C.gray}space freed:${C.reset} ${savedStr}`,
|
|
244
263
|
];
|
|
245
264
|
// Phase 3 — compact progress bar: session tokens saved toward the rolling goal.
|
|
@@ -248,7 +267,10 @@ export class MegaRuntime {
|
|
|
248
267
|
const pct = Math.min(100, Math.round((this.rt.tokensSaved / goal) * 100));
|
|
249
268
|
const filled = Math.round((pct / 100) * 10);
|
|
250
269
|
const bar = "▓".repeat(filled) + "░".repeat(10 - filled);
|
|
251
|
-
|
|
270
|
+
// Session tokens saved, with the repo-wide total held alongside so the
|
|
271
|
+
// bar reads "saved X of goal" and the right side shows saved vs total.
|
|
272
|
+
const totalHeld = st.totalTokenEstimate > 0 ? st.totalTokenEstimate : repo.totalTokenEstimate;
|
|
273
|
+
lines.push(` ${C.green}saved ${fmt(this.rt.tokensSaved)} ${bar}${C.reset} ${pct}% of ${fmt(goal)} ${C.gray}│${C.reset} ${C.blue}${fmt(this.rt.tokensSaved)}${C.reset}/${C.blue}${fmt(totalHeld)}${C.reset} tok held`);
|
|
252
274
|
}
|
|
253
275
|
// Live "now processing" line + why + recent deduped/compacted events,
|
|
254
276
|
// collapsed to ONE rotating line (fresh only). The ticker ring buffer
|
|
@@ -0,0 +1,291 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* openclaw-mega-compact — OpenClaw plugin adapter for the pi-mega-compact engine.
|
|
3
|
+
*
|
|
4
|
+
* Wires the pi-agnostic Trident engine (src/) into OpenClaw's plugin lifecycle:
|
|
5
|
+
* - Registers a CompactionProvider that replaces the built-in summarizeInStages.
|
|
6
|
+
* - Exposes `mega_status` and `mega_recall` tools for on-demand inspection.
|
|
7
|
+
* - Hooks into `before_compaction` / `after_compaction` for diagnostics.
|
|
8
|
+
*
|
|
9
|
+
* Design constraints:
|
|
10
|
+
* - NO imports from `@earendil-works/pi-coding-agent` or pi-agent-core.
|
|
11
|
+
* - The engine core (src/) is pi-agnostic; this file is the sole OpenClaw boundary.
|
|
12
|
+
* - No network at runtime — everything is local (stores + extractive summarizer).
|
|
13
|
+
*/
|
|
14
|
+
import { definePluginEntry } from "openclaw/plugin-sdk/plugin-entry";
|
|
15
|
+
import { compactSession, setDefaultStore, } from "../src/engine.js";
|
|
16
|
+
import { recallAndInline } from "../src/recall.js";
|
|
17
|
+
import { VectorStore } from "../src/vectorStore.js";
|
|
18
|
+
// ---------------------------------------------------------------------------
|
|
19
|
+
// Constants
|
|
20
|
+
// ---------------------------------------------------------------------------
|
|
21
|
+
const PLUGIN_ID = "mega-compact";
|
|
22
|
+
const PLUGIN_LABEL = "Mega Compact (Trident)";
|
|
23
|
+
/** Default state directory for vector store persistence. */
|
|
24
|
+
const STATE_DIR = process.env.MEGA_COMPACT_STATE_DIR ?? undefined;
|
|
25
|
+
/** Minimum messages before we bother compacting. */
|
|
26
|
+
const MIN_MESSAGES_FOR_COMPACT = 6;
|
|
27
|
+
// ---------------------------------------------------------------------------
|
|
28
|
+
// Message conversion — OpenClaw unknown[] → EngineMessage[]
|
|
29
|
+
// ---------------------------------------------------------------------------
|
|
30
|
+
/**
|
|
31
|
+
* Best-effort conversion from OpenClaw's opaque message array to our
|
|
32
|
+
* EngineMessage shape. OpenClaw messages are typed as `unknown[]` so we
|
|
33
|
+
* handle whatever shape comes through gracefully.
|
|
34
|
+
*/
|
|
35
|
+
function toEngineMessages(messages) {
|
|
36
|
+
return messages.map((msg) => {
|
|
37
|
+
if (!msg || typeof msg !== "object") {
|
|
38
|
+
// Primitive fallback — treat as custom text.
|
|
39
|
+
return {
|
|
40
|
+
role: "custom",
|
|
41
|
+
text: String(msg ?? ""),
|
|
42
|
+
};
|
|
43
|
+
}
|
|
44
|
+
const m = msg;
|
|
45
|
+
const role = typeof m.role === "string" ? m.role : "custom";
|
|
46
|
+
// Normalize role to one of our four engine roles.
|
|
47
|
+
let engineRole;
|
|
48
|
+
switch (role) {
|
|
49
|
+
case "user":
|
|
50
|
+
engineRole = "user";
|
|
51
|
+
break;
|
|
52
|
+
case "assistant":
|
|
53
|
+
engineRole = "assistant";
|
|
54
|
+
break;
|
|
55
|
+
case "tool":
|
|
56
|
+
case "function":
|
|
57
|
+
engineRole = "tool";
|
|
58
|
+
break;
|
|
59
|
+
default:
|
|
60
|
+
engineRole = "custom";
|
|
61
|
+
break;
|
|
62
|
+
}
|
|
63
|
+
// Extract text content from common message shapes.
|
|
64
|
+
const text = typeof m.content === "string"
|
|
65
|
+
? m.content
|
|
66
|
+
: typeof m.text === "string"
|
|
67
|
+
? m.text
|
|
68
|
+
: Array.isArray(m.content)
|
|
69
|
+
? m.content
|
|
70
|
+
.filter((part) => part.type === "text" && typeof part.text === "string")
|
|
71
|
+
.map((part) => part.text)
|
|
72
|
+
.join("\n")
|
|
73
|
+
: "";
|
|
74
|
+
// Preserve tool metadata when present.
|
|
75
|
+
const toolName = typeof m.name === "string"
|
|
76
|
+
? m.name
|
|
77
|
+
: typeof m.toolName === "string"
|
|
78
|
+
? m.toolName
|
|
79
|
+
: undefined;
|
|
80
|
+
const input = typeof m.input === "string"
|
|
81
|
+
? m.input
|
|
82
|
+
: typeof m.arguments === "string"
|
|
83
|
+
? m.arguments
|
|
84
|
+
: m.arguments !== undefined
|
|
85
|
+
? JSON.stringify(m.arguments)
|
|
86
|
+
: undefined;
|
|
87
|
+
const output = typeof m.output === "string"
|
|
88
|
+
? m.output
|
|
89
|
+
: engineRole === "tool" && typeof m.content === "string"
|
|
90
|
+
? m.content
|
|
91
|
+
: undefined;
|
|
92
|
+
return { role: engineRole, text, toolName, input, output };
|
|
93
|
+
});
|
|
94
|
+
}
|
|
95
|
+
// ---------------------------------------------------------------------------
|
|
96
|
+
// Compaction provider
|
|
97
|
+
// ---------------------------------------------------------------------------
|
|
98
|
+
function createCompactionProvider(store) {
|
|
99
|
+
return {
|
|
100
|
+
id: PLUGIN_ID,
|
|
101
|
+
label: PLUGIN_LABEL,
|
|
102
|
+
async summarize({ messages, signal, compressionRatio, }) {
|
|
103
|
+
// Abort check — bail early if the caller cancelled.
|
|
104
|
+
if (signal?.aborted) {
|
|
105
|
+
throw new DOMException("Aborted", "AbortError");
|
|
106
|
+
}
|
|
107
|
+
const engineMessages = toEngineMessages(messages);
|
|
108
|
+
// Nothing meaningful to compact.
|
|
109
|
+
if (engineMessages.length < MIN_MESSAGES_FOR_COMPACT) {
|
|
110
|
+
return "";
|
|
111
|
+
}
|
|
112
|
+
// Map compression ratio → keepFrom boundary.
|
|
113
|
+
// compressionRatio=0.5 means "compact the oldest 50%".
|
|
114
|
+
// Default to compacting the oldest half if not specified.
|
|
115
|
+
const ratio = compressionRatio ?? 0.5;
|
|
116
|
+
const keepFrom = Math.max(MIN_MESSAGES_FOR_COMPACT, Math.floor(engineMessages.length * (1 - ratio)));
|
|
117
|
+
// Abort check after conversion (conversion is cheap but check anyway).
|
|
118
|
+
if (signal?.aborted) {
|
|
119
|
+
throw new DOMException("Aborted", "AbortError");
|
|
120
|
+
}
|
|
121
|
+
const sessionId = `openclaw-${Date.now()}`;
|
|
122
|
+
const input = {
|
|
123
|
+
sessionId,
|
|
124
|
+
messages: engineMessages,
|
|
125
|
+
keepFrom,
|
|
126
|
+
};
|
|
127
|
+
const result = compactSession(input, store);
|
|
128
|
+
if (result.skipped) {
|
|
129
|
+
return "";
|
|
130
|
+
}
|
|
131
|
+
return result.summary;
|
|
132
|
+
},
|
|
133
|
+
};
|
|
134
|
+
}
|
|
135
|
+
// ---------------------------------------------------------------------------
|
|
136
|
+
// Plugin entry
|
|
137
|
+
// ---------------------------------------------------------------------------
|
|
138
|
+
export default definePluginEntry({
|
|
139
|
+
id: PLUGIN_ID,
|
|
140
|
+
name: "Mega Compact",
|
|
141
|
+
description: "Layered, local, vector-backed context compressor (Trident engine) for OpenClaw compaction.",
|
|
142
|
+
register(api) {
|
|
143
|
+
const logger = api.logger;
|
|
144
|
+
// Resolve state directory — prefer plugin config override.
|
|
145
|
+
const pluginCfg = (api.pluginConfig ?? {});
|
|
146
|
+
const stateDir = typeof pluginCfg.stateDir === "string" && pluginCfg.stateDir.length > 0
|
|
147
|
+
? pluginCfg.stateDir
|
|
148
|
+
: STATE_DIR;
|
|
149
|
+
// Initialize vector store.
|
|
150
|
+
let store;
|
|
151
|
+
try {
|
|
152
|
+
store = new VectorStore({ stateDir });
|
|
153
|
+
setDefaultStore(store);
|
|
154
|
+
logger.info?.(`${PLUGIN_ID}: vector store initialized (stateDir=${stateDir ?? "default"})`);
|
|
155
|
+
}
|
|
156
|
+
catch (err) {
|
|
157
|
+
logger.error?.(`${PLUGIN_ID}: failed to init vector store:`, err);
|
|
158
|
+
return; // Hard bail — no point registering if store is broken.
|
|
159
|
+
}
|
|
160
|
+
// -----------------------------------------------------------------------
|
|
161
|
+
// Register compaction provider
|
|
162
|
+
// -----------------------------------------------------------------------
|
|
163
|
+
const provider = createCompactionProvider(store);
|
|
164
|
+
api.registerCompactionProvider(provider);
|
|
165
|
+
logger.info?.(`${PLUGIN_ID}: registered compaction provider "${provider.id}"`);
|
|
166
|
+
// -----------------------------------------------------------------------
|
|
167
|
+
// Hooks — before / after compaction diagnostics
|
|
168
|
+
// -----------------------------------------------------------------------
|
|
169
|
+
api.registerHook({
|
|
170
|
+
event: "before_compaction",
|
|
171
|
+
handler: async (ctx) => {
|
|
172
|
+
const msgCount = Array.isArray(ctx?.messages) ? ctx.messages.length : 0;
|
|
173
|
+
logger.info?.(`${PLUGIN_ID}: before_compaction — ${msgCount} messages in scope`);
|
|
174
|
+
},
|
|
175
|
+
});
|
|
176
|
+
api.registerHook({
|
|
177
|
+
event: "after_compaction",
|
|
178
|
+
handler: async (ctx) => {
|
|
179
|
+
const summaryLen = typeof ctx?.summary === "string" ? ctx.summary.length : 0;
|
|
180
|
+
logger.info?.(`${PLUGIN_ID}: after_compaction — summary ${summaryLen} chars`);
|
|
181
|
+
},
|
|
182
|
+
});
|
|
183
|
+
// -----------------------------------------------------------------------
|
|
184
|
+
// Tool: mega_status
|
|
185
|
+
// -----------------------------------------------------------------------
|
|
186
|
+
api.registerTool({
|
|
187
|
+
name: "mega_status",
|
|
188
|
+
description: "Show the current status of the mega-compact engine: vector store stats, checkpoint count, and recent compaction activity.",
|
|
189
|
+
parameters: {
|
|
190
|
+
type: "object",
|
|
191
|
+
properties: {
|
|
192
|
+
sessionId: {
|
|
193
|
+
type: "string",
|
|
194
|
+
description: "Optional session ID to scope stats to.",
|
|
195
|
+
},
|
|
196
|
+
},
|
|
197
|
+
additionalProperties: false,
|
|
198
|
+
},
|
|
199
|
+
handler: async (args) => {
|
|
200
|
+
const sessionId = args?.sessionId ?? "global";
|
|
201
|
+
try {
|
|
202
|
+
const stats = store.stats(sessionId);
|
|
203
|
+
const parts = [
|
|
204
|
+
`**Mega Compact Status**`,
|
|
205
|
+
`Session: ${sessionId}`,
|
|
206
|
+
`Checkpoints: ${stats.checkpointCount}`,
|
|
207
|
+
`Total tokens saved: ${stats.totalTokenEstimate}`,
|
|
208
|
+
`Last checkpoint: ${stats.lastCheckpointId ?? "—"}`,
|
|
209
|
+
`Injected count: ${stats.injectedCount}`,
|
|
210
|
+
`Dedup hit rate: ${(stats.dedupHitRate * 100).toFixed(0)}%`,
|
|
211
|
+
];
|
|
212
|
+
if (stats.lastSummary) {
|
|
213
|
+
parts.push(`\nLast summary (truncated):\n ${stats.lastSummary.slice(0, 120).replace(/\n/g, " ")}…`);
|
|
214
|
+
}
|
|
215
|
+
return { content: [{ type: "text", text: parts.join("\n") }] };
|
|
216
|
+
}
|
|
217
|
+
catch (err) {
|
|
218
|
+
return {
|
|
219
|
+
content: [{ type: "text", text: `Error reading mega-compact status: ${err}` }],
|
|
220
|
+
isError: true,
|
|
221
|
+
};
|
|
222
|
+
}
|
|
223
|
+
},
|
|
224
|
+
});
|
|
225
|
+
// -----------------------------------------------------------------------
|
|
226
|
+
// Tool: mega_recall
|
|
227
|
+
// -----------------------------------------------------------------------
|
|
228
|
+
api.registerTool({
|
|
229
|
+
name: "mega_recall",
|
|
230
|
+
description: "Recall and inline relevant context from the mega-compact vector store for the current session.",
|
|
231
|
+
parameters: {
|
|
232
|
+
type: "object",
|
|
233
|
+
properties: {
|
|
234
|
+
sessionId: {
|
|
235
|
+
type: "string",
|
|
236
|
+
description: "Session ID to recall context for.",
|
|
237
|
+
},
|
|
238
|
+
query: {
|
|
239
|
+
type: "string",
|
|
240
|
+
description: "Natural language query for relevant context.",
|
|
241
|
+
},
|
|
242
|
+
limit: {
|
|
243
|
+
type: "number",
|
|
244
|
+
description: "Max checkpoints to recall (default 3).",
|
|
245
|
+
},
|
|
246
|
+
},
|
|
247
|
+
required: ["sessionId", "query"],
|
|
248
|
+
additionalProperties: false,
|
|
249
|
+
},
|
|
250
|
+
handler: async (args) => {
|
|
251
|
+
const { sessionId, query, limit } = args;
|
|
252
|
+
if (!sessionId || !query) {
|
|
253
|
+
return {
|
|
254
|
+
content: [{ type: "text", text: "Both `sessionId` and `query` are required." }],
|
|
255
|
+
isError: true,
|
|
256
|
+
};
|
|
257
|
+
}
|
|
258
|
+
try {
|
|
259
|
+
const result = recallAndInline({ sessionId, query, limit: limit ?? 3, source: "command", skipInjected: false }, store);
|
|
260
|
+
if (result.toInject.length === 0) {
|
|
261
|
+
return {
|
|
262
|
+
content: [{ type: "text", text: "No relevant context found in the mega-compact store." }],
|
|
263
|
+
};
|
|
264
|
+
}
|
|
265
|
+
const parts = [
|
|
266
|
+
`**Recalled ${result.toInject.length} checkpoint(s):**`,
|
|
267
|
+
...result.report,
|
|
268
|
+
"",
|
|
269
|
+
"---",
|
|
270
|
+
result.block,
|
|
271
|
+
];
|
|
272
|
+
return { content: [{ type: "text", text: parts.join("\n") }] };
|
|
273
|
+
}
|
|
274
|
+
catch (err) {
|
|
275
|
+
return {
|
|
276
|
+
content: [{ type: "text", text: `Error during mega-recall: ${err}` }],
|
|
277
|
+
isError: true,
|
|
278
|
+
};
|
|
279
|
+
}
|
|
280
|
+
},
|
|
281
|
+
});
|
|
282
|
+
// -----------------------------------------------------------------------
|
|
283
|
+
// Cleanup on shutdown
|
|
284
|
+
// -----------------------------------------------------------------------
|
|
285
|
+
api.on("shutdown", () => {
|
|
286
|
+
logger.info?.(`${PLUGIN_ID}: shutting down — clearing default store`);
|
|
287
|
+
setDefaultStore(undefined);
|
|
288
|
+
});
|
|
289
|
+
logger.info?.(`${PLUGIN_ID}: plugin registered (tools: mega_status, mega_recall)`);
|
|
290
|
+
},
|
|
291
|
+
});
|
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* minilm.ts — local MiniLM (all-MiniLM-L6-v2) sentence embedder (Sprint 12).
|
|
3
|
+
*
|
|
4
|
+
* Implements the `Embedder` interface so it drops into the existing VectorStore
|
|
5
|
+
* dedup cascade and search with no call-site changes. Inference is 100% local:
|
|
6
|
+
* the ONNX model + WordPiece vocab are on-disk artifacts fetched once by
|
|
7
|
+
* scripts/setup-minilm.mjs. There is NO network call at runtime (PREVENT-PI-004).
|
|
8
|
+
*
|
|
9
|
+
* Inputs (dynamic): input_ids, attention_mask, token_type_ids (int64).
|
|
10
|
+
* Output: last_hidden_state (batch, seq, 384). We mean-pool over non-padded
|
|
11
|
+
* tokens (attention_mask == 1) and L2-normalize → 384-dim unit vector.
|
|
12
|
+
*
|
|
13
|
+
* The ONNX session + tokenizer are loaded LAZILY on first embed() so the default
|
|
14
|
+
* TrigramEmbedder path (and its zero native-init cost) is untouched unless
|
|
15
|
+
* MEGACOMPACT_EMBEDDER=minilm is selected.
|
|
16
|
+
*/
|
|
17
|
+
import { join } from "node:path";
|
|
18
|
+
import { homedir } from "node:os";
|
|
19
|
+
import { existsSync } from "node:fs";
|
|
20
|
+
import { l2Normalize, awaitSync } from "./embedder.js";
|
|
21
|
+
import { WordPieceTokenizer } from "./wordpiece.js";
|
|
22
|
+
export const MINILM_DIM = 384;
|
|
23
|
+
export const MINILM_MAX_LEN = 256;
|
|
24
|
+
/** Resolve the model directory: MEGACOMPACT_MINILM_DIR > ./models/minilm > ~/.pi … */
|
|
25
|
+
function resolveModelDir() {
|
|
26
|
+
if (process.env.MEGACOMPACT_MINILM_DIR)
|
|
27
|
+
return process.env.MEGACOMPACT_MINILM_DIR;
|
|
28
|
+
// Repo-local vendored path (gitignored).
|
|
29
|
+
const local = join(process.cwd(), "models", "minilm");
|
|
30
|
+
if (existsSync(local))
|
|
31
|
+
return local;
|
|
32
|
+
return join(homedir(), ".pi", "agent", "extensions", "mega-compact", "models", "minilm");
|
|
33
|
+
}
|
|
34
|
+
export class MiniLMEmbedder {
|
|
35
|
+
dim = MINILM_DIM;
|
|
36
|
+
session = null;
|
|
37
|
+
tokenizer = null;
|
|
38
|
+
modelDir;
|
|
39
|
+
loadPromise = null;
|
|
40
|
+
constructor(modelDir = resolveModelDir()) {
|
|
41
|
+
this.modelDir = modelDir;
|
|
42
|
+
}
|
|
43
|
+
async ensureLoaded() {
|
|
44
|
+
if (this.session && this.tokenizer)
|
|
45
|
+
return;
|
|
46
|
+
if (this.loadPromise)
|
|
47
|
+
return this.loadPromise;
|
|
48
|
+
this.loadPromise = (async () => {
|
|
49
|
+
const ort = await import("onnxruntime-node");
|
|
50
|
+
const modelPath = join(this.modelDir, "model_quantized.onnx");
|
|
51
|
+
const vocabPath = join(this.modelDir, "vocab.txt");
|
|
52
|
+
if (!existsSync(modelPath) || !existsSync(vocabPath)) {
|
|
53
|
+
throw new Error(`MiniLM artifacts missing in ${this.modelDir}. Run: node scripts/setup-minilm.mjs`);
|
|
54
|
+
}
|
|
55
|
+
// 1 thread is plenty for a single short-region embed and bounds CPU.
|
|
56
|
+
this.session = await ort.InferenceSession.create(modelPath, {
|
|
57
|
+
executionProviders: ["cpu"],
|
|
58
|
+
graphOptimizationLevel: "all",
|
|
59
|
+
});
|
|
60
|
+
this.tokenizer = WordPieceTokenizer.fromVocabFile(vocabPath);
|
|
61
|
+
})();
|
|
62
|
+
return this.loadPromise;
|
|
63
|
+
}
|
|
64
|
+
embed(text) {
|
|
65
|
+
awaitSync(this.ensureLoaded());
|
|
66
|
+
const enc = this.tokenizer.encode(text, MINILM_MAX_LEN);
|
|
67
|
+
const n = enc.inputIds.length;
|
|
68
|
+
const BigInt64 = (arr) => arr.map((x) => BigInt(x));
|
|
69
|
+
const ort = awaitSync(import("onnxruntime-node"));
|
|
70
|
+
const tensors = {
|
|
71
|
+
input_ids: new ort.Tensor("int64", BigInt64(enc.inputIds), [1, n]),
|
|
72
|
+
attention_mask: new ort.Tensor("int64", BigInt64(enc.attentionMask), [1, n]),
|
|
73
|
+
token_type_ids: new ort.Tensor("int64", BigInt64(enc.tokenTypeIds), [1, n]),
|
|
74
|
+
};
|
|
75
|
+
const out = awaitSync(this.session.run(tensors));
|
|
76
|
+
const hidden = out.last_hidden_state.data;
|
|
77
|
+
// hidden shape: [1, n, 384]. Mean-pool over non-padded positions.
|
|
78
|
+
const pooled = new Array(MINILM_DIM).fill(0);
|
|
79
|
+
let count = 0;
|
|
80
|
+
for (let i = 0; i < n; i++) {
|
|
81
|
+
if (enc.attentionMask[i] === 0)
|
|
82
|
+
continue;
|
|
83
|
+
const base = i * MINILM_DIM;
|
|
84
|
+
for (let d = 0; d < MINILM_DIM; d++)
|
|
85
|
+
pooled[d] += hidden[base + d];
|
|
86
|
+
count++;
|
|
87
|
+
}
|
|
88
|
+
if (count === 0)
|
|
89
|
+
return l2Normalize(new Array(MINILM_DIM).fill(0));
|
|
90
|
+
return l2Normalize(pooled.map((x) => x / count));
|
|
91
|
+
}
|
|
92
|
+
}
|
|
@@ -0,0 +1,129 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* wordpiece.ts — a self-contained WordPiece tokenizer for BERT/MiniLM.
|
|
3
|
+
*
|
|
4
|
+
* Loads the canonical `vocab.txt` (bert-base-uncased, ~30K tokens) from disk and
|
|
5
|
+
* implements the standard uncased BERT preprocessing + greedy longest-match
|
|
6
|
+
* WordPiece segmentation. No native dependency, no network — the vocab file is a
|
|
7
|
+
* local artifact fetched once by scripts/setup-minilm.mjs (PREVENT-PI-004).
|
|
8
|
+
*
|
|
9
|
+
* This mirrors HuggingFace `BertTokenizer` closely enough for sentence-embedding
|
|
10
|
+
* use: lowercase, strip accents, split on whitespace + punctuation, then
|
|
11
|
+
* WordPiece each token with the `##` continuation convention. Special tokens
|
|
12
|
+
* [CLS]/[SEP] are added by the caller's encode().
|
|
13
|
+
*/
|
|
14
|
+
import { readFileSync, existsSync } from "node:fs";
|
|
15
|
+
const UNK = "[UNK]";
|
|
16
|
+
const CLS = "[CLS]";
|
|
17
|
+
const SEP = "[SEP]";
|
|
18
|
+
const PAD = "[PAD]";
|
|
19
|
+
const MAX_INPUT_CHARS_PER_WORD = 200;
|
|
20
|
+
export class WordPieceTokenizer {
|
|
21
|
+
vocab;
|
|
22
|
+
clsId;
|
|
23
|
+
sepId;
|
|
24
|
+
padId;
|
|
25
|
+
unkId;
|
|
26
|
+
constructor(vocab) {
|
|
27
|
+
this.vocab = vocab;
|
|
28
|
+
this.clsId = vocab.get(CLS) ?? 101;
|
|
29
|
+
this.sepId = vocab.get(SEP) ?? 102;
|
|
30
|
+
this.padId = vocab.get(PAD) ?? 0;
|
|
31
|
+
this.unkId = vocab.get(UNK) ?? 100;
|
|
32
|
+
}
|
|
33
|
+
/** Build a tokenizer from a vocab.txt file (one token per line, index = line). */
|
|
34
|
+
static fromVocabFile(path) {
|
|
35
|
+
if (!existsSync(path)) {
|
|
36
|
+
throw new Error(`WordPiece vocab not found at ${path}. Run: node scripts/setup-minilm.mjs`);
|
|
37
|
+
}
|
|
38
|
+
const lines = readFileSync(path, "utf-8").split("\n");
|
|
39
|
+
const vocab = new Map();
|
|
40
|
+
for (let i = 0; i < lines.length; i++) {
|
|
41
|
+
const tok = lines[i].replace(/\r$/, "");
|
|
42
|
+
if (tok.length > 0 || i < lines.length - 1)
|
|
43
|
+
vocab.set(tok, i);
|
|
44
|
+
}
|
|
45
|
+
return new WordPieceTokenizer(vocab);
|
|
46
|
+
}
|
|
47
|
+
/** Uncased BERT basic tokenization: lowercase, strip accents, split on ws+punct. */
|
|
48
|
+
basicTokenize(text) {
|
|
49
|
+
// NFD + strip combining marks (accent removal), then lowercase.
|
|
50
|
+
const cleaned = text
|
|
51
|
+
.normalize("NFD")
|
|
52
|
+
.replace(/[̀-ͯ]/g, "")
|
|
53
|
+
.toLowerCase();
|
|
54
|
+
const tokens = [];
|
|
55
|
+
let buf = "";
|
|
56
|
+
const flush = () => {
|
|
57
|
+
if (buf.length > 0) {
|
|
58
|
+
tokens.push(buf);
|
|
59
|
+
buf = "";
|
|
60
|
+
}
|
|
61
|
+
};
|
|
62
|
+
for (const ch of cleaned) {
|
|
63
|
+
if (/\s/.test(ch)) {
|
|
64
|
+
flush();
|
|
65
|
+
}
|
|
66
|
+
else if (/[!-/:-@[-`{-~¡-¿]/.test(ch)) {
|
|
67
|
+
// Punctuation becomes its own token.
|
|
68
|
+
flush();
|
|
69
|
+
tokens.push(ch);
|
|
70
|
+
}
|
|
71
|
+
else {
|
|
72
|
+
buf += ch;
|
|
73
|
+
}
|
|
74
|
+
}
|
|
75
|
+
flush();
|
|
76
|
+
return tokens;
|
|
77
|
+
}
|
|
78
|
+
/** Greedy longest-match WordPiece for a single word. */
|
|
79
|
+
wordpiece(word) {
|
|
80
|
+
if (word.length > MAX_INPUT_CHARS_PER_WORD)
|
|
81
|
+
return [UNK];
|
|
82
|
+
const pieces = [];
|
|
83
|
+
let start = 0;
|
|
84
|
+
while (start < word.length) {
|
|
85
|
+
let end = word.length;
|
|
86
|
+
let cur = null;
|
|
87
|
+
while (start < end) {
|
|
88
|
+
let sub = word.slice(start, end);
|
|
89
|
+
if (start > 0)
|
|
90
|
+
sub = "##" + sub;
|
|
91
|
+
if (this.vocab.has(sub)) {
|
|
92
|
+
cur = sub;
|
|
93
|
+
break;
|
|
94
|
+
}
|
|
95
|
+
end--;
|
|
96
|
+
}
|
|
97
|
+
if (cur === null)
|
|
98
|
+
return [UNK]; // any unmatchable piece → whole word is UNK
|
|
99
|
+
pieces.push(cur);
|
|
100
|
+
start = end;
|
|
101
|
+
}
|
|
102
|
+
return pieces;
|
|
103
|
+
}
|
|
104
|
+
/** Tokenize text into WordPiece token strings (no special tokens). */
|
|
105
|
+
tokenize(text) {
|
|
106
|
+
const out = [];
|
|
107
|
+
for (const word of this.basicTokenize(text)) {
|
|
108
|
+
for (const piece of this.wordpiece(word))
|
|
109
|
+
out.push(piece);
|
|
110
|
+
}
|
|
111
|
+
return out;
|
|
112
|
+
}
|
|
113
|
+
/**
|
|
114
|
+
* Encode text into model inputs with [CLS]…[SEP], truncated to `maxLen`.
|
|
115
|
+
* attention_mask is all 1s (no padding for single-sequence inference).
|
|
116
|
+
*/
|
|
117
|
+
encode(text, maxLen = 256) {
|
|
118
|
+
const pieces = this.tokenize(text).slice(0, Math.max(0, maxLen - 2));
|
|
119
|
+
const inputIds = [this.clsId];
|
|
120
|
+
for (const p of pieces)
|
|
121
|
+
inputIds.push(this.vocab.get(p) ?? this.unkId);
|
|
122
|
+
inputIds.push(this.sepId);
|
|
123
|
+
return {
|
|
124
|
+
inputIds,
|
|
125
|
+
attentionMask: inputIds.map(() => 1),
|
|
126
|
+
tokenTypeIds: inputIds.map(() => 0),
|
|
127
|
+
};
|
|
128
|
+
}
|
|
129
|
+
}
|
|
@@ -131,6 +131,9 @@ function readIndex(): { updatedAt: string; summary: unknown; repos: unknown[] }
|
|
|
131
131
|
// Types
|
|
132
132
|
// ---------------------------------------------------------------------------
|
|
133
133
|
|
|
134
|
+
/** Package version of this extension, surfaced in the dashboard header. */
|
|
135
|
+
let dashboardServerVersion = "0.0.0";
|
|
136
|
+
|
|
134
137
|
interface Snapshot {
|
|
135
138
|
version: number;
|
|
136
139
|
updatedAt: string | null;
|
|
@@ -253,6 +256,7 @@ function dashboardHtml(tierName: string): string {
|
|
|
253
256
|
body { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif; background: #0d1117; color: #c9d1d9; padding: 24px; line-height: 1.5; }
|
|
254
257
|
h1 { font-size: 20px; font-weight: 600; margin-bottom: 20px; display: flex; align-items: center; gap: 10px; color: #f0f6fc; }
|
|
255
258
|
h1 .tier { background: #1f6feb; color: #fff; font-size: 11px; font-weight: 700; padding: 2px 8px; border-radius: 10px; text-transform: uppercase; letter-spacing: .5px; }
|
|
259
|
+
h1 .version-pill { background: #30363d; color: #8b949e; font-size: 11px; font-weight: 600; padding: 2px 8px; border-radius: 10px; font-family: ui-monospace, SFMono-Regular, Menlo, monospace; }
|
|
256
260
|
.grid { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; margin-bottom: 20px; }
|
|
257
261
|
.card { background: #161b22; border: 1px solid #30363d; border-radius: 8px; padding: 16px; }
|
|
258
262
|
.card.safe { border-color: #238636; }
|
|
@@ -339,7 +343,7 @@ function dashboardHtml(tierName: string): string {
|
|
|
339
343
|
|
|
340
344
|
<div class="offline-banner" id="offline-banner">Dashboard data unavailable — waiting for a pi session to write snapshot...</div>
|
|
341
345
|
|
|
342
|
-
<h1><span>mega-compact</span><span class="tier">${tierName}</span><span class="model-pill" id="hdr-model">—</span></h1>
|
|
346
|
+
<h1><span>mega-compact</span><span class="tier">${tierName}</span><span class="version-pill">v${dashboardServerVersion}</span><span class="model-pill" id="hdr-model">—</span></h1>
|
|
343
347
|
|
|
344
348
|
<nav class="tabs">
|
|
345
349
|
<button class="tab active" data-tab="current">Current repo</button>
|
|
@@ -772,7 +776,7 @@ export async function launchDashboardServer(stateDir: string): Promise<{ port: n
|
|
|
772
776
|
for (const p of candidates) {
|
|
773
777
|
if (!existsSync(p)) continue;
|
|
774
778
|
const pkg = JSON.parse(readFileSync(p, "utf-8"));
|
|
775
|
-
if (pkg.version) { SERVER_VERSION = pkg.version; break; }
|
|
779
|
+
if (pkg.version) { SERVER_VERSION = pkg.version; dashboardServerVersion = pkg.version; break; }
|
|
776
780
|
}
|
|
777
781
|
} catch { /* non-fatal */ }
|
|
778
782
|
|
|
@@ -12,7 +12,9 @@
|
|
|
12
12
|
import type { ExtensionContext } from "@earendil-works/pi-coding-agent";
|
|
13
13
|
import { sessionEntryToContextMessages } from "@earendil-works/pi-coding-agent";
|
|
14
14
|
import type { AgentMessage } from "@earendil-works/pi-agent-core";
|
|
15
|
-
import { join } from "node:path";
|
|
15
|
+
import { join, dirname } from "node:path";
|
|
16
|
+
import { fileURLToPath } from "node:url";
|
|
17
|
+
import { readFileSync } from "node:fs";
|
|
16
18
|
import { VectorStore } from "../src/vectorStore.js";
|
|
17
19
|
import { toEngineMessages } from "../src/adapt.js";
|
|
18
20
|
import { normalizeSessionId } from "../src/store.js";
|
|
@@ -25,6 +27,22 @@ export const STATUS_KEY = "mega-compact";
|
|
|
25
27
|
export const WIDGET_KEY = "mega-compact-stats";
|
|
26
28
|
export const MARKER_TYPE = "mega-compact-marker";
|
|
27
29
|
|
|
30
|
+
/** Cached npm version, read once from this extension's own package.json. */
|
|
31
|
+
let CACHED_VERSION: string | null = null;
|
|
32
|
+
function ownVersion(): string {
|
|
33
|
+
if (CACHED_VERSION !== null) return CACHED_VERSION;
|
|
34
|
+
let v = "?";
|
|
35
|
+
try {
|
|
36
|
+
const here = dirname(fileURLToPath(import.meta.url)); // .../extensions
|
|
37
|
+
const pkg = JSON.parse(readFileSync(join(here, "..", "package.json"), "utf-8"));
|
|
38
|
+
v = pkg.version ?? "?";
|
|
39
|
+
} catch {
|
|
40
|
+
v = "?";
|
|
41
|
+
}
|
|
42
|
+
CACHED_VERSION = v;
|
|
43
|
+
return v;
|
|
44
|
+
}
|
|
45
|
+
|
|
28
46
|
/** Per-session runtime state kept in the closure (mirrors neuralwatt-mcr). */
|
|
29
47
|
interface SessionRuntime {
|
|
30
48
|
sessionId: string;
|
|
@@ -265,7 +283,7 @@ export class MegaRuntime {
|
|
|
265
283
|
// Phase 3 — pulsing status glyph while a compaction is in flight.
|
|
266
284
|
const pulse = this.pulsing ? `${C.cyan}${PULSE[Math.floor(Date.now() / 250) % PULSE.length]}${C.reset} ` : "";
|
|
267
285
|
const lines = [
|
|
268
|
-
` ${C.amber}⚡ ${this.config.tier}${C.reset} │ ${tokStr}/${maxStr} tokens (${C.bold}${pctStr}${C.reset}) │ ${st.checkpointCount} saved${agentStr}${turnStr}`,
|
|
286
|
+
` ${C.amber}⚡ ${this.config.tier}${C.reset} v${C.bold}${ownVersion()}${C.reset} │ ${tokStr}/${maxStr} tokens (${C.bold}${pctStr}${C.reset}) │ ${st.checkpointCount} saved${agentStr}${turnStr}`,
|
|
269
287
|
` ${triggerLabel} │ ${C.magenta}repeat-skipped: ${dedupStr}${C.reset} │ ${C.gray}memory held:${C.reset} ${usedStr} │ ${C.gray}space freed:${C.reset} ${savedStr}`,
|
|
270
288
|
];
|
|
271
289
|
// Phase 3 — compact progress bar: session tokens saved toward the rolling goal.
|
|
@@ -274,7 +292,10 @@ export class MegaRuntime {
|
|
|
274
292
|
const pct = Math.min(100, Math.round((this.rt.tokensSaved / goal) * 100));
|
|
275
293
|
const filled = Math.round((pct / 100) * 10);
|
|
276
294
|
const bar = "▓".repeat(filled) + "░".repeat(10 - filled);
|
|
277
|
-
|
|
295
|
+
// Session tokens saved, with the repo-wide total held alongside so the
|
|
296
|
+
// bar reads "saved X of goal" and the right side shows saved vs total.
|
|
297
|
+
const totalHeld = st.totalTokenEstimate > 0 ? st.totalTokenEstimate : repo.totalTokenEstimate;
|
|
298
|
+
lines.push(` ${C.green}saved ${fmt(this.rt.tokensSaved)} ${bar}${C.reset} ${pct}% of ${fmt(goal)} ${C.gray}│${C.reset} ${C.blue}${fmt(this.rt.tokensSaved)}${C.reset}/${C.blue}${fmt(totalHeld)}${C.reset} tok held`);
|
|
278
299
|
}
|
|
279
300
|
// Live "now processing" line + why + recent deduped/compacted events,
|
|
280
301
|
// collapsed to ONE rotating line (fresh only). The ticker ring buffer
|
package/package.json
CHANGED