pi-mega-compact 0.7.4 → 0.7.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -6,15 +6,18 @@ sessions into a **local SQLite store** and offers **deduped inline recall** —
6
6
  running **locally inside the extension**, with **no remote MCP server** and
7
7
  **zero network calls at runtime** (PREVENT-PI-004).
8
8
 
9
- > **Current version:** `v0.6.9` — storage backend is **`node:sqlite`**
9
+ > **Current version:** `v0.7.5` — storage backend is **`node:sqlite`**
10
10
  > (`DatabaseSync`, a Node ≥22.13 built-in), replacing the old `better-sqlite3`
11
11
  > native addon and the per-session gzipped JSON checkpoint files. **Zero native
12
- > build step, fully local, zero network at runtime.** Legacy
13
- > `.checkpoints.json.gz` snapshots are retained as disaster-recovery fallbacks
14
- > and auto-imported on first run. The S24 line ties auto-compact, the tier
15
- > label, trim depth, and durable-memory review to one **unified pressure
16
- > signal**, adds a **cross-repo memory-RAG index**, and relieves context
17
- > **during team runs** (not just at the end).
12
+ > build step, fully local, zero network at runtime.** S27 added a **raw-transcript
13
+ > mirror + dedup pipeline** (byte-stable prompt cache via deterministic epoch
14
+ > nonce) and **DB maintenance /commands** (`/mega-db-stats` · `prune` ·
15
+ > `vacuum` · `check` · `reconcile`) plus best-effort auto-maintenance on
16
+ > `session_start`. Legacy `.checkpoints.json.gz` snapshots are retained as
17
+ > disaster-recovery fallbacks and auto-imported on first run. The S24 line ties
18
+ > auto-compact, the tier label, trim depth, and durable-memory review to one
19
+ > **unified pressure signal**, adds a **cross-repo memory-RAG index**, and
20
+ > relieves context **during team runs** (not just at the end).
18
21
 
19
22
  ---
20
23
 
@@ -234,6 +237,11 @@ The commands (slash commands inside pi):
234
237
  | `/mega-view <chkpt\|recent>` | Show a checkpoint's verbatim original region. |
235
238
  | `/mega-help` | Explain the toolbar widget terms (live tier, gate, dedup, tokens saved). |
236
239
  | `/mega-compat-check` | Detect extension conflicts (duplicate commands / overlapping handlers) across installed pi extensions. |
240
+ | `/mega-db-stats` | Show mega-compact SQLite DB stats: table row counts, disk footprint (db + WAL + SHM), page count, freelist %, WAL frames. Read-only; safe any time. |
241
+ | `/mega-db-prune [days]` | DELETE `raw_transcript` + `checkpoint_epochs` rows older than N days (default 30) + orphan `dedup_mirror` rows. Reports deleted counts + reclaimed bytes. |
242
+ | `/mega-db-vacuum` | `VACUUM` the DB (rebuild pages, reclaim freelist). Heavy: briefly doubles disk usage. |
243
+ | `/mega-db-check` | `PRAGMA integrity_check` + `wal_checkpoint(TRUNCATE)`. Fold the WAL into the main file and verify DB health. Use after a crash. |
244
+ | `/mega-db-reconcile` | Fix `dedup_mirror.ref_count` drift vs actual `raw_transcript` refs, delete orphan dedup rows, backfill missing `content_ref`. Run after `/mega-db-prune` or a crash. |
237
245
 
238
246
  The **tier** you see in the toolbar and dashboard is a *live pressure band* (`low` → `medium` → `high` → `ultra` → `mega`) that climbs automatically as your context window fills and falls back as it's relieved — it is driven by `currentTokens / effectiveThreshold`, not a manual setting. The base compaction *threshold* is set by `MEGACOMPACT_TIER` at startup as a **% of the model context window** (`low` 50% · `medium` 60% · `high` 70% · `ultra` 70% · `mega` 75%; default `low`) — the fire point is `tierPct × contextWindow`, so it always lands below pi's native ~80% auto-compaction (any model size). The old static token amounts (50k/100k/200k/1M/10M) are now only the boot fallback used before the first context event reports a window. `/mega-tier` was removed in v0.6.0. Higher pressure also deepens the live trim and reviews durable memory more often — the whole system reacts as one.
239
247
  | `/mega-dashboard` | Start the **localhost-only** live dashboard and open it in a browser (token gauge, store stats, live event stream, per-repo + cross-repo drift). |
@@ -259,6 +267,13 @@ Above the pi editor the extension shows a compact widget:
259
267
  - **Dedup hit rate** — % of checkpoints collapsed as duplicates
260
268
  - **Active agents / turn** — sub-agent count and conversation turn (when > 0)
261
269
 
270
+ > **DB housekeeping** — `/mega-db-stats` / `prune` / `vacuum` / `check` / `reconcile`
271
+ > give you manual control over the SQLite store. In addition, a best-effort
272
+ > **auto-maintenance** pass runs on `session_start`: it prunes rows older than
273
+ > 30d, checkpoints the WAL if it's over 10 MB, and VACUUMs if the DB is over
274
+ > 100 MB AND the freelist is >20% of pages. It never blocks session start and
275
+ > logs a one-line summary to the diagnostic log. (v0.7.5+)
276
+
262
277
  ---
263
278
 
264
279
  ## Configuration (env-backed)
@@ -30,6 +30,7 @@ import { registerEventHandlers } from "./mega-events.js";
30
30
  import { registerCommands } from "./mega-commands.js";
31
31
  import { registerDashboardCommands } from "./mega-dashboard-cmds.js";
32
32
  import { registerConflictCommands } from "./mega-conflict-cmds.js";
33
+ import { registerDbCommands } from "./mega-db-cmds.js";
33
34
  export default function (pi) {
34
35
  const config = loadConfig();
35
36
  const runtime = new MegaRuntime(config);
@@ -37,4 +38,5 @@ export default function (pi) {
37
38
  registerCommands(pi, runtime, config);
38
39
  registerDashboardCommands(pi, runtime);
39
40
  registerConflictCommands(pi, runtime);
41
+ registerDbCommands(pi, runtime);
40
42
  }
@@ -0,0 +1,85 @@
1
+ /**
2
+ * mega-db-cmds.ts — S27 Task 10 DB maintenance /commands.
3
+ *
4
+ * Registers /mega-db-stats, /mega-db-prune, /mega-db-vacuum, /mega-db-check,
5
+ * /mega-db-reconcile slash commands backed by the maintenance primitives in
6
+ * src/store/sqlite.ts. All operations are local SQLite (PREVENT-PI-004) with
7
+ * parameterized queries (PREVENT-002).
8
+ *
9
+ * Auto-maintenance (prune + WAL checkpoint) also runs once per session_start
10
+ * via the wiring in mega-events.ts (best-effort, non-blocking).
11
+ */
12
+ import { getDbStats, pruneOldRows, checkpointWal, vacuumDb, integrityCheck, reconcileDedupMirror, } from "../src/store/sqlite.js";
13
+ /** Format a byte count as a human-readable string (KB / MB / GB). */
14
+ function fmtBytes(n) {
15
+ if (n < 1024)
16
+ return `${n}B`;
17
+ if (n < 1024 * 1024)
18
+ return `${(n / 1024).toFixed(1)}KB`;
19
+ if (n < 1024 * 1024 * 1024)
20
+ return `${(n / (1024 * 1024)).toFixed(1)}MB`;
21
+ return `${(n / (1024 * 1024 * 1024)).toFixed(2)}GB`;
22
+ }
23
+ /** Register the /mega-db-* maintenance commands. */
24
+ export function registerDbCommands(pi, runtime) {
25
+ const stateDir = runtime.currentStateDir;
26
+ pi.registerCommand("mega-db-stats", {
27
+ description: "Show mega-compact SQLite DB stats: table row counts, disk footprint (db + WAL + SHM), page count, freelist, WAL frames.",
28
+ handler: async (_args, ctx) => {
29
+ const s = getDbStats(stateDir);
30
+ ctx.ui.notify(`[mega-compact] DB stats — ${stateDir}`);
31
+ ctx.ui.notify(` main: ${fmtBytes(s.dbBytes)} wal: ${fmtBytes(s.walBytes)} shm: ${fmtBytes(s.shmBytes)}`);
32
+ ctx.ui.notify(` pages: ${s.pageCount} (${s.pageSize}B each), freelist: ${s.freelistPages} (${s.pageCount > 0 ? ((s.freelistPages / s.pageCount) * 100).toFixed(1) : "0"}% reusable), wal frames: ${s.walFrames}`);
33
+ const tableLines = Object.entries(s.tableCounts)
34
+ .sort((a, b) => b[1] - a[1])
35
+ .map(([t, c]) => ` ${t.padEnd(22)} ${String(c).padStart(8)}`);
36
+ if (tableLines.length === 0) {
37
+ ctx.ui.notify(" (no tables populated yet)");
38
+ }
39
+ else {
40
+ ctx.ui.notify(" table row counts:");
41
+ for (const l of tableLines)
42
+ ctx.ui.notify(l);
43
+ }
44
+ },
45
+ });
46
+ pi.registerCommand("mega-db-prune", {
47
+ description: "Prune raw_transcript + checkpoint_epochs + orphan dedup_mirror rows older than N days (default 30). Usage: /mega-db-prune [days]",
48
+ handler: async (args, ctx) => {
49
+ const days = Number.parseInt(args.trim().split(/\s+/)[0] ?? "30", 10);
50
+ const d = Number.isFinite(days) && days > 0 ? days : 30;
51
+ const r = pruneOldRows(stateDir, d);
52
+ ctx.ui.notify(`[mega-compact] ${r.summary} (reclaimed ${fmtBytes(r.reclaimedBytes)})`);
53
+ },
54
+ });
55
+ pi.registerCommand("mega-db-vacuum", {
56
+ description: "VACUUM the mega-compact SQLite DB (rebuild pages, reclaim freelist space). Heavy: briefly doubles disk usage.",
57
+ handler: async (_args, ctx) => {
58
+ const r = vacuumDb(stateDir);
59
+ ctx.ui.notify(`[mega-compact] ${r.summary}`);
60
+ },
61
+ });
62
+ pi.registerCommand("mega-db-check", {
63
+ description: "Run PRAGMA integrity_check + a WAL checkpoint on the mega-compact SQLite DB. Use after a crash or to fold the WAL into the main file.",
64
+ handler: async (_args, ctx) => {
65
+ const lines = integrityCheck(stateDir);
66
+ const healthy = lines.length === 1 && lines[0] === "ok";
67
+ ctx.ui.notify(`[mega-compact] integrity_check: ${healthy ? "✓ ok" : `⚠ ${lines.length} issue(s)`}`);
68
+ if (!healthy) {
69
+ for (const l of lines.slice(0, 10))
70
+ ctx.ui.notify(` ${l}`);
71
+ if (lines.length > 10)
72
+ ctx.ui.notify(` … and ${lines.length - 10} more`);
73
+ }
74
+ const ck = checkpointWal(stateDir);
75
+ ctx.ui.notify(`[mega-compact] ${ck.summary}`);
76
+ },
77
+ });
78
+ pi.registerCommand("mega-db-reconcile", {
79
+ description: "Reconcile dedup_mirror.ref_count vs actual raw_transcript refs: fix drift, delete orphan dedup rows, backfill missing content_ref. Run after /mega-db-prune or a crash.",
80
+ handler: async (_args, ctx) => {
81
+ const r = reconcileDedupMirror(stateDir);
82
+ ctx.ui.notify(`[mega-compact] dedup reconcile: fixed ${r.fixedRefCount} ref_count drift, deleted ${r.orphansDeleted} orphan(s), backfilled ${r.refsBackfilled} content_ref`);
83
+ },
84
+ });
85
+ }
@@ -7,7 +7,7 @@
7
7
  * sync and delegates the heavy lifting to the pipeline + command modules.
8
8
  */
9
9
  import { normalizeSessionId } from "../src/store.js";
10
- import { openStore, appendRawTranscript, writeCheckpointEpoch } from "../src/store/sqlite.js";
10
+ import { openStore, appendRawTranscript, writeCheckpointEpoch, autoMaintain } from "../src/store/sqlite.js";
11
11
  import { epochIdFor } from "../src/mirror/epoch.js";
12
12
  import { autoCompactCheck } from "../src/compact.js";
13
13
  import { estimateSessionTokens, estimateBlockTokens } from "../src/tokens.js";
@@ -116,6 +116,18 @@ export function registerEventHandlers(pi, runtime, config) {
116
116
  runtime.logger.warn("memory-recall skipped", { err: String(err) });
117
117
  }
118
118
  }
119
+ // S27 Task 10: best-effort auto-maintenance on session start (prune rows
120
+ // older than 30d, checkpoint WAL if >10MB, VACUUM if DB >100MB + >20%
121
+ // freelist). Never blocks session start — swallows errors and logs a
122
+ // one-line summary for diagnostics.
123
+ try {
124
+ const m = autoMaintain(runtime.currentStateDir);
125
+ if (m && !m.endsWith("nothing to do"))
126
+ runtime.logger.info("db-auto-maintain", { result: m });
127
+ }
128
+ catch (e) {
129
+ runtime.logger.warn("db-auto-maintain-fail", { error: String(e) });
130
+ }
119
131
  runtime.dashboard.event("session_start", {
120
132
  reason: event.reason,
121
133
  sessionId: runtime.rt.sessionId,
@@ -0,0 +1,158 @@
1
+ /**
2
+ * Tests for S27 Task 10 DB maintenance primitives.
3
+ *
4
+ * Covers getDbStats, pruneOldRows, checkpointWal, vacuumDb, integrityCheck,
5
+ * reconcileDedupMirror, autoMaintain. All against a tmp SQLite store.
6
+ */
7
+ import { describe, it, beforeEach, afterEach } from "node:test";
8
+ import assert from "node:assert/strict";
9
+ import { mkdtempSync, rmSync, statSync } from "node:fs";
10
+ import { join } from "node:path";
11
+ import { tmpdir } from "node:os";
12
+ import { openStore, appendRawTranscript, writeCheckpointEpoch, upsertDedupMirror, updateRawTranscriptRef, getDbStats, pruneOldRows, checkpointWal, vacuumDb, integrityCheck, reconcileDedupMirror, autoMaintain, } from "./sqlite.js";
13
+ function makeTmp() {
14
+ return mkdtempSync(join(tmpdir(), "dbmaint-test-"));
15
+ }
16
+ function makeRow(overrides = {}) {
17
+ return {
18
+ contentHash: `hash-${Math.random().toString(36).slice(2, 10)}`,
19
+ sessionId: "sess-1",
20
+ seq: 0,
21
+ role: "user",
22
+ contentBytes: "hello world",
23
+ toolName: null,
24
+ messageTimestamp: Date.now(),
25
+ checkpointEpoch: "epoch-1",
26
+ ...overrides,
27
+ };
28
+ }
29
+ function makeEpoch(overrides = {}) {
30
+ return {
31
+ epochId: `epoch-${Math.random().toString(36).slice(2, 10)}`,
32
+ sessionId: "sess-1",
33
+ startedSeq: 0,
34
+ committedSeq: 10,
35
+ checkpointId: "cp-1",
36
+ cutIndex: 10,
37
+ summaryMessageText: "test summary",
38
+ createdAt: Date.now(),
39
+ ...overrides,
40
+ };
41
+ }
42
+ describe("DB maintenance primitives (S27 Task 10)", () => {
43
+ let dir;
44
+ let db;
45
+ beforeEach(() => {
46
+ dir = makeTmp();
47
+ db = openStore(dir);
48
+ });
49
+ afterEach(() => {
50
+ rmSync(dir, { recursive: true, force: true });
51
+ });
52
+ describe("getDbStats", () => {
53
+ it("returns zero counts on an empty store", () => {
54
+ const s = getDbStats(dir);
55
+ assert.equal(s.tableCounts.raw_transcript, 0);
56
+ assert.equal(s.tableCounts.checkpoint_epochs, 0);
57
+ assert.equal(s.tableCounts.dedup_mirror, 0);
58
+ // main DB file exists (schema was written)
59
+ assert.ok(s.dbBytes > 0);
60
+ assert.ok(s.pageSize > 0);
61
+ assert.ok(s.pageCount > 0);
62
+ });
63
+ it("counts rows after inserts", () => {
64
+ appendRawTranscript(db, makeRow());
65
+ appendRawTranscript(db, makeRow());
66
+ const s = getDbStats(dir);
67
+ assert.equal(s.tableCounts.raw_transcript, 2);
68
+ });
69
+ });
70
+ describe("pruneOldRows", () => {
71
+ it("deletes nothing when all rows are recent", () => {
72
+ appendRawTranscript(db, makeRow({ messageTimestamp: Date.now() }));
73
+ const r = pruneOldRows(dir, 30);
74
+ assert.equal(r.affected, 0);
75
+ });
76
+ it("deletes raw_transcript rows older than the cutoff", () => {
77
+ const oldTs = Date.now() - 31 * 86_400_000;
78
+ appendRawTranscript(db, makeRow({ messageTimestamp: oldTs }));
79
+ appendRawTranscript(db, makeRow({ messageTimestamp: Date.now() }));
80
+ const r = pruneOldRows(dir, 30);
81
+ assert.equal(r.affected, 1);
82
+ assert.equal(getDbStats(dir).tableCounts.raw_transcript, 1);
83
+ });
84
+ it("deletes checkpoint_epochs older than the cutoff", () => {
85
+ const oldTs = Date.now() - 40 * 86_400_000;
86
+ writeCheckpointEpoch(db, makeEpoch({ createdAt: oldTs }));
87
+ writeCheckpointEpoch(db, makeEpoch({ createdAt: Date.now() }));
88
+ const r = pruneOldRows(dir, 30);
89
+ assert.equal(r.affected, 1);
90
+ assert.equal(getDbStats(dir).tableCounts.checkpoint_epochs, 1);
91
+ });
92
+ });
93
+ describe("integrityCheck", () => {
94
+ it("returns ['ok'] on a healthy DB", () => {
95
+ const lines = integrityCheck(dir);
96
+ assert.deepEqual(lines, ["ok"]);
97
+ });
98
+ });
99
+ describe("checkpointWal", () => {
100
+ it("runs without error and reports checkpointed frames", () => {
101
+ const r = checkpointWal(dir);
102
+ assert.ok(r.summary.includes("wal_checkpoint(TRUNCATE)"));
103
+ });
104
+ });
105
+ describe("vacuumDb", () => {
106
+ it("rebuilds the DB file", () => {
107
+ appendRawTranscript(db, makeRow());
108
+ const r = vacuumDb(dir);
109
+ assert.ok(r.summary.includes("VACUUM"));
110
+ // DB file still exists after vacuum
111
+ assert.ok(statSync(join(dir, "sqlite.db")).size > 0);
112
+ });
113
+ });
114
+ describe("reconcileDedupMirror", () => {
115
+ it("fixes ref_count drift and deletes orphans", () => {
116
+ // Insert a dedup_mirror row with ref_count = 5 (drift: actual refs = 1).
117
+ const hash = "hash-recon-1";
118
+ upsertDedupMirror(db, hash, "content-bytes-1", 0);
119
+ // Force a drift: bump ref_count without a matching raw_transcript row.
120
+ db.prepare("UPDATE dedup_mirror SET ref_count = 5 WHERE content_hash = ?").run(hash);
121
+ // Insert one raw_transcript row pointing at it. appendRawTranscript auto-
122
+ // assigns seq = MAX(seq)+1 = 1 for the first row in this session.
123
+ appendRawTranscript(db, makeRow({ contentHash: "rt-recon-1" }));
124
+ updateRawTranscriptRef(db, "sess-1", 1, hash);
125
+ const r = reconcileDedupMirror(dir);
126
+ // ref_count should now be 1 (one raw_transcript row points at it).
127
+ const dm = db
128
+ .prepare("SELECT ref_count FROM dedup_mirror WHERE content_hash = ?")
129
+ .get(hash);
130
+ assert.equal(dm?.ref_count, 1);
131
+ assert.equal(r.fixedRefCount, 1);
132
+ });
133
+ it("deletes orphan dedup_mirror rows (no raw_transcript refs)", () => {
134
+ // Insert a dedup_mirror row with ref_count > 0 but NO raw_transcript ref.
135
+ const hash = "hash-orphan-1";
136
+ upsertDedupMirror(db, hash, "orphan-content", 0);
137
+ db.prepare("UPDATE dedup_mirror SET ref_count = 3 WHERE content_hash = ?").run(hash);
138
+ const r = reconcileDedupMirror(dir);
139
+ const exists = db
140
+ .prepare("SELECT 1 FROM dedup_mirror WHERE content_hash = ?")
141
+ .get(hash);
142
+ assert.equal(exists, undefined);
143
+ assert.ok(r.orphansDeleted > 0);
144
+ });
145
+ });
146
+ describe("autoMaintain", () => {
147
+ it("runs best-effort and returns a summary string", () => {
148
+ appendRawTranscript(db, makeRow());
149
+ const result = autoMaintain(dir);
150
+ assert.ok(typeof result === "string");
151
+ assert.ok(result.startsWith("auto-maintain:"));
152
+ });
153
+ it("reports nothing to do on a fresh empty DB", () => {
154
+ const result = autoMaintain(dir);
155
+ assert.equal(result, "auto-maintain: nothing to do");
156
+ });
157
+ });
158
+ });
@@ -17,7 +17,7 @@
17
17
  * All queries are parameterized (PREVENT-002) — never string-concatenated.
18
18
  */
19
19
  import { DatabaseSync } from "node:sqlite";
20
- import { existsSync, mkdirSync } from "node:fs";
20
+ import { existsSync, mkdirSync, statSync } from "node:fs";
21
21
  import { homedir, tmpdir } from "node:os";
22
22
  import { join } from "node:path";
23
23
  import { getStateDir } from "../store.js";
@@ -1359,3 +1359,228 @@ export function updateRawTranscriptRef(db, sessionId, seq, contentHash) {
1359
1359
  "@seq": seq,
1360
1360
  });
1361
1361
  }
1362
+ const DB_TABLE_NAMES = [
1363
+ "context_chunks",
1364
+ "session_state",
1365
+ "raw_transcript",
1366
+ "checkpoint_epochs",
1367
+ "dedup_mirror",
1368
+ "memories",
1369
+ "dedup_stats",
1370
+ "daily_log",
1371
+ ];
1372
+ function fileSizeIfExists(path) {
1373
+ try {
1374
+ const st = statSync(path);
1375
+ return st.size;
1376
+ }
1377
+ catch {
1378
+ return 0;
1379
+ }
1380
+ }
1381
+ /**
1382
+ * Gather DB stats for /mega-db-stats: per-table row counts, disk footprint
1383
+ * (main + WAL + SHM), page count, freelist, WAL frame count.
1384
+ *
1385
+ * Read-only: no PRAGMA writes, no VACUUM. Safe to call any time.
1386
+ */
1387
+ export function getDbStats(stateDir = getStateDir()) {
1388
+ const db = openStore(stateDir);
1389
+ const tableCounts = {};
1390
+ for (const t of DB_TABLE_NAMES) {
1391
+ try {
1392
+ const row = db.prepare(`SELECT COUNT(*) AS c FROM ${t}`).get();
1393
+ if (row)
1394
+ tableCounts[t] = row.c;
1395
+ }
1396
+ catch {
1397
+ // Table doesn't exist on this DB (e.g. raw_transcript on a pre-S27 store).
1398
+ // Skip silently — /mega-db-stats lists only tables that exist.
1399
+ }
1400
+ }
1401
+ const pageStat = db.prepare("PRAGMA page_count").get();
1402
+ const freelistStat = db.prepare("PRAGMA freelist_count").get();
1403
+ const pageSizeStat = db.prepare("PRAGMA page_size").get();
1404
+ let walFrames = 0;
1405
+ try {
1406
+ const walInfo = db.prepare("PRAGMA wal_info").get();
1407
+ walFrames = walInfo?.frames ?? 0;
1408
+ }
1409
+ catch {
1410
+ // node:sqlite may not expose wal_info on all versions; not fatal.
1411
+ }
1412
+ const dbPath = join(stateDir, "sqlite.db");
1413
+ return {
1414
+ tableCounts,
1415
+ dbBytes: fileSizeIfExists(dbPath),
1416
+ walBytes: fileSizeIfExists(`${dbPath}-wal`),
1417
+ shmBytes: fileSizeIfExists(`${dbPath}-shm`),
1418
+ pageSize: pageSizeStat?.page_size ?? 0,
1419
+ pageCount: pageStat?.page_count ?? 0,
1420
+ freelistPages: freelistStat?.freelist_count ?? 0,
1421
+ walFrames,
1422
+ };
1423
+ }
1424
+ /**
1425
+ * Prune raw_transcript + checkpoint_epochs rows older than `daysOld`.
1426
+ * Uses `message_timestamp` (raw_transcript) and `created_at` (epochs), both
1427
+ * epoch-ms. Returns the total deleted rows + reclaimed disk bytes.
1428
+ *
1429
+ * PREVENT-002: parameterized. PREVENT-PI-004: local SQLite only.
1430
+ */
1431
+ export function pruneOldRows(stateDir = getStateDir(), daysOld = 30) {
1432
+ const db = openStore(stateDir);
1433
+ const cutoff = Date.now() - daysOld * 86_400_000;
1434
+ const beforeBytes = fileSizeIfExists(join(stateDir, "sqlite.db"));
1435
+ // raw_transcript: message_timestamp may be NULL (pre-S27 rows); those use
1436
+ // the row's insertion order implicitly via seq, so we prune NULL-ts rows
1437
+ // only when the whole session is older than the cutoff (join via session_id
1438
+ // to checkpoint_epochs.created_at). Simpler: prune NULL-ts rows older than
1439
+ // cutoff by falling back to the MIN(created_at) of their epoch.
1440
+ // Delete raw_transcript rows whose message_timestamp is older than cutoff,
1441
+ // OR whose message_timestamp is NULL and the session's latest epoch is older.
1442
+ const delRt = db.prepare(`DELETE FROM raw_transcript
1443
+ WHERE message_timestamp IS NOT NULL AND message_timestamp < ?
1444
+ OR (message_timestamp IS NULL
1445
+ AND session_id IN (
1446
+ SELECT session_id FROM checkpoint_epochs
1447
+ GROUP BY session_id HAVING MAX(created_at) < ?
1448
+ ))`).run(cutoff, cutoff);
1449
+ const rtDeleted = delRt?.changes ?? 0;
1450
+ // checkpoint_epochs: created_at is NOT NULL.
1451
+ const delEp = db.prepare(`DELETE FROM checkpoint_epochs WHERE created_at < ?`).run(cutoff);
1452
+ const epDeleted = delEp?.changes ?? 0;
1453
+ // dedup_mirror: cascade-delete orphan rows whose ref_count has dropped to 0
1454
+ // after the raw_transcript deletes. Safe even if FK is off (raw_transcript has
1455
+ // no FK to dedup_mirror; ref_count is maintained by the dedup pipeline).
1456
+ const delDedup = db.prepare(`DELETE FROM dedup_mirror WHERE ref_count <= 0`).run();
1457
+ const dedupDeleted = delDedup?.changes ?? 0;
1458
+ const afterBytes = fileSizeIfExists(join(stateDir, "sqlite.db"));
1459
+ const total = rtDeleted + epDeleted + dedupDeleted;
1460
+ return {
1461
+ affected: total,
1462
+ reclaimedBytes: Math.max(0, beforeBytes - afterBytes),
1463
+ summary: `pruned ${rtDeleted} raw_transcript + ${epDeleted} epochs + ${dedupDeleted} dedup_mirror rows older than ${daysOld}d`,
1464
+ };
1465
+ }
1466
+ /**
1467
+ * Force a WAL checkpoint (TRUNCATE mode) so the -wal sidecar is reclaimed.
1468
+ * Returns the WAL bytes reclaimed (pre-wal size minus post-wal size).
1469
+ */
1470
+ export function checkpointWal(stateDir = getStateDir()) {
1471
+ const db = openStore(stateDir);
1472
+ const dbPath = join(stateDir, "sqlite.db");
1473
+ const beforeWal = fileSizeIfExists(`${dbPath}-wal`);
1474
+ // PRAGMA wal_checkpoint(TRUNCATE) blocks until all frames are folded into the
1475
+ // main db and the WAL file is truncated to 0 bytes.
1476
+ const res = db.prepare("PRAGMA wal_checkpoint(TRUNCATE)").get();
1477
+ const afterWal = fileSizeIfExists(`${dbPath}-wal`);
1478
+ const reclaimed = Math.max(0, beforeWal - afterWal);
1479
+ return {
1480
+ affected: res?.checkpointed ?? 0,
1481
+ reclaimedBytes: reclaimed,
1482
+ summary: `wal_checkpoint(TRUNCATE): ${res?.checkpointed ?? 0} frames folded, WAL ${beforeWal}→${afterWal} bytes${res?.busy ? " (busy: " + res.busy + ")" : ""}`,
1483
+ };
1484
+ }
1485
+ /**
1486
+ * VACUUM the main DB file (rebuilds pages, reclaims freelist space).
1487
+ * Heavy: briefly doubles disk usage. Run only when freelist is large or the
1488
+ * user explicitly invokes /mega-db-vacuum.
1489
+ */
1490
+ export function vacuumDb(stateDir = getStateDir()) {
1491
+ const db = openStore(stateDir);
1492
+ const dbPath = join(stateDir, "sqlite.db");
1493
+ const beforeBytes = fileSizeIfExists(dbPath);
1494
+ db.exec("VACUUM"); // VACUUM cannot be parameterized; it rewrites the whole DB.
1495
+ const afterBytes = fileSizeIfExists(dbPath);
1496
+ const reclaimed = Math.max(0, beforeBytes - afterBytes);
1497
+ return {
1498
+ affected: 0,
1499
+ reclaimedBytes: reclaimed,
1500
+ summary: `VACUUM: db ${beforeBytes}→${afterBytes} bytes (reclaimed ${reclaimed})`,
1501
+ };
1502
+ }
1503
+ /**
1504
+ * Run `PRAGMA integrity_check` and return the result lines.
1505
+ * Returns ["ok"] when the DB is healthy; otherwise returns the error lines.
1506
+ */
1507
+ export function integrityCheck(stateDir = getStateDir()) {
1508
+ const db = openStore(stateDir);
1509
+ const rows = db.prepare("PRAGMA integrity_check").all();
1510
+ return (rows ?? []).map((r) => r.integrity_check);
1511
+ }
1512
+ /**
1513
+ * Reconcile dedup_mirror vs raw_transcript after pruning or crashes:
1514
+ * 1. Recompute ref_count = COUNT(raw_transcript rows pointing at this hash).
1515
+ * 2. Delete orphan dedup_mirror rows whose recomputed ref_count is 0.
1516
+ * 3. Backfill raw_transcript.content_ref for rows still storing inline bytes.
1517
+ *
1518
+ * Idempotent. Read-modify-write within a single transaction (withTx).
1519
+ */
1520
+ export function reconcileDedupMirror(stateDir = getStateDir()) {
1521
+ const db = openStore(stateDir);
1522
+ const result = { fixedRefCount: 0, orphansDeleted: 0, refsBackfilled: 0 };
1523
+ withTx(db, () => {
1524
+ // 1. Recompute ref_count for every dedup_mirror row from the actual
1525
+ // raw_transcript references.
1526
+ const recompute = db.prepare(`UPDATE dedup_mirror AS dm
1527
+ SET ref_count = COALESCE((
1528
+ SELECT COUNT(*) FROM raw_transcript rt WHERE rt.content_ref = dm.content_hash
1529
+ ), 0)
1530
+ WHERE dm.ref_count != COALESCE((
1531
+ SELECT COUNT(*) FROM raw_transcript rt WHERE rt.content_ref = dm.content_hash
1532
+ ), 0)`).run();
1533
+ result.fixedRefCount = recompute?.changes ?? 0;
1534
+ // 2. Delete orphan dedup_mirror rows (no raw_transcript refs).
1535
+ const delOrphans = db.prepare(`DELETE FROM dedup_mirror
1536
+ WHERE content_hash NOT IN (SELECT DISTINCT content_ref FROM raw_transcript WHERE content_ref IS NOT NULL)`).run();
1537
+ result.orphansDeleted = delOrphans?.changes ?? 0;
1538
+ // 3. Backfill content_ref for rows still storing inline content_bytes (no
1539
+ // ref yet). Only safe when a matching dedup_mirror row exists; otherwise
1540
+ // we'd need to insert one, which is the dedup pipeline's job, not the
1541
+ // reconciler's.
1542
+ const backfill = db.prepare(`UPDATE raw_transcript AS rt
1543
+ SET content_ref = (
1544
+ SELECT dm.content_hash FROM dedup_mirror dm WHERE dm.content_bytes = rt.content_bytes
1545
+ )
1546
+ WHERE rt.content_ref IS NULL
1547
+ AND EXISTS (SELECT 1 FROM dedup_mirror dm WHERE dm.content_bytes = rt.content_bytes)`).run();
1548
+ result.refsBackfilled = backfill?.changes ?? 0;
1549
+ });
1550
+ return result;
1551
+ }
1552
+ /**
1553
+ * One-shot auto-maintenance pass for the session_start hook: prune old rows,
1554
+ * checkpoint the WAL if it's grown large, and (only if the DB is huge) VACUUM.
1555
+ * Best-effort: swallows errors so a session never fails to start over a
1556
+ * housekeeping hiccup. Returns a short summary for the diagnostic log.
1557
+ */
1558
+ export function autoMaintain(stateDir = getStateDir()) {
1559
+ try {
1560
+ const stats = getDbStats(stateDir);
1561
+ const parts = [];
1562
+ // Prune rows older than 30d (default retention).
1563
+ const prune = pruneOldRows(stateDir, 30);
1564
+ if (prune.affected > 0)
1565
+ parts.push(`pruned ${prune.affected}`);
1566
+ // Checkpoint the WAL if it's over 10 MB (avoid pathological WAL growth).
1567
+ if (stats.walBytes > 10 * 1024 * 1024) {
1568
+ const ck = checkpointWal(stateDir);
1569
+ if (ck.reclaimedBytes > 0)
1570
+ parts.push(`wal -${ck.reclaimedBytes}B`);
1571
+ }
1572
+ // VACUUM only if the DB is over 100 MB AND freelist is >20% of pages.
1573
+ if (stats.dbBytes > 100 * 1024 * 1024 &&
1574
+ stats.pageCount > 0 &&
1575
+ stats.freelistPages / stats.pageCount > 0.2) {
1576
+ const v = vacuumDb(stateDir);
1577
+ if (v.reclaimedBytes > 0)
1578
+ parts.push(`vacuum -${v.reclaimedBytes}B`);
1579
+ }
1580
+ return parts.length ? `auto-maintain: ${parts.join(", ")}` : "auto-maintain: nothing to do";
1581
+ }
1582
+ catch (err) {
1583
+ // Never block session start over housekeeping.
1584
+ return `auto-maintain: skipped (${err.message})`;
1585
+ }
1586
+ }
@@ -32,6 +32,7 @@ import { registerEventHandlers } from "./mega-events.js";
32
32
  import { registerCommands } from "./mega-commands.js";
33
33
  import { registerDashboardCommands } from "./mega-dashboard-cmds.js";
34
34
  import { registerConflictCommands } from "./mega-conflict-cmds.js";
35
+ import { registerDbCommands } from "./mega-db-cmds.js";
35
36
 
36
37
  export default function (pi: ExtensionAPI) {
37
38
  const config = loadConfig();
@@ -40,4 +41,5 @@ export default function (pi: ExtensionAPI) {
40
41
  registerCommands(pi, runtime, config);
41
42
  registerDashboardCommands(pi, runtime);
42
43
  registerConflictCommands(pi, runtime);
44
+ registerDbCommands(pi, runtime);
43
45
  }
@@ -0,0 +1,107 @@
1
+ /**
2
+ * mega-db-cmds.ts — S27 Task 10 DB maintenance /commands.
3
+ *
4
+ * Registers /mega-db-stats, /mega-db-prune, /mega-db-vacuum, /mega-db-check,
5
+ * /mega-db-reconcile slash commands backed by the maintenance primitives in
6
+ * src/store/sqlite.ts. All operations are local SQLite (PREVENT-PI-004) with
7
+ * parameterized queries (PREVENT-002).
8
+ *
9
+ * Auto-maintenance (prune + WAL checkpoint) also runs once per session_start
10
+ * via the wiring in mega-events.ts (best-effort, non-blocking).
11
+ */
12
+
13
+ import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent";
14
+ import type { MegaRuntime } from "./mega-runtime.js";
15
+ import {
16
+ getDbStats,
17
+ pruneOldRows,
18
+ checkpointWal,
19
+ vacuumDb,
20
+ integrityCheck,
21
+ reconcileDedupMirror,
22
+ type DedupReconcileResult,
23
+ } from "../src/store/sqlite.js";
24
+
25
+ /** Format a byte count as a human-readable string (KB / MB / GB). */
26
+ function fmtBytes(n: number): string {
27
+ if (n < 1024) return `${n}B`;
28
+ if (n < 1024 * 1024) return `${(n / 1024).toFixed(1)}KB`;
29
+ if (n < 1024 * 1024 * 1024) return `${(n / (1024 * 1024)).toFixed(1)}MB`;
30
+ return `${(n / (1024 * 1024 * 1024)).toFixed(2)}GB`;
31
+ }
32
+
33
+ /** Register the /mega-db-* maintenance commands. */
34
+ export function registerDbCommands(pi: ExtensionAPI, runtime: MegaRuntime): void {
35
+ const stateDir = runtime.currentStateDir;
36
+
37
+ pi.registerCommand("mega-db-stats", {
38
+ description:
39
+ "Show mega-compact SQLite DB stats: table row counts, disk footprint (db + WAL + SHM), page count, freelist, WAL frames.",
40
+ handler: async (_args: string, ctx: ExtensionContext) => {
41
+ const s = getDbStats(stateDir);
42
+ ctx.ui.notify(`[mega-compact] DB stats — ${stateDir}`);
43
+ ctx.ui.notify(` main: ${fmtBytes(s.dbBytes)} wal: ${fmtBytes(s.walBytes)} shm: ${fmtBytes(s.shmBytes)}`);
44
+ ctx.ui.notify(
45
+ ` pages: ${s.pageCount} (${s.pageSize}B each), freelist: ${s.freelistPages} (${s.pageCount > 0 ? ((s.freelistPages / s.pageCount) * 100).toFixed(1) : "0"}% reusable), wal frames: ${s.walFrames}`,
46
+ );
47
+ const tableLines = Object.entries(s.tableCounts)
48
+ .sort((a, b) => b[1] - a[1])
49
+ .map(([t, c]) => ` ${t.padEnd(22)} ${String(c).padStart(8)}`);
50
+ if (tableLines.length === 0) {
51
+ ctx.ui.notify(" (no tables populated yet)");
52
+ } else {
53
+ ctx.ui.notify(" table row counts:");
54
+ for (const l of tableLines) ctx.ui.notify(l);
55
+ }
56
+ },
57
+ });
58
+
59
+ pi.registerCommand("mega-db-prune", {
60
+ description:
61
+ "Prune raw_transcript + checkpoint_epochs + orphan dedup_mirror rows older than N days (default 30). Usage: /mega-db-prune [days]",
62
+ handler: async (args: string, ctx: ExtensionContext) => {
63
+ const days = Number.parseInt(args.trim().split(/\s+/)[0] ?? "30", 10);
64
+ const d = Number.isFinite(days) && days > 0 ? days : 30;
65
+ const r = pruneOldRows(stateDir, d);
66
+ ctx.ui.notify(`[mega-compact] ${r.summary} (reclaimed ${fmtBytes(r.reclaimedBytes)})`);
67
+ },
68
+ });
69
+
70
+ pi.registerCommand("mega-db-vacuum", {
71
+ description:
72
+ "VACUUM the mega-compact SQLite DB (rebuild pages, reclaim freelist space). Heavy: briefly doubles disk usage.",
73
+ handler: async (_args: string, ctx: ExtensionContext) => {
74
+ const r = vacuumDb(stateDir);
75
+ ctx.ui.notify(`[mega-compact] ${r.summary}`);
76
+ },
77
+ });
78
+
79
+ pi.registerCommand("mega-db-check", {
80
+ description:
81
+ "Run PRAGMA integrity_check + a WAL checkpoint on the mega-compact SQLite DB. Use after a crash or to fold the WAL into the main file.",
82
+ handler: async (_args: string, ctx: ExtensionContext) => {
83
+ const lines = integrityCheck(stateDir);
84
+ const healthy = lines.length === 1 && lines[0] === "ok";
85
+ ctx.ui.notify(
86
+ `[mega-compact] integrity_check: ${healthy ? "✓ ok" : `⚠ ${lines.length} issue(s)`}`,
87
+ );
88
+ if (!healthy) {
89
+ for (const l of lines.slice(0, 10)) ctx.ui.notify(` ${l}`);
90
+ if (lines.length > 10) ctx.ui.notify(` … and ${lines.length - 10} more`);
91
+ }
92
+ const ck = checkpointWal(stateDir);
93
+ ctx.ui.notify(`[mega-compact] ${ck.summary}`);
94
+ },
95
+ });
96
+
97
+ pi.registerCommand("mega-db-reconcile", {
98
+ description:
99
+ "Reconcile dedup_mirror.ref_count vs actual raw_transcript refs: fix drift, delete orphan dedup rows, backfill missing content_ref. Run after /mega-db-prune or a crash.",
100
+ handler: async (_args: string, ctx: ExtensionContext) => {
101
+ const r: DedupReconcileResult = reconcileDedupMirror(stateDir);
102
+ ctx.ui.notify(
103
+ `[mega-compact] dedup reconcile: fixed ${r.fixedRefCount} ref_count drift, deleted ${r.orphansDeleted} orphan(s), backfilled ${r.refsBackfilled} content_ref`,
104
+ );
105
+ },
106
+ });
107
+ }
@@ -16,7 +16,7 @@ import type {
16
16
  } from "@earendil-works/pi-coding-agent";
17
17
  import type { AgentMessage } from "@earendil-works/pi-agent-core";
18
18
  import { normalizeSessionId } from "../src/store.js";
19
- import { openStore, appendRawTranscript, writeCheckpointEpoch, type CheckpointEpoch } from "../src/store/sqlite.js";
19
+ import { openStore, appendRawTranscript, writeCheckpointEpoch, autoMaintain, type CheckpointEpoch } from "../src/store/sqlite.js";
20
20
  import { epochIdFor } from "../src/mirror/epoch.js";
21
21
  import { autoCompactCheck } from "../src/compact.js";
22
22
  import { estimateSessionTokens, estimateBlockTokens } from "../src/tokens.js";
@@ -158,6 +158,16 @@ export function registerEventHandlers(
158
158
  runtime.logger.warn("memory-recall skipped", { err: String(err) });
159
159
  }
160
160
  }
161
+ // S27 Task 10: best-effort auto-maintenance on session start (prune rows
162
+ // older than 30d, checkpoint WAL if >10MB, VACUUM if DB >100MB + >20%
163
+ // freelist). Never blocks session start — swallows errors and logs a
164
+ // one-line summary for diagnostics.
165
+ try {
166
+ const m = autoMaintain(runtime.currentStateDir);
167
+ if (m && !m.endsWith("nothing to do")) runtime.logger.info("db-auto-maintain", { result: m });
168
+ } catch (e) {
169
+ runtime.logger.warn("db-auto-maintain-fail", { error: String(e) });
170
+ }
161
171
  runtime.dashboard.event("session_start", {
162
172
  reason: event.reason,
163
173
  sessionId: runtime.rt.sessionId,
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pi-mega-compact",
3
- "version": "0.7.4",
3
+ "version": "0.7.6",
4
4
  "description": "Layered, local, vector-backed context compressor for pi — supersede/collapse/cluster compaction with deduped inline recall.",
5
5
  "type": "module",
6
6
  "license": "BSD-2-Clause",
@@ -0,0 +1,193 @@
1
+ /**
2
+ * Tests for S27 Task 10 DB maintenance primitives.
3
+ *
4
+ * Covers getDbStats, pruneOldRows, checkpointWal, vacuumDb, integrityCheck,
5
+ * reconcileDedupMirror, autoMaintain. All against a tmp SQLite store.
6
+ */
7
+ import { describe, it, beforeEach, afterEach } from "node:test";
8
+ import assert from "node:assert/strict";
9
+ import { mkdtempSync, rmSync, statSync } from "node:fs";
10
+ import { join } from "node:path";
11
+ import { tmpdir } from "node:os";
12
+ import {
13
+ openStore,
14
+ appendRawTranscript,
15
+ writeCheckpointEpoch,
16
+ upsertDedupMirror,
17
+ updateRawTranscriptRef,
18
+ getDbStats,
19
+ pruneOldRows,
20
+ checkpointWal,
21
+ vacuumDb,
22
+ integrityCheck,
23
+ reconcileDedupMirror,
24
+ autoMaintain,
25
+ type RawTranscriptRow,
26
+ type CheckpointEpoch,
27
+ } from "./sqlite.js";
28
+
29
+ function makeTmp(): string {
30
+ return mkdtempSync(join(tmpdir(), "dbmaint-test-"));
31
+ }
32
+
33
+ function makeRow(overrides: Partial<RawTranscriptRow> = {}): RawTranscriptRow {
34
+ return {
35
+ contentHash: `hash-${Math.random().toString(36).slice(2, 10)}`,
36
+ sessionId: "sess-1",
37
+ seq: 0,
38
+ role: "user",
39
+ contentBytes: "hello world",
40
+ toolName: null,
41
+ messageTimestamp: Date.now(),
42
+ checkpointEpoch: "epoch-1",
43
+ ...overrides,
44
+ };
45
+ }
46
+
47
+ function makeEpoch(overrides: Partial<CheckpointEpoch> = {}): CheckpointEpoch {
48
+ return {
49
+ epochId: `epoch-${Math.random().toString(36).slice(2, 10)}`,
50
+ sessionId: "sess-1",
51
+ startedSeq: 0,
52
+ committedSeq: 10,
53
+ checkpointId: "cp-1",
54
+ cutIndex: 10,
55
+ summaryMessageText: "test summary",
56
+ createdAt: Date.now(),
57
+ ...overrides,
58
+ };
59
+ }
60
+
61
+ describe("DB maintenance primitives (S27 Task 10)", () => {
62
+ let dir: string;
63
+ let db: ReturnType<typeof openStore>;
64
+
65
+ beforeEach(() => {
66
+ dir = makeTmp();
67
+ db = openStore(dir);
68
+ });
69
+
70
+ afterEach(() => {
71
+ rmSync(dir, { recursive: true, force: true });
72
+ });
73
+
74
+ describe("getDbStats", () => {
75
+ it("returns zero counts on an empty store", () => {
76
+ const s = getDbStats(dir);
77
+ assert.equal(s.tableCounts.raw_transcript, 0);
78
+ assert.equal(s.tableCounts.checkpoint_epochs, 0);
79
+ assert.equal(s.tableCounts.dedup_mirror, 0);
80
+ // main DB file exists (schema was written)
81
+ assert.ok(s.dbBytes > 0);
82
+ assert.ok(s.pageSize > 0);
83
+ assert.ok(s.pageCount > 0);
84
+ });
85
+
86
+ it("counts rows after inserts", () => {
87
+ appendRawTranscript(db, makeRow());
88
+ appendRawTranscript(db, makeRow());
89
+ const s = getDbStats(dir);
90
+ assert.equal(s.tableCounts.raw_transcript, 2);
91
+ });
92
+ });
93
+
94
+ describe("pruneOldRows", () => {
95
+ it("deletes nothing when all rows are recent", () => {
96
+ appendRawTranscript(db, makeRow({ messageTimestamp: Date.now() }));
97
+ const r = pruneOldRows(dir, 30);
98
+ assert.equal(r.affected, 0);
99
+ });
100
+
101
+ it("deletes raw_transcript rows older than the cutoff", () => {
102
+ const oldTs = Date.now() - 31 * 86_400_000;
103
+ appendRawTranscript(db, makeRow({ messageTimestamp: oldTs }));
104
+ appendRawTranscript(db, makeRow({ messageTimestamp: Date.now() }));
105
+ const r = pruneOldRows(dir, 30);
106
+ assert.equal(r.affected, 1);
107
+ assert.equal(getDbStats(dir).tableCounts.raw_transcript, 1);
108
+ });
109
+
110
+ it("deletes checkpoint_epochs older than the cutoff", () => {
111
+ const oldTs = Date.now() - 40 * 86_400_000;
112
+ writeCheckpointEpoch(db, makeEpoch({ createdAt: oldTs }));
113
+ writeCheckpointEpoch(db, makeEpoch({ createdAt: Date.now() }));
114
+ const r = pruneOldRows(dir, 30);
115
+ assert.equal(r.affected, 1);
116
+ assert.equal(getDbStats(dir).tableCounts.checkpoint_epochs, 1);
117
+ });
118
+ });
119
+
120
+ describe("integrityCheck", () => {
121
+ it("returns ['ok'] on a healthy DB", () => {
122
+ const lines = integrityCheck(dir);
123
+ assert.deepEqual(lines, ["ok"]);
124
+ });
125
+ });
126
+
127
+ describe("checkpointWal", () => {
128
+ it("runs without error and reports checkpointed frames", () => {
129
+ const r = checkpointWal(dir);
130
+ assert.ok(r.summary.includes("wal_checkpoint(TRUNCATE)"));
131
+ });
132
+ });
133
+
134
+ describe("vacuumDb", () => {
135
+ it("rebuilds the DB file", () => {
136
+ appendRawTranscript(db, makeRow());
137
+ const r = vacuumDb(dir);
138
+ assert.ok(r.summary.includes("VACUUM"));
139
+ // DB file still exists after vacuum
140
+ assert.ok(statSync(join(dir, "sqlite.db")).size > 0);
141
+ });
142
+ });
143
+
144
+ describe("reconcileDedupMirror", () => {
145
+ it("fixes ref_count drift and deletes orphans", () => {
146
+ // Insert a dedup_mirror row with ref_count = 5 (drift: actual refs = 1).
147
+ const hash = "hash-recon-1";
148
+ upsertDedupMirror(db, hash, "content-bytes-1", 0);
149
+ // Force a drift: bump ref_count without a matching raw_transcript row.
150
+ db.prepare("UPDATE dedup_mirror SET ref_count = 5 WHERE content_hash = ?").run(hash);
151
+ // Insert one raw_transcript row pointing at it. appendRawTranscript auto-
152
+ // assigns seq = MAX(seq)+1 = 1 for the first row in this session.
153
+ appendRawTranscript(db, makeRow({ contentHash: "rt-recon-1" }));
154
+ updateRawTranscriptRef(db, "sess-1", 1, hash);
155
+
156
+ const r = reconcileDedupMirror(dir);
157
+ // ref_count should now be 1 (one raw_transcript row points at it).
158
+ const dm = db
159
+ .prepare("SELECT ref_count FROM dedup_mirror WHERE content_hash = ?")
160
+ .get(hash) as { ref_count: number } | undefined;
161
+ assert.equal(dm?.ref_count, 1);
162
+ assert.equal(r.fixedRefCount, 1);
163
+ });
164
+
165
+ it("deletes orphan dedup_mirror rows (no raw_transcript refs)", () => {
166
+ // Insert a dedup_mirror row with ref_count > 0 but NO raw_transcript ref.
167
+ const hash = "hash-orphan-1";
168
+ upsertDedupMirror(db, hash, "orphan-content", 0);
169
+ db.prepare("UPDATE dedup_mirror SET ref_count = 3 WHERE content_hash = ?").run(hash);
170
+
171
+ const r = reconcileDedupMirror(dir);
172
+ const exists = db
173
+ .prepare("SELECT 1 FROM dedup_mirror WHERE content_hash = ?")
174
+ .get(hash);
175
+ assert.equal(exists, undefined);
176
+ assert.ok(r.orphansDeleted > 0);
177
+ });
178
+ });
179
+
180
+ describe("autoMaintain", () => {
181
+ it("runs best-effort and returns a summary string", () => {
182
+ appendRawTranscript(db, makeRow());
183
+ const result = autoMaintain(dir);
184
+ assert.ok(typeof result === "string");
185
+ assert.ok(result.startsWith("auto-maintain:"));
186
+ });
187
+
188
+ it("reports nothing to do on a fresh empty DB", () => {
189
+ const result = autoMaintain(dir);
190
+ assert.equal(result, "auto-maintain: nothing to do");
191
+ });
192
+ });
193
+ });
@@ -18,7 +18,7 @@
18
18
  */
19
19
 
20
20
  import { DatabaseSync } from "node:sqlite";
21
- import { existsSync, mkdirSync } from "node:fs";
21
+ import { existsSync, mkdirSync, statSync } from "node:fs";
22
22
  import { homedir, tmpdir } from "node:os";
23
23
  import { join } from "node:path";
24
24
  import { getStateDir } from "../store.js";
@@ -1886,3 +1886,293 @@ export function updateRawTranscriptRef(
1886
1886
  "@seq": seq,
1887
1887
  });
1888
1888
  }
1889
+
1890
+ // ---------------------------------------------------------------------------
1891
+ // S27 Task 10 — DB maintenance / housekeeping primitives.
1892
+ // All pi-agnostic, all parameterized (PREVENT-002), all local (PREVENT-PI-004).
1893
+ // Exposed via the /mega-db-* slash commands in extensions/mega-db-cmds.ts.
1894
+ // ---------------------------------------------------------------------------
1895
+
1896
+ /** Per-table row counts + DB file sizes for the /mega-db-stats command. */
1897
+ export interface DbStats {
1898
+ /** Row count per table (keys are table names that exist in this DB). */
1899
+ tableCounts: Record<string, number>;
1900
+ /** Bytes used by the main DB file on disk. */
1901
+ dbBytes: number;
1902
+ /** Bytes used by the -wal sidecar file (0 if absent). */
1903
+ walBytes: number;
1904
+ /** Bytes used by the -shm sidecar file (0 if absent). */
1905
+ shmBytes: number;
1906
+ /** SQLite page size in bytes. */
1907
+ pageSize: number;
1908
+ /** Total pages (freelist + in-use). */
1909
+ pageCount: number;
1910
+ /** Freelist pages (reusable by VACUUM). */
1911
+ freelistPages: number;
1912
+ /** WAL frame count from PRAGMA wal_info (best-effort; 0 if unsupported). */
1913
+ walFrames: number;
1914
+ }
1915
+
1916
+ const DB_TABLE_NAMES = [
1917
+ "context_chunks",
1918
+ "session_state",
1919
+ "raw_transcript",
1920
+ "checkpoint_epochs",
1921
+ "dedup_mirror",
1922
+ "memories",
1923
+ "dedup_stats",
1924
+ "daily_log",
1925
+ ] as const;
1926
+
1927
+ function fileSizeIfExists(path: string): number {
1928
+ try {
1929
+ const st = statSync(path);
1930
+ return st.size;
1931
+ } catch {
1932
+ return 0;
1933
+ }
1934
+ }
1935
+
1936
+ /**
1937
+ * Gather DB stats for /mega-db-stats: per-table row counts, disk footprint
1938
+ * (main + WAL + SHM), page count, freelist, WAL frame count.
1939
+ *
1940
+ * Read-only: no PRAGMA writes, no VACUUM. Safe to call any time.
1941
+ */
1942
+ export function getDbStats(stateDir: string = getStateDir()): DbStats {
1943
+ const db = openStore(stateDir);
1944
+ const tableCounts: Record<string, number> = {};
1945
+ for (const t of DB_TABLE_NAMES) {
1946
+ try {
1947
+ const row = db.prepare(`SELECT COUNT(*) AS c FROM ${t}`).get() as { c: number } | undefined;
1948
+ if (row) tableCounts[t] = row.c;
1949
+ } catch {
1950
+ // Table doesn't exist on this DB (e.g. raw_transcript on a pre-S27 store).
1951
+ // Skip silently — /mega-db-stats lists only tables that exist.
1952
+ }
1953
+ }
1954
+ const pageStat = db.prepare("PRAGMA page_count").get() as { page_count?: number } | undefined;
1955
+ const freelistStat = db.prepare("PRAGMA freelist_count").get() as { freelist_count?: number } | undefined;
1956
+ const pageSizeStat = db.prepare("PRAGMA page_size").get() as { page_size?: number } | undefined;
1957
+ let walFrames = 0;
1958
+ try {
1959
+ const walInfo = db.prepare("PRAGMA wal_info").get() as { frames?: number } | undefined;
1960
+ walFrames = walInfo?.frames ?? 0;
1961
+ } catch {
1962
+ // node:sqlite may not expose wal_info on all versions; not fatal.
1963
+ }
1964
+ const dbPath = join(stateDir, "sqlite.db");
1965
+ return {
1966
+ tableCounts,
1967
+ dbBytes: fileSizeIfExists(dbPath),
1968
+ walBytes: fileSizeIfExists(`${dbPath}-wal`),
1969
+ shmBytes: fileSizeIfExists(`${dbPath}-shm`),
1970
+ pageSize: pageSizeStat?.page_size ?? 0,
1971
+ pageCount: pageStat?.page_count ?? 0,
1972
+ freelistPages: freelistStat?.freelist_count ?? 0,
1973
+ walFrames,
1974
+ };
1975
+ }
1976
+
1977
+ /** Result of a prune / VACUUM / checkpoint operation (reclaimed bytes). */
1978
+ export interface MaintenanceResult {
1979
+ /** Rows deleted (prune) or pages reclaimed (VACUUM / checkpoint). */
1980
+ affected: number;
1981
+ /** Bytes reclaimed on disk (best-effort: post-op size minus pre-op size). */
1982
+ reclaimedBytes: number;
1983
+ /** Human-readable summary line for the command output. */
1984
+ summary: string;
1985
+ }
1986
+
1987
+ /**
1988
+ * Prune raw_transcript + checkpoint_epochs rows older than `daysOld`.
1989
+ * Uses `message_timestamp` (raw_transcript) and `created_at` (epochs), both
1990
+ * epoch-ms. Returns the total deleted rows + reclaimed disk bytes.
1991
+ *
1992
+ * PREVENT-002: parameterized. PREVENT-PI-004: local SQLite only.
1993
+ */
1994
+ export function pruneOldRows(stateDir: string = getStateDir(), daysOld = 30): MaintenanceResult {
1995
+ const db = openStore(stateDir);
1996
+ const cutoff = Date.now() - daysOld * 86_400_000;
1997
+ const beforeBytes = fileSizeIfExists(join(stateDir, "sqlite.db"));
1998
+ // raw_transcript: message_timestamp may be NULL (pre-S27 rows); those use
1999
+ // the row's insertion order implicitly via seq, so we prune NULL-ts rows
2000
+ // only when the whole session is older than the cutoff (join via session_id
2001
+ // to checkpoint_epochs.created_at). Simpler: prune NULL-ts rows older than
2002
+ // cutoff by falling back to the MIN(created_at) of their epoch.
2003
+ // Delete raw_transcript rows whose message_timestamp is older than cutoff,
2004
+ // OR whose message_timestamp is NULL and the session's latest epoch is older.
2005
+ const delRt = db.prepare(
2006
+ `DELETE FROM raw_transcript
2007
+ WHERE message_timestamp IS NOT NULL AND message_timestamp < ?
2008
+ OR (message_timestamp IS NULL
2009
+ AND session_id IN (
2010
+ SELECT session_id FROM checkpoint_epochs
2011
+ GROUP BY session_id HAVING MAX(created_at) < ?
2012
+ ))`,
2013
+ ).run(cutoff, cutoff) as { changes?: number } | undefined;
2014
+ const rtDeleted = delRt?.changes ?? 0;
2015
+ // checkpoint_epochs: created_at is NOT NULL.
2016
+ const delEp = db.prepare(`DELETE FROM checkpoint_epochs WHERE created_at < ?`).run(cutoff) as {
2017
+ changes?: number;
2018
+ } | undefined;
2019
+ const epDeleted = delEp?.changes ?? 0;
2020
+ // dedup_mirror: cascade-delete orphan rows whose ref_count has dropped to 0
2021
+ // after the raw_transcript deletes. Safe even if FK is off (raw_transcript has
2022
+ // no FK to dedup_mirror; ref_count is maintained by the dedup pipeline).
2023
+ const delDedup = db.prepare(`DELETE FROM dedup_mirror WHERE ref_count <= 0`).run() as {
2024
+ changes?: number;
2025
+ } | undefined;
2026
+ const dedupDeleted = delDedup?.changes ?? 0;
2027
+ const afterBytes = fileSizeIfExists(join(stateDir, "sqlite.db"));
2028
+ const total = rtDeleted + epDeleted + dedupDeleted;
2029
+ return {
2030
+ affected: total,
2031
+ reclaimedBytes: Math.max(0, beforeBytes - afterBytes),
2032
+ summary: `pruned ${rtDeleted} raw_transcript + ${epDeleted} epochs + ${dedupDeleted} dedup_mirror rows older than ${daysOld}d`,
2033
+ };
2034
+ }
2035
+
2036
+ /**
2037
+ * Force a WAL checkpoint (TRUNCATE mode) so the -wal sidecar is reclaimed.
2038
+ * Returns the WAL bytes reclaimed (pre-wal size minus post-wal size).
2039
+ */
2040
+ export function checkpointWal(stateDir: string = getStateDir()): MaintenanceResult {
2041
+ const db = openStore(stateDir);
2042
+ const dbPath = join(stateDir, "sqlite.db");
2043
+ const beforeWal = fileSizeIfExists(`${dbPath}-wal`);
2044
+ // PRAGMA wal_checkpoint(TRUNCATE) blocks until all frames are folded into the
2045
+ // main db and the WAL file is truncated to 0 bytes.
2046
+ const res = db.prepare("PRAGMA wal_checkpoint(TRUNCATE)").get() as {
2047
+ busy?: number;
2048
+ log?: number;
2049
+ checkpointed?: number;
2050
+ } | undefined;
2051
+ const afterWal = fileSizeIfExists(`${dbPath}-wal`);
2052
+ const reclaimed = Math.max(0, beforeWal - afterWal);
2053
+ return {
2054
+ affected: res?.checkpointed ?? 0,
2055
+ reclaimedBytes: reclaimed,
2056
+ summary: `wal_checkpoint(TRUNCATE): ${res?.checkpointed ?? 0} frames folded, WAL ${beforeWal}→${afterWal} bytes${res?.busy ? " (busy: " + res.busy + ")" : ""}`,
2057
+ };
2058
+ }
2059
+
2060
+ /**
2061
+ * VACUUM the main DB file (rebuilds pages, reclaims freelist space).
2062
+ * Heavy: briefly doubles disk usage. Run only when freelist is large or the
2063
+ * user explicitly invokes /mega-db-vacuum.
2064
+ */
2065
+ export function vacuumDb(stateDir: string = getStateDir()): MaintenanceResult {
2066
+ const db = openStore(stateDir);
2067
+ const dbPath = join(stateDir, "sqlite.db");
2068
+ const beforeBytes = fileSizeIfExists(dbPath);
2069
+ db.exec("VACUUM"); // VACUUM cannot be parameterized; it rewrites the whole DB.
2070
+ const afterBytes = fileSizeIfExists(dbPath);
2071
+ const reclaimed = Math.max(0, beforeBytes - afterBytes);
2072
+ return {
2073
+ affected: 0,
2074
+ reclaimedBytes: reclaimed,
2075
+ summary: `VACUUM: db ${beforeBytes}→${afterBytes} bytes (reclaimed ${reclaimed})`,
2076
+ };
2077
+ }
2078
+
2079
+ /**
2080
+ * Run `PRAGMA integrity_check` and return the result lines.
2081
+ * Returns ["ok"] when the DB is healthy; otherwise returns the error lines.
2082
+ */
2083
+ export function integrityCheck(stateDir: string = getStateDir()): string[] {
2084
+ const db = openStore(stateDir);
2085
+ const rows = db.prepare("PRAGMA integrity_check").all() as Array<{ integrity_check: string }> | undefined;
2086
+ return (rows ?? []).map((r) => r.integrity_check);
2087
+ }
2088
+
2089
+ /** Reconcile drift in dedup_mirror.ref_count vs actual raw_transcript refs. */
2090
+ export interface DedupReconcileResult {
2091
+ /** Rows whose ref_count was corrected. */
2092
+ fixedRefCount: number;
2093
+ /** Orphan dedup_mirror rows (content_hash with 0 raw_transcript refs) deleted. */
2094
+ orphansDeleted: number;
2095
+ /** raw_transcript rows whose content_ref was NULL but now set (backfill). */
2096
+ refsBackfilled: number;
2097
+ }
2098
+
2099
+ /**
2100
+ * Reconcile dedup_mirror vs raw_transcript after pruning or crashes:
2101
+ * 1. Recompute ref_count = COUNT(raw_transcript rows pointing at this hash).
2102
+ * 2. Delete orphan dedup_mirror rows whose recomputed ref_count is 0.
2103
+ * 3. Backfill raw_transcript.content_ref for rows still storing inline bytes.
2104
+ *
2105
+ * Idempotent. Read-modify-write within a single transaction (withTx).
2106
+ */
2107
+ export function reconcileDedupMirror(stateDir: string = getStateDir()): DedupReconcileResult {
2108
+ const db = openStore(stateDir);
2109
+ const result: DedupReconcileResult = { fixedRefCount: 0, orphansDeleted: 0, refsBackfilled: 0 };
2110
+ withTx(db, () => {
2111
+ // 1. Recompute ref_count for every dedup_mirror row from the actual
2112
+ // raw_transcript references.
2113
+ const recompute = db.prepare(
2114
+ `UPDATE dedup_mirror AS dm
2115
+ SET ref_count = COALESCE((
2116
+ SELECT COUNT(*) FROM raw_transcript rt WHERE rt.content_ref = dm.content_hash
2117
+ ), 0)
2118
+ WHERE dm.ref_count != COALESCE((
2119
+ SELECT COUNT(*) FROM raw_transcript rt WHERE rt.content_ref = dm.content_hash
2120
+ ), 0)`,
2121
+ ).run() as { changes?: number } | undefined;
2122
+ result.fixedRefCount = recompute?.changes ?? 0;
2123
+ // 2. Delete orphan dedup_mirror rows (no raw_transcript refs).
2124
+ const delOrphans = db.prepare(
2125
+ `DELETE FROM dedup_mirror
2126
+ WHERE content_hash NOT IN (SELECT DISTINCT content_ref FROM raw_transcript WHERE content_ref IS NOT NULL)`,
2127
+ ).run() as { changes?: number } | undefined;
2128
+ result.orphansDeleted = delOrphans?.changes ?? 0;
2129
+ // 3. Backfill content_ref for rows still storing inline content_bytes (no
2130
+ // ref yet). Only safe when a matching dedup_mirror row exists; otherwise
2131
+ // we'd need to insert one, which is the dedup pipeline's job, not the
2132
+ // reconciler's.
2133
+ const backfill = db.prepare(
2134
+ `UPDATE raw_transcript AS rt
2135
+ SET content_ref = (
2136
+ SELECT dm.content_hash FROM dedup_mirror dm WHERE dm.content_bytes = rt.content_bytes
2137
+ )
2138
+ WHERE rt.content_ref IS NULL
2139
+ AND EXISTS (SELECT 1 FROM dedup_mirror dm WHERE dm.content_bytes = rt.content_bytes)`,
2140
+ ).run() as { changes?: number } | undefined;
2141
+ result.refsBackfilled = backfill?.changes ?? 0;
2142
+ });
2143
+ return result;
2144
+ }
2145
+
2146
+ /**
2147
+ * One-shot auto-maintenance pass for the session_start hook: prune old rows,
2148
+ * checkpoint the WAL if it's grown large, and (only if the DB is huge) VACUUM.
2149
+ * Best-effort: swallows errors so a session never fails to start over a
2150
+ * housekeeping hiccup. Returns a short summary for the diagnostic log.
2151
+ */
2152
+ export function autoMaintain(stateDir: string = getStateDir()): string {
2153
+ try {
2154
+ const stats = getDbStats(stateDir);
2155
+ const parts: string[] = [];
2156
+ // Prune rows older than 30d (default retention).
2157
+ const prune = pruneOldRows(stateDir, 30);
2158
+ if (prune.affected > 0) parts.push(`pruned ${prune.affected}`);
2159
+ // Checkpoint the WAL if it's over 10 MB (avoid pathological WAL growth).
2160
+ if (stats.walBytes > 10 * 1024 * 1024) {
2161
+ const ck = checkpointWal(stateDir);
2162
+ if (ck.reclaimedBytes > 0) parts.push(`wal -${ck.reclaimedBytes}B`);
2163
+ }
2164
+ // VACUUM only if the DB is over 100 MB AND freelist is >20% of pages.
2165
+ if (
2166
+ stats.dbBytes > 100 * 1024 * 1024 &&
2167
+ stats.pageCount > 0 &&
2168
+ stats.freelistPages / stats.pageCount > 0.2
2169
+ ) {
2170
+ const v = vacuumDb(stateDir);
2171
+ if (v.reclaimedBytes > 0) parts.push(`vacuum -${v.reclaimedBytes}B`);
2172
+ }
2173
+ return parts.length ? `auto-maintain: ${parts.join(", ")}` : "auto-maintain: nothing to do";
2174
+ } catch (err) {
2175
+ // Never block session start over housekeeping.
2176
+ return `auto-maintain: skipped (${(err as Error).message})`;
2177
+ }
2178
+ }