@wipcomputer/memory-crystal 0.7.11 → 0.7.13

package/CHANGELOG.md

@@ -10,6 +10,176 @@
 
 
 
+
+
+## 0.7.13 (2026-03-13)
+
+# Dev Update: Orphan Cleanup, DELETE Trigger, Doctor Fix
+
+**Date:** 2026-03-13
+**Author:** CC-Mini
+**Session:** memory-db-fix
+
+---
+
+## What Happened
+
+Parker ran the Memory Crystal install prompt and `crystal doctor` reported "Embeddings: FAILING ... no provider configured in env." Investigation revealed two separate issues:
+
+### Issue 1: Doctor False Positive
+
+`checkEmbeddingProvider()` in `doctor.ts` only checked `process.env.OPENAI_API_KEY`. But the cron job and hooks resolve the key from 1Password via the SA token at `~/.openclaw/secrets/op-sa-token`. The doctor didn't know about this path.
+
+**Fix:** Added `checkOpEmbeddings()` helper to `doctor.ts` that checks for the SA token file, then does a live `op read` to verify it works. Doctor now reports `ok: openai (via 1Password)` instead of `fail`.
+
+### Issue 2: Orphaned Vectors and FTS Entries
+
+On 2026-03-11, 141,652 bulk file-scan chunks were correctly deleted from the `chunks` table. Parker said: "Why are we indexing documents?" These were raw file scans (Python venv packages, TypeScript source, vendor code) with no conversational context.
+
+The deletion used raw SQL (`DELETE FROM chunks WHERE agent_id = 'system'`). But Memory Crystal had no DELETE trigger. The corresponding entries in `chunks_vec` (sqlite-vec) and `chunks_fts` (FTS5) were left orphaned.
+
+**Impact:**
+- 141,651 orphaned vectors (~875 MB)
+- 141,652 orphaned FTS entries
+- ~7% of search queries hit phantom results (silently filtered out)
+- Database: 1.96 GB (should have been ~1 GB)
+
+**Fix (three parts):**
+
+1. **DELETE trigger** added to `initChunksTables()` in `core.ts`:
+```sql
+CREATE TRIGGER IF NOT EXISTS chunks_cleanup AFTER DELETE ON chunks
+BEGIN
+  DELETE FROM chunks_vec WHERE chunk_id = OLD.id;
+  INSERT INTO chunks_fts(chunks_fts, rowid, text) VALUES('delete', OLD.id, OLD.text);
+END;
+```
+
+2. **`cleanOrphans()` method** added to Crystal class in `core.ts`. Counts orphaned vec/FTS entries, deletes vec orphans in batches of 1000, rebuilds FTS5 from scratch.
+
+3. **`crystal cleanup` CLI command** added to `cli.ts`. Handles the full workflow: backup, pause cron, clean orphans, VACUUM, resume cron. Supports `--dry-run`.
+
+**Cleanup results:**
+- 141,651 orphaned vectors removed
+- FTS rebuilt from 73,813 chunks in 5.7s
+- Database: 1.96 GB -> 1.45 GB (525 MB saved)
+- Verification: chunks = FTS entries = 73,813. Match: YES
+- Zero orphans remaining
+
+### Side Discovery: Plaintext SA Token
+
+During investigation, discovered that `~/.openclaw/secrets/op-sa-token` is a plaintext 1Password SA token readable by any process running as `lesa`. This is the bootstrap credential that unlocks all secrets. Bug report filed. Long-term fix: Lesa iOS app with remote biometrics (product doc written).
+
+### Product Rule Established
+
+Memory Crystal indexes conversations only. File content that appears in conversation turns (agent reads a file, discusses it) is captured as part of the conversation. Raw directory scanning without conversational context is not what Memory Crystal is for.
+
+## Files Changed
+
+| File | Change |
+|------|--------|
+| `src/core.ts` | Added `chunks_cleanup` DELETE trigger, `cleanOrphans()` method |
+| `src/cli.ts` | Added `crystal cleanup` command, updated imports and USAGE |
+| `src/doctor.ts` | Added `checkOpEmbeddings()` for 1Password detection |
+| `ai/product/bugs/2026-03-13--orphaned-vectors-and-fts-after-bulk-delete.md` | Bug report |
+
+## Related (wip-secrets-ios-private)
+
+| File | What |
+|------|------|
+| `ai/product/product-ideas/lesa-app-remote-biometrics.md` | Lesa app: remote biometrics for agent computers |
+| `ai/product/bugs/2026-03-13--plaintext-sa-token-on-disk.md` | Plaintext SA token bug report |
+
+## Status
+
+- Code deployed and running (cleanup already executed)
+- Not yet committed / PR'd / released
+- Needs: branch, commit, PR, merge, `wip-release patch`
+
+## 0.7.12 (2026-03-13)
+
+# Dev Update: Orphan Cleanup, DELETE Trigger, Doctor Fix
+
+**Date:** 2026-03-13
+**Author:** CC-Mini
+**Session:** memory-db-fix
+
+---
+
+## What Happened
+
+Parker ran the Memory Crystal install prompt and `crystal doctor` reported "Embeddings: FAILING ... no provider configured in env." Investigation revealed two separate issues:
+
+### Issue 1: Doctor False Positive
+
+`checkEmbeddingProvider()` in `doctor.ts` only checked `process.env.OPENAI_API_KEY`. But the cron job and hooks resolve the key from 1Password via the SA token at `~/.openclaw/secrets/op-sa-token`. The doctor didn't know about this path.
+
+**Fix:** Added `checkOpEmbeddings()` helper to `doctor.ts` that checks for the SA token file, then does a live `op read` to verify it works. Doctor now reports `ok: openai (via 1Password)` instead of `fail`.
+
+### Issue 2: Orphaned Vectors and FTS Entries
+
+On 2026-03-11, 141,652 bulk file-scan chunks were correctly deleted from the `chunks` table. Parker said: "Why are we indexing documents?" These were raw file scans (Python venv packages, TypeScript source, vendor code) with no conversational context.
+
+The deletion used raw SQL (`DELETE FROM chunks WHERE agent_id = 'system'`). But Memory Crystal had no DELETE trigger. The corresponding entries in `chunks_vec` (sqlite-vec) and `chunks_fts` (FTS5) were left orphaned.
+
+**Impact:**
+- 141,651 orphaned vectors (~875 MB)
+- 141,652 orphaned FTS entries
+- ~7% of search queries hit phantom results (silently filtered out)
+- Database: 1.96 GB (should have been ~1 GB)
+
+**Fix (three parts):**
+
+1. **DELETE trigger** added to `initChunksTables()` in `core.ts`:
+```sql
+CREATE TRIGGER IF NOT EXISTS chunks_cleanup AFTER DELETE ON chunks
+BEGIN
+  DELETE FROM chunks_vec WHERE chunk_id = OLD.id;
+  INSERT INTO chunks_fts(chunks_fts, rowid, text) VALUES('delete', OLD.id, OLD.text);
+END;
+```
+
+2. **`cleanOrphans()` method** added to Crystal class in `core.ts`. Counts orphaned vec/FTS entries, deletes vec orphans in batches of 1000, rebuilds FTS5 from scratch.
+
+3. **`crystal cleanup` CLI command** added to `cli.ts`. Handles the full workflow: backup, pause cron, clean orphans, VACUUM, resume cron. Supports `--dry-run`.
+
+**Cleanup results:**
+- 141,651 orphaned vectors removed
+- FTS rebuilt from 73,813 chunks in 5.7s
+- Database: 1.96 GB -> 1.45 GB (525 MB saved)
+- Verification: chunks = FTS entries = 73,813. Match: YES
+- Zero orphans remaining
+
+### Side Discovery: Plaintext SA Token
+
+During investigation, discovered that `~/.openclaw/secrets/op-sa-token` is a plaintext 1Password SA token readable by any process running as `lesa`. This is the bootstrap credential that unlocks all secrets. Bug report filed. Long-term fix: Lesa iOS app with remote biometrics (product doc written).
+
+### Product Rule Established
+
+Memory Crystal indexes conversations only. File content that appears in conversation turns (agent reads a file, discusses it) is captured as part of the conversation. Raw directory scanning without conversational context is not what Memory Crystal is for.
+
+## Files Changed
+
+| File | Change |
+|------|--------|
+| `src/core.ts` | Added `chunks_cleanup` DELETE trigger, `cleanOrphans()` method |
+| `src/cli.ts` | Added `crystal cleanup` command, updated imports and USAGE |
+| `src/doctor.ts` | Added `checkOpEmbeddings()` for 1Password detection |
+| `ai/product/bugs/2026-03-13--orphaned-vectors-and-fts-after-bulk-delete.md` | Bug report |
+
+## Related (wip-secrets-ios-private)
+
+| File | What |
+|------|------|
+| `ai/product/product-ideas/lesa-app-remote-biometrics.md` | Lesa app: remote biometrics for agent computers |
+| `ai/product/bugs/2026-03-13--plaintext-sa-token-on-disk.md` | Plaintext SA token bug report |
+
+## Status
+
+- Code deployed and running (cleanup already executed)
+- Not yet committed / PR'd / released
+- Needs: branch, commit, PR, merge, `wip-release patch`
+
 ## 0.7.11 (2026-03-13)
 
 # Dev Update: Orphan Cleanup, DELETE Trigger, Doctor Fix

package/dist/installer.js

@@ -38,7 +38,7 @@ function getRepoRoot() {
     if (existsSync(pkgPath)) {
       try {
         const pkg = JSON.parse(readFileSync(pkgPath, "utf-8"));
-        if (pkg.name === "memory-crystal") return dir;
+        if (pkg.name === "@wipcomputer/memory-crystal") return dir;
       } catch {
       }
     }
@@ -46,13 +46,26 @@ function getRepoRoot() {
   }
   return dirname(thisDir);
 }
+function getLatestNpmVersion() {
+  const names = ["@wipcomputer/memory-crystal", "memory-crystal"];
+  for (const name of names) {
+    try {
+      const v = execSync(`npm view ${name} version 2>/dev/null`, { encoding: "utf-8", timeout: 1e4 }).trim();
+      if (v) return v;
+    } catch {
+    }
+  }
+  return null;
+}
 function detectInstallState() {
   const ldmExtDir = join(LDM_ROOT, "extensions", "memory-crystal");
   const ocExtDir = join(OC_ROOT, "extensions", "memory-crystal");
   const paths = ldmPaths();
   const installedVersion = readVersion(join(ldmExtDir, "package.json"));
   const repoRoot = getRepoRoot();
-  const repoVersion = readVersion(join(repoRoot, "package.json")) || "0.0.0";
+  let repoVersion = readVersion(join(repoRoot, "package.json")) || "0.0.0";
+  const npmVersion = getLatestNpmVersion();
+  if (npmVersion && npmVersion > repoVersion) repoVersion = npmVersion;
   const ccHookDeployed = existsSync(join(ldmExtDir, "dist", "cc-hook.js"));
   let ccHookConfigured = false;
   try {
@@ -355,6 +368,18 @@ function ldmCliAvailable() {
     return false;
   }
 }
+function bootstrapLdmOs(steps) {
+  try {
+    steps.push("Installing LDM OS infrastructure...");
+    execSync("npm install -g @wipcomputer/wip-ldm-os", { stdio: "pipe", timeout: 12e4 });
+    execSync("ldm --version", { stdio: "pipe", timeout: 5e3 });
+    steps.push("LDM OS installed.");
+    return true;
+  } catch {
+    steps.push("LDM OS install skipped (npm offline or permissions issue). Using standalone.");
+    return false;
+  }
+}
 function runLdmInstall(repoDir) {
   const steps = [];
   try {
@@ -393,7 +418,25 @@ async function runInstallOrUpdate(options) {
       steps: [`Already at v${state.repoVersion}. Nothing to do.`]
     };
   }
-  const hasLdmCli = ldmCliAvailable();
+  if (isUpdate && state.installedVersion) {
+    const npmV = getLatestNpmVersion();
+    if (npmV && npmV > state.installedVersion) {
+      steps.push(`Upgrading v${state.installedVersion} -> v${npmV} via npm...`);
+      try {
+        execSync("npm install -g @wipcomputer/memory-crystal 2>&1", { encoding: "utf-8", timeout: 6e4, stdio: "pipe" });
+        steps.push(`Installed @wipcomputer/memory-crystal@${npmV}`);
+        steps.push("Restarting init with updated code...");
+        execSync("crystal init", { stdio: "inherit", timeout: 12e4 });
+        return { action: "updated", version: npmV, deployedTo: ["global", "ldm", "openclaw"], steps };
+      } catch (err) {
+        steps.push(`npm upgrade failed: ${err.message}. Continuing with local code.`);
+      }
+    }
+  }
+  let hasLdmCli = ldmCliAvailable();
+  if (!hasLdmCli) {
+    hasLdmCli = bootstrapLdmOs(steps);
+  }
   let ldmDelegated = false;
   if (hasLdmCli) {
     steps.push("LDM OS detected. Using ldm install for deployment...");
@@ -590,7 +633,7 @@ async function runInstallOrUpdate(options) {
   }
   if (hasLdmCli) {
     steps.push('Tip: Run "ldm install" to see more components you can add.');
-  } else {
+  } else if (!ldmDelegated) {
     steps.push("Tip: Install LDM OS for more components: npm install -g @wipcomputer/wip-ldm-os");
   }
   return {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@wipcomputer/memory-crystal",
-  "version": "0.7.11",
+  "version": "0.7.13",
   "description": "Sovereign memory system — local-first with ephemeral encrypted relay. Your memory, your machine, your rules.",
   "type": "module",
   "main": "dist/core.js",

package/skills/memory/SKILL.md

@@ -86,6 +86,24 @@ Before installing, explain exactly what will be created:
 >
 > Nothing gets installed outside this folder. Nothing phones home. Want me to go ahead?"
 
+### Step 0: Upgrade if already installed
+
+If `crystal` is already on PATH, check if it's current before running init:
+
+```bash
+crystal status 2>/dev/null | head -1
+npm view @wipcomputer/memory-crystal version 2>/dev/null
+```
+
+If npm version is newer, upgrade first. If installed under the old unscoped name (`memory-crystal`), switch to the scoped name:
+
+```bash
+npm uninstall -g memory-crystal 2>/dev/null
+npm install -g @wipcomputer/memory-crystal
+```
+
+Then proceed to Step 1.
+
 ### Step 1: Initialize
 
 ```bash