@wipcomputer/memory-crystal 0.7.12 → 0.7.14

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (3) hide show
  1. package/CHANGELOG.md +170 -0
  2. package/package.json +1 -1
  3. package/skills/memory/SKILL.md +11 -0
package/CHANGELOG.md CHANGED
@@ -11,6 +11,176 @@
11
11
 
12
12
 
13
13
 
14
+
15
+
16
+ ## 0.7.14 (2026-03-13)
17
+
18
+ # Dev Update: Orphan Cleanup, DELETE Trigger, Doctor Fix
19
+
20
+ **Date:** 2026-03-13
21
+ **Author:** CC-Mini
22
+ **Session:** memory-db-fix
23
+
24
+ ---
25
+
26
+ ## What Happened
27
+
28
+ Parker ran the Memory Crystal install prompt and `crystal doctor` reported "Embeddings: FAILING ... no provider configured in env." Investigation revealed two separate issues:
29
+
30
+ ### Issue 1: Doctor False Positive
31
+
32
+ `checkEmbeddingProvider()` in `doctor.ts` only checked `process.env.OPENAI_API_KEY`. But the cron job and hooks resolve the key from 1Password via the SA token at `~/.openclaw/secrets/op-sa-token`. The doctor didn't know about this path.
33
+
34
+ **Fix:** Added `checkOpEmbeddings()` helper to `doctor.ts` that checks for the SA token file, then does a live `op read` to verify it works. Doctor now reports `ok: openai (via 1Password)` instead of `fail`.
35
+
36
+ ### Issue 2: Orphaned Vectors and FTS Entries
37
+
38
+ On 2026-03-11, 141,652 bulk file-scan chunks were correctly deleted from the `chunks` table. Parker said: "Why are we indexing documents?" These were raw file scans (Python venv packages, TypeScript source, vendor code) with no conversational context.
39
+
40
+ The deletion used raw SQL (`DELETE FROM chunks WHERE agent_id = 'system'`). But Memory Crystal had no DELETE trigger. The corresponding entries in `chunks_vec` (sqlite-vec) and `chunks_fts` (FTS5) were left orphaned.
41
+
42
+ **Impact:**
43
+ - 141,651 orphaned vectors (~875 MB)
44
+ - 141,652 orphaned FTS entries
45
+ - ~7% of search queries hit phantom results (silently filtered out)
46
+ - Database: 1.96 GB (should have been ~1 GB)
47
+
48
+ **Fix (three parts):**
49
+
50
+ 1. **DELETE trigger** added to `initChunksTables()` in `core.ts`:
51
+ ```sql
52
+ CREATE TRIGGER IF NOT EXISTS chunks_cleanup AFTER DELETE ON chunks
53
+ BEGIN
54
+ DELETE FROM chunks_vec WHERE chunk_id = OLD.id;
55
+ INSERT INTO chunks_fts(chunks_fts, rowid, text) VALUES('delete', OLD.id, OLD.text);
56
+ END;
57
+ ```
58
+
59
+ 2. **`cleanOrphans()` method** added to Crystal class in `core.ts`. Counts orphaned vec/FTS entries, deletes vec orphans in batches of 1000, rebuilds FTS5 from scratch.
60
+
61
+ 3. **`crystal cleanup` CLI command** added to `cli.ts`. Handles the full workflow: backup, pause cron, clean orphans, VACUUM, resume cron. Supports `--dry-run`.
62
+
63
+ **Cleanup results:**
64
+ - 141,651 orphaned vectors removed
65
+ - FTS rebuilt from 73,813 chunks in 5.7s
66
+ - Database: 1.96 GB -> 1.45 GB (525 MB saved)
67
+ - Verification: chunks = FTS entries = 73,813. Match: YES
68
+ - Zero orphans remaining
69
+
70
+ ### Side Discovery: Plaintext SA Token
71
+
72
+ During investigation, discovered that `~/.openclaw/secrets/op-sa-token` is a plaintext 1Password SA token readable by any process running as `lesa`. This is the bootstrap credential that unlocks all secrets. Bug report filed. Long-term fix: Lesa iOS app with remote biometrics (product doc written).
73
+
74
+ ### Product Rule Established
75
+
76
+ Memory Crystal indexes conversations only. File content that appears in conversation turns (agent reads a file, discusses it) is captured as part of the conversation. Raw directory scanning without conversational context is not what Memory Crystal is for.
77
+
78
+ ## Files Changed
79
+
80
+ | File | Change |
81
+ |------|--------|
82
+ | `src/core.ts` | Added `chunks_cleanup` DELETE trigger, `cleanOrphans()` method |
83
+ | `src/cli.ts` | Added `crystal cleanup` command, updated imports and USAGE |
84
+ | `src/doctor.ts` | Added `checkOpEmbeddings()` for 1Password detection |
85
+ | `ai/product/bugs/2026-03-13--orphaned-vectors-and-fts-after-bulk-delete.md` | Bug report |
86
+
87
+ ## Related (wip-secrets-ios-private)
88
+
89
+ | File | What |
90
+ |------|------|
91
+ | `ai/product/product-ideas/lesa-app-remote-biometrics.md` | Lesa app: remote biometrics for agent computers |
92
+ | `ai/product/bugs/2026-03-13--plaintext-sa-token-on-disk.md` | Plaintext SA token bug report |
93
+
94
+ ## Status
95
+
96
+ - Code deployed and running (cleanup already executed)
97
+ - Not yet committed / PR'd / released
98
+ - Needs: branch, commit, PR, merge, `wip-release patch`
99
+
100
+ ## 0.7.13 (2026-03-13)
101
+
102
+ # Dev Update: Orphan Cleanup, DELETE Trigger, Doctor Fix
103
+
104
+ **Date:** 2026-03-13
105
+ **Author:** CC-Mini
106
+ **Session:** memory-db-fix
107
+
108
+ ---
109
+
110
+ ## What Happened
111
+
112
+ Parker ran the Memory Crystal install prompt and `crystal doctor` reported "Embeddings: FAILING ... no provider configured in env." Investigation revealed two separate issues:
113
+
114
+ ### Issue 1: Doctor False Positive
115
+
116
+ `checkEmbeddingProvider()` in `doctor.ts` only checked `process.env.OPENAI_API_KEY`. But the cron job and hooks resolve the key from 1Password via the SA token at `~/.openclaw/secrets/op-sa-token`. The doctor didn't know about this path.
117
+
118
+ **Fix:** Added `checkOpEmbeddings()` helper to `doctor.ts` that checks for the SA token file, then does a live `op read` to verify it works. Doctor now reports `ok: openai (via 1Password)` instead of `fail`.
119
+
120
+ ### Issue 2: Orphaned Vectors and FTS Entries
121
+
122
+ On 2026-03-11, 141,652 bulk file-scan chunks were correctly deleted from the `chunks` table. Parker said: "Why are we indexing documents?" These were raw file scans (Python venv packages, TypeScript source, vendor code) with no conversational context.
123
+
124
+ The deletion used raw SQL (`DELETE FROM chunks WHERE agent_id = 'system'`). But Memory Crystal had no DELETE trigger. The corresponding entries in `chunks_vec` (sqlite-vec) and `chunks_fts` (FTS5) were left orphaned.
125
+
126
+ **Impact:**
127
+ - 141,651 orphaned vectors (~875 MB)
128
+ - 141,652 orphaned FTS entries
129
+ - ~7% of search queries hit phantom results (silently filtered out)
130
+ - Database: 1.96 GB (should have been ~1 GB)
131
+
132
+ **Fix (three parts):**
133
+
134
+ 1. **DELETE trigger** added to `initChunksTables()` in `core.ts`:
135
+ ```sql
136
+ CREATE TRIGGER IF NOT EXISTS chunks_cleanup AFTER DELETE ON chunks
137
+ BEGIN
138
+ DELETE FROM chunks_vec WHERE chunk_id = OLD.id;
139
+ INSERT INTO chunks_fts(chunks_fts, rowid, text) VALUES('delete', OLD.id, OLD.text);
140
+ END;
141
+ ```
142
+
143
+ 2. **`cleanOrphans()` method** added to Crystal class in `core.ts`. Counts orphaned vec/FTS entries, deletes vec orphans in batches of 1000, rebuilds FTS5 from scratch.
144
+
145
+ 3. **`crystal cleanup` CLI command** added to `cli.ts`. Handles the full workflow: backup, pause cron, clean orphans, VACUUM, resume cron. Supports `--dry-run`.
146
+
147
+ **Cleanup results:**
148
+ - 141,651 orphaned vectors removed
149
+ - FTS rebuilt from 73,813 chunks in 5.7s
150
+ - Database: 1.96 GB -> 1.45 GB (525 MB saved)
151
+ - Verification: chunks = FTS entries = 73,813. Match: YES
152
+ - Zero orphans remaining
153
+
154
+ ### Side Discovery: Plaintext SA Token
155
+
156
+ During investigation, discovered that `~/.openclaw/secrets/op-sa-token` is a plaintext 1Password SA token readable by any process running as `lesa`. This is the bootstrap credential that unlocks all secrets. Bug report filed. Long-term fix: Lesa iOS app with remote biometrics (product doc written).
157
+
158
+ ### Product Rule Established
159
+
160
+ Memory Crystal indexes conversations only. File content that appears in conversation turns (agent reads a file, discusses it) is captured as part of the conversation. Raw directory scanning without conversational context is not what Memory Crystal is for.
161
+
162
+ ## Files Changed
163
+
164
+ | File | Change |
165
+ |------|--------|
166
+ | `src/core.ts` | Added `chunks_cleanup` DELETE trigger, `cleanOrphans()` method |
167
+ | `src/cli.ts` | Added `crystal cleanup` command, updated imports and USAGE |
168
+ | `src/doctor.ts` | Added `checkOpEmbeddings()` for 1Password detection |
169
+ | `ai/product/bugs/2026-03-13--orphaned-vectors-and-fts-after-bulk-delete.md` | Bug report |
170
+
171
+ ## Related (wip-secrets-ios-private)
172
+
173
+ | File | What |
174
+ |------|------|
175
+ | `ai/product/product-ideas/lesa-app-remote-biometrics.md` | Lesa app: remote biometrics for agent computers |
176
+ | `ai/product/bugs/2026-03-13--plaintext-sa-token-on-disk.md` | Plaintext SA token bug report |
177
+
178
+ ## Status
179
+
180
+ - Code deployed and running (cleanup already executed)
181
+ - Not yet committed / PR'd / released
182
+ - Needs: branch, commit, PR, merge, `wip-release patch`
183
+
14
184
  ## 0.7.12 (2026-03-13)
15
185
 
16
186
  # Dev Update: Orphan Cleanup, DELETE Trigger, Doctor Fix
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@wipcomputer/memory-crystal",
3
- "version": "0.7.12",
3
+ "version": "0.7.14",
4
4
  "description": "Sovereign memory system — local-first with ephemeral encrypted relay. Your memory, your machine, your rules.",
5
5
  "type": "module",
6
6
  "main": "dist/core.js",
package/skills/memory/SKILL.md CHANGED
@@ -86,6 +86,17 @@ Before installing, explain exactly what will be created:
86
86
  >
87
87
  > Nothing gets installed outside this folder. Nothing phones home. Want me to go ahead?"
88
88
 
89
+ ### Step 0: Install or upgrade the global CLI (ALWAYS run this)
90
+
91
+ **Always run this before crystal init.** Even if crystal is already installed. This ensures the global CLI is the latest version. Never skip this step. Never build from source and copy to extensions instead.
92
+
93
+ ```bash
94
+ npm uninstall -g memory-crystal 2>/dev/null
95
+ npm install -g @wipcomputer/memory-crystal
96
+ ```
97
+
98
+ Then proceed to Step 1.
99
+
89
100
  ### Step 1: Initialize
90
101
 
91
102
  ```bash