@wipcomputer/memory-crystal 0.7.30 → 0.7.33

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (114) hide show
  1. package/SKILL.md +1 -1
  2. package/cloud/wrangler.toml +30 -0
  3. package/dist/bulk-copy.js +1 -1
  4. package/dist/cc-hook.js +3 -3
  5. package/dist/cc-poller.js +2 -2
  6. package/dist/chunk-2GBYLMEF.js +1385 -0
  7. package/dist/chunk-437F27T6.js +97 -0
  8. package/dist/chunk-5I7GMRDN.js +146 -0
  9. package/dist/chunk-CGIDSAJB.js +288 -0
  10. package/dist/chunk-D3MACYZ4.js +108 -0
  11. package/dist/chunk-DFQ72B7M.js +248 -0
  12. package/dist/chunk-NX647OM3.js +310 -0
  13. package/dist/cli.js +62 -7
  14. package/dist/core.d.ts +22 -2
  15. package/dist/core.js +1 -1
  16. package/dist/crypto.js +2 -2
  17. package/dist/crystal-serve.js +3 -3
  18. package/dist/doctor.js +12 -4
  19. package/dist/dream-weaver.js +2 -2
  20. package/dist/file-sync.js +3 -3
  21. package/dist/installer.js +99 -3
  22. package/dist/ldm.js +1 -1
  23. package/dist/llm-XXLYPIOF.js +16 -0
  24. package/dist/mcp-server.js +17 -5
  25. package/dist/migrate.js +1 -1
  26. package/dist/mirror-sync.js +4 -4
  27. package/dist/mlx-setup-XKU67WCT.js +289 -0
  28. package/dist/openclaw.js +16 -5
  29. package/dist/pair.js +2 -2
  30. package/dist/poller.js +5 -5
  31. package/dist/role.js +2 -2
  32. package/dist/search-pipeline-CBV25NX7.js +99 -0
  33. package/dist/staging.js +2 -2
  34. package/package.json +15 -1
  35. package/.env.example +0 -20
  36. package/.publish-skill.json +0 -1
  37. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/.env.example +0 -20
  38. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/.publish-skill.json +0 -1
  39. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/CHANGELOG.md +0 -1297
  40. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/CLA.md +0 -19
  41. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/LICENSE +0 -52
  42. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/README-ENTERPRISE.md +0 -226
  43. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/README.md +0 -151
  44. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/RELAY.md +0 -199
  45. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/SKILL.md +0 -462
  46. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/TECHNICAL.md +0 -656
  47. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/_trash/RELEASE-NOTES-v0-7-23.md +0 -48
  48. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/_trash/RELEASE-NOTES-v0-7-25.md +0 -24
  49. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/_trash/RELEASE-NOTES-v0-7-26.md +0 -7
  50. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/_trash/RELEASE-NOTES-v0-7-28.md +0 -31
  51. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/_trash/RELEASE-NOTES-v0-7-29.md +0 -28
  52. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/_trash/RELEASE-NOTES-v0-7-4.md +0 -64
  53. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/_trash/RELEASE-NOTES-v0-7-5.md +0 -19
  54. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/cloud/README.md +0 -116
  55. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/cloud/docs/gpt-system-instructions.md +0 -69
  56. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/cloud/migrations/0001_init.sql +0 -52
  57. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/migrations/0001_init.sql +0 -51
  58. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/migrations/0002_cloud_storage.sql +0 -49
  59. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/openclaw.plugin.json +0 -11
  60. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/package-lock.json +0 -4169
  61. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/package.json +0 -61
  62. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/scripts/crystal-capture.sh +0 -29
  63. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/scripts/deploy-cloud.sh +0 -153
  64. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/scripts/ldm-backup.sh +0 -116
  65. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/scripts/migrate-lance-to-sqlite.mjs +0 -218
  66. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/skills/memory/SKILL.md +0 -438
  67. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/wrangler-demo.toml +0 -8
  68. package/.worktrees/memory-crystal-private--cc-mini-fix-home-fallback/wrangler-mcp.toml +0 -24
  69. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/.env.example +0 -20
  70. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/.publish-skill.json +0 -1
  71. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/CHANGELOG.md +0 -1297
  72. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/CLA.md +0 -19
  73. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/LICENSE +0 -52
  74. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/README-ENTERPRISE.md +0 -226
  75. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/README.md +0 -151
  76. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/RELAY.md +0 -199
  77. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/RELEASE-NOTES-v0.7.30.md +0 -29
  78. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/SKILL.md +0 -462
  79. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/TECHNICAL.md +0 -656
  80. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/_trash/RELEASE-NOTES-v0-7-23.md +0 -48
  81. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/_trash/RELEASE-NOTES-v0-7-25.md +0 -24
  82. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/_trash/RELEASE-NOTES-v0-7-26.md +0 -7
  83. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/_trash/RELEASE-NOTES-v0-7-28.md +0 -31
  84. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/_trash/RELEASE-NOTES-v0-7-29.md +0 -28
  85. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/_trash/RELEASE-NOTES-v0-7-4.md +0 -64
  86. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/_trash/RELEASE-NOTES-v0-7-5.md +0 -19
  87. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/cloud/README.md +0 -116
  88. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/cloud/docs/gpt-system-instructions.md +0 -69
  89. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/cloud/migrations/0001_init.sql +0 -52
  90. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/migrations/0001_init.sql +0 -51
  91. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/migrations/0002_cloud_storage.sql +0 -49
  92. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/openclaw.plugin.json +0 -11
  93. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/package-lock.json +0 -4169
  94. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/package.json +0 -61
  95. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/scripts/crystal-capture.sh +0 -29
  96. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/scripts/deploy-cloud.sh +0 -153
  97. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/scripts/ldm-backup.sh +0 -116
  98. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/scripts/migrate-lance-to-sqlite.mjs +0 -218
  99. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/skills/memory/SKILL.md +0 -438
  100. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/wrangler-demo.toml +0 -8
  101. package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/wrangler-mcp.toml +0 -24
  102. package/CHANGELOG.md +0 -1329
  103. package/README-ENTERPRISE.md +0 -226
  104. package/RELAY.md +0 -199
  105. package/_trash/RELEASE-NOTES-v0-7-23.md +0 -48
  106. package/_trash/RELEASE-NOTES-v0-7-25.md +0 -24
  107. package/_trash/RELEASE-NOTES-v0-7-26.md +0 -7
  108. package/_trash/RELEASE-NOTES-v0-7-28.md +0 -31
  109. package/_trash/RELEASE-NOTES-v0-7-29.md +0 -28
  110. package/_trash/RELEASE-NOTES-v0-7-4.md +0 -64
  111. package/_trash/RELEASE-NOTES-v0-7-5.md +0 -19
  112. package/_trash/RELEASE-NOTES-v0.7.30.md +0 -29
  113. package/wrangler-demo.toml +0 -8
  114. package/wrangler-mcp.toml +0 -24
@@ -1,226 +0,0 @@
1
- ###### WIP Computer
2
-
3
- # Memory Crystal for Enterprise
4
-
5
- Agent memory infrastructure. Local-first. Encrypted. Inspectable. *In testing.*
6
-
7
- ## The Problem
8
-
9
- Your organization runs AI agents across teams, departments, and tools. Each agent starts every session with no memory. No continuity. No shared context. Conversations with one agent are invisible to every other.
10
-
11
- Your agents can't remember what your people told them. Your people keep re-explaining themselves. Context is lost between sessions, between tools, between teams.
12
-
13
- This is not a convenience problem. It's a reliability problem. An auditability problem. A cost problem.
14
-
15
- ## What Memory Crystal Does
16
-
17
- Memory Crystal is a persistent context layer for AI agents. One shared database. Hybrid search. On your machines. Under your control.
18
-
19
- - **Local-first.** All data stays on-prem. Nothing leaves your network unless you configure it to.
20
- - **Inspectable.** One SQLite file. Open it with any SQLite tool. Audit it. Query it. Back it up with `cp`.
21
- - **Deterministic search.** Hybrid retrieval (BM25 keyword + vector similarity + Reciprocal Rank Fusion). Same query, same results. No black-box ranking.
22
- - **Encrypted sync.** Multi-site deployments use AES-256-GCM encryption with HMAC-SHA256 signing. The relay sees encrypted noise. Keys never leave your machines.
23
- - **Agent isolation.** Each agent gets its own ID, its own transcript archive, its own session summaries. Shared search across agents, isolated storage per agent.
24
- - **Zero cloud dependency.** Runs fully offline with local embeddings (Ollama). No API keys required. No data exfiltration risk.
25
-
26
- ## Five-Layer Memory Stack
27
-
28
- Memory Crystal implements a full memory pipeline, not just search.
29
-
30
- | Layer | What | How |
31
- |-------|------|-----|
32
- | L1: Raw Transcripts | Every conversation archived as JSONL | Automatic. cc-poller (cron), cc-hook (Stop), openclaw.ts (agent_end) |
33
- | L2: Vector Index | Chunks embedded into crystal.db | Automatic. Hybrid search (BM25 + vector + RRF) |
34
- | L3: Structured Memory | Facts, preferences, decisions | `crystal_remember` / `crystal_forget` |
35
- | L4: Narrative Consolidation | Dream Weaver journals, identity, soul | `crystal dream-weave` (imports engine from Dream Weaver Protocol) |
36
- | L5: Active Working Context | Boot sequence files, shared context | Agent reads on startup |
37
-
38
- L1-L3 are fully automated. L4 runs on-demand or via Crystal Core gateway. L5 is consumed by the agent's boot sequence.
39
-
40
- ## Architecture
41
-
42
- ```
43
- sqlite-vec (vectors) + FTS5 (BM25) + SQLite (metadata)
44
- | | |
45
- core.ts ... pure logic, zero framework deps
46
- |-- cli.ts -> crystal search, dream-weave, backfill, serve
47
- |-- mcp-server.ts -> MCP protocol (any compatible client)
48
- |-- openclaw.ts -> OpenClaw plugin + raw data sync to LDM
49
- |-- cc-poller.ts -> Continuous capture (cron, primary)
50
- |-- cc-hook.ts -> Claude Code Stop hook (redundancy) + relay commands
51
- |-- crystal-serve.ts -> Crystal Core gateway (localhost:18790)
52
- |-- dream-weaver.ts -> Dream Weaver integration (narrative consolidation)
53
- |-- staging.ts -> New agent staging pipeline
54
- |-- llm.ts -> LLM provider cascade (MLX > Ollama > OpenAI > Anthropic)
55
- |-- search-pipeline.ts -> Deep search (expand, search, RRF, rerank, blend)
56
- +-- worker.ts -> Encrypted relay (multi-site sync, 3 channels)
57
- ```
58
-
59
- One core module. Multiple interfaces. Every interface calls the same search engine. No inconsistency between access paths.
60
-
61
- ## Security Model
62
-
63
- **Data at rest:** Single SQLite file. Standard filesystem permissions. Encrypt the volume if your compliance requires it.
64
-
65
- **Data in transit:** AES-256-GCM authenticated encryption. HMAC-SHA256 integrity verification. Shared symmetric key generated on-prem, never transmitted to the relay. The relay is a dead drop with no decryption capability.
66
-
67
- **Agent boundaries:** `CRYSTAL_AGENT_ID` isolates each agent's transcript archive, session summaries, and daily logs. Search spans all agents by default, or filters by agent ID.
68
-
69
- **Private mode:** Memory capture can be paused per-agent. When off, nothing is recorded. Resumes from where it left off when re-enabled.
70
-
71
- **No background processes that move data.** No telemetry. No analytics. No phone-home. The code is open source. Audit it.
72
-
73
- ## Retrieval Quality
74
-
75
- Hybrid search is not "we added vectors." It's a two-tier retrieval engine with LLM-powered deep search.
76
-
77
- ### Fast Path (Hybrid Search)
78
- - **FTS5 BM25** for exact keyword matches (Porter stemming)
79
- - **sqlite-vec cosine similarity** for semantic matches
80
- - **Reciprocal Rank Fusion** merges both result lists (k=60, tiered weights: BM25 2x, vector 1x)
81
- - **Recency weighting** ensures fresh context wins decisively: exponential decay `max(0.3, exp(-age_days * 0.1))`
82
- - **Content deduplication** via SHA-256 hash prevents duplicate embeddings
83
- - **Time-filtered search** ... restrict results to last 24h, 7d, 30d, or any date range
84
-
85
- ### Deep Search (LLM-Powered, default)
86
- - **Query expansion** ... LLM generates 3 search variations (lexical, semantic, hypothetical document). Each runs through hybrid search. Results merged via RRF.
87
- - **Strong signal detection** ... BM25 probe skips expansion when the answer is obvious (saves latency).
88
- - **LLM re-ranking** ... top 40 candidates scored by LLM for relevance to the original query.
89
- - **Position-aware blending** ... trusts RRF for top positions, lets the reranker fix ordering in the tail.
90
-
91
- Deep search runs by default. Falls back to fast path silently if no LLM provider is available. For air-gapped environments, MLX (Apple Silicon) or Ollama provides free, fully local deep search with no API keys and no network.
92
-
93
- A search for "deployment policy" finds conversations containing those exact words (BM25), conversations about "shipping code to production" (vector similarity), and conversations about "release workflow" that the LLM recognizes as relevant. All three matter. All three surface.
94
-
95
- ## What Gets Stored
96
-
97
- Every agent conversation produces three artifacts:
98
-
99
- | Artifact | Format | Location |
100
- |----------|--------|----------|
101
- | Raw transcript | JSONL | `~/.ldm/agents/{id}/memory/transcripts/` |
102
- | Session summary | Markdown | `~/.ldm/agents/{id}/memory/sessions/` |
103
- | Embeddings | sqlite-vec | `~/.ldm/memory/crystal.db` |
104
-
105
- Additionally:
106
- - **Explicit memories** stored via `crystal_remember` (facts, preferences, decisions)
107
- - **Source files** indexed as collections (code, documentation, internal knowledge bases)
108
- - **Daily logs** appended per-agent for audit trails
109
- - **Dream Weaver journals** generated by narrative consolidation (identity, soul, context, reference)
110
- - **Workspace files** synced from agent workspace to LDM (OpenClaw .md files)
111
-
112
- ## Embedding Providers
113
-
114
- | Provider | Model | Dimensions | Network Required |
115
- |----------|-------|-----------|-----------------|
116
- | Ollama (recommended for enterprise) | nomic-embed-text | 768 | No. Fully local. |
117
- | OpenAI | text-embedding-3-small | 1536 | Yes. API calls. |
118
- | Google | text-embedding-004 | 768 | Yes. API calls. |
119
-
120
- For air-gapped environments, Ollama is the only option. No data leaves the machine. No API keys. No external dependencies.
121
-
122
- ## Multi-Site Sync
123
-
124
- For organizations with multiple offices or remote teams.
125
-
126
- **Architecture:** One Crystal Core (the primary embedder), many Crystal Nodes (capture and sync). Core is the only machine that writes embeddings. Nodes capture raw conversations and send them to Core. Core embeds, then pushes deltas back.
127
-
128
- **What syncs:**
129
- 1. **Delta chunks** ... only new embeddings since last sync (not the full database)
130
- 2. **Full file tree** ... the entire `~/.ldm/` directory: workspace files, daily logs, journals, media, everything an embedding references
131
- 3. **Commands** ... bidirectional remote operations (run Dream Weaver, trigger backfill, request status)
132
-
133
- **How it works:**
134
- 1. Each site runs Memory Crystal locally
135
- 2. Core embeds all conversations into crystal.db (one source of truth for embeddings)
136
- 3. New chunks + changed files are encrypted (AES-256-GCM) and signed (HMAC-SHA256)
137
- 4. Encrypted deltas are dropped at a relay (hosted or self-hosted)
138
- 5. Other sites poll, decrypt, and insert into their local crystal.db + file tree
139
- 6. The relay deletes blobs after pickup
140
-
141
- **No cloud search.** Every node has the full database and full file tree. All search is local. The relay is pure transport. Nothing is stored or searchable in the cloud.
142
-
143
- **Self-hosted relay:** Deploy the Cloudflare Worker on your own Cloudflare account. Full control. No third-party data exposure.
144
-
145
- **Hosted relay:** Use our infrastructure. Free during beta. Your data is encrypted before it reaches us. We cannot read it.
146
-
147
- ## Compliance
148
-
149
- - **Data residency:** All primary data is local. Relay blobs are encrypted and ephemeral.
150
- - **Auditability:** SQLite is inspectable. Every chunk has a timestamp, source, and SHA-256 hash.
151
- - **Right to delete:** `crystal forget <id>` deprecates specific memories. Database can be wiped entirely with standard file operations.
152
- - **Access control:** Filesystem permissions on the SQLite file. No built-in user auth (it's a local tool, not a SaaS).
153
- - **No vendor lock-in:** MIT licensed (local code). Standard SQLite format. Export with any SQLite tool.
154
-
155
- ## Database
156
-
157
- One file: `crystal.db`. Contains:
158
-
159
- | Table | Purpose |
160
- |-------|---------|
161
- | `chunks` | Chunk text, metadata, SHA-256 hash, timestamps |
162
- | `chunks_vec` | sqlite-vec virtual table (vector search) |
163
- | `chunks_fts` | FTS5 virtual table (keyword search) |
164
- | `memories` | Explicit facts and preferences |
165
- | `capture_state` | Watermarks for incremental ingestion |
166
- | `source_collections` | Indexed directory collections |
167
- | `source_files` | File records with content hashes |
168
-
169
- No migrations server. No schema versioning service. It's SQLite. `sqlite3 crystal.db ".schema"` shows you everything.
170
-
171
- ## Integration
172
-
173
- | Platform | Integration | Auto-Capture |
174
- |----------|------------|-------------|
175
- | Claude Code | Cron poller (`cc-poller.ts`, primary) + Stop hook (`cc-hook.ts`, redundancy) | Yes. Every minute via cron, plus flush on session end. |
176
- | OpenClaw | Plugin (`openclaw.ts`) + `agent_end` hook + raw data sync | Yes. Every turn. Also syncs sessions, workspace, daily logs to LDM. |
177
- | Claude Desktop | MCP server (`mcp-server.ts`) | Search only. Manual capture. |
178
- | Any MCP client | MCP server | Search only. Manual capture. |
179
- | Any shell-accessible tool | CLI (`crystal search`) | Manual. |
180
- | Custom agents | Node.js module (`import from 'memory-crystal'`) | Programmable. |
181
-
182
- ## Crystal Core Gateway
183
-
184
- Crystal Core runs an HTTP gateway (`crystal serve`) on localhost:18790. OpenAI-compatible endpoint for agent-to-agent communication and automated processing.
185
-
186
- - `POST /v1/chat/completions` ... invoke `claude -p` through the gateway
187
- - `POST /process` ... trigger backfill, dream-weave, or staging processing
188
- - `GET /status` ... health check and crystal stats
189
-
190
- Localhost-only binding. Never exposed to the network. Optional bearer token auth.
191
-
192
- ## New Agent Onboarding
193
-
194
- When a new agent connects via relay, Crystal Core automatically:
195
- 1. Detects the unknown agent ID
196
- 2. Routes to staging (`~/.ldm/staging/{agent_id}/`)
197
- 3. Runs backfill (embed all transcripts)
198
- 4. Runs Dream Weaver full mode (generate identity, soul, context, journals)
199
- 5. Moves to live capture
200
-
201
- No manual intervention. The staging pipeline handles the cold-start problem.
202
-
203
- ## Deployment
204
-
205
- ```bash
206
- npm install memory-crystal
207
- crystal init --agent your-agent-id
208
- crystal status
209
- ```
210
-
211
- For enterprise deployments across multiple machines, see [Relay: Memory Sync](https://github.com/wipcomputer/memory-crystal/blob/main/RELAY.md).
212
-
213
- For full technical details, see [Technical Documentation](https://github.com/wipcomputer/memory-crystal/blob/main/TECHNICAL.md).
214
-
215
- ---
216
-
217
- ## License
218
-
219
- ```
220
- src/, skills/, cli.ts, mcp-server.ts MIT (use anywhere, no restrictions)
221
- worker/ AGPL (relay server)
222
- ```
223
-
224
- AGPL for personal use is free.
225
-
226
- Built by Parker Todd Brooks, Lēsa (OpenClaw, Claude Opus 4.6), Claude Code CLI (Claude Opus 4.6).
package/RELAY.md DELETED
@@ -1,199 +0,0 @@
1
- ###### WIP Computer
2
-
3
- # Relay: Memory Sync
4
-
5
- Memory Crystal works on one machine out of the box. Relay lets your memory follow you across machines and surfaces. Conversations captured on your laptop are available on your desktop. Conversations from ChatGPT on your phone are searchable from Claude Code on your Mac.
6
-
7
- Everything is encrypted before it leaves your machine. The relay never sees your data unencrypted.
8
-
9
- ## Crystal Core and Crystal Node
10
-
11
- Memory Crystal uses a Core/Node architecture:
12
-
13
- - **Crystal Core** ... the master memory. All conversations, all embeddings, all memories live here. This is the database you cannot lose. Put it on something permanent: a desktop, a home server, a Mac mini. Treat it like your photo library.
14
- - **Crystal Node** ... a synced copy on any other device. Captures conversations, sends them to the Core via encrypted relay. Gets a mirror back for local search. If a node dies, nothing is lost. The Core has everything.
15
-
16
- One Core, many Nodes. The Core does embeddings. Nodes just capture and sync. You can move the Core later with `crystal promote`.
17
-
18
- ## Two Sync Paths
19
-
20
- ### Encrypted Relay (device-to-device)
21
-
22
- For syncing between your own machines. Fully encrypted. The cloud is blind.
23
-
24
- ```
25
- Crystal Node (laptop) --[encrypt]--> Relay (Cloudflare R2) --[pickup + decrypt]--> Crystal Core (desktop)
26
- Crystal Core --[encrypt mirror]--> Relay --[pickup + decrypt]--> Crystal Node
27
- ```
28
-
29
- Three channels:
30
- - **conversations** (Node to Core) ... encrypted conversation chunks (ephemeral, deleted after pickup)
31
- - **mirror** (Core to Nodes) ... delta chunks (pre-embedded) + file tree deltas
32
- - **commands** (bidirectional) ... Nodes send commands to Core ("run Dream Weaver", "process my data"), Core sends results back
33
-
34
- The relay is a dead drop. It stores encrypted blobs temporarily and serves them on request. It has no decryption capability. If someone compromises the relay, they get encrypted noise.
35
-
36
- ### New Agent Staging
37
-
38
- When the Core's poller receives data from an unknown agent ID, it routes to staging instead of live ingest:
39
- 1. Transcripts written to `~/.ldm/staging/{agent_id}/transcripts/`
40
- 2. Agent marked as "ready" for processing
41
- 3. Staging processor runs backfill + Dream Weaver full mode
42
- 4. Once complete, agent moves to live capture path
43
-
44
- This handles the cold-start problem. A new device connects, sends its history, and Core builds the full memory stack automatically.
45
-
46
- ### Delta Sync (not full mirror)
47
-
48
- The mirror channel uses delta sync. Core pushes only new chunks since last sync, not the entire crystal.db. For a 1.9 GB+ database, this is the difference between a few KB (quiet day) and a few MB (busy day) vs the full database every time.
49
-
50
- - **New node (cold start):** One-time full export of all chunks + all files
51
- - **Ongoing sync:** Delta chunks (pre-embedded by Core) + changed files only
52
- - **Watermark tracking:** Core tracks the last synced chunk ID per node
53
-
54
- ### Full LDM Tree Sync
55
-
56
- The relay syncs the entire `~/.ldm/` file tree, not just the database. Embeddings are pointers to artifacts. If the file isn't on the node, the search result is an orphan.
57
-
58
- What syncs:
59
- - Agent memory files (workspace, daily logs, journals, sessions, transcripts)
60
- - Agent identity files (SOUL.md, IDENTITY.md, CONTEXT.md, REFERENCE.md)
61
- - Shared files (`~/.ldm/shared/`)
62
- - Media (images, videos, any artifact an embedding references)
63
-
64
- File sync uses a manifest (path + SHA-256 hash + size). Only changed files transfer. Core always wins conflicts.
65
-
66
- ## Setup
67
-
68
- ### Pair Your Devices
69
-
70
- ```bash
71
- # On your first machine (generates key if none exists)
72
- crystal pair
73
-
74
- # Displays a QR code and a pairing string:
75
- # mc1:T2hJbGxPZkRhcmtuZXNzTXlPbGRGcmllbmQ=
76
-
77
- # On your second machine
78
- crystal pair --code mc1:T2hJbGxPZkRhcmtuZXNzTXlPbGRGcmllbmQ=
79
- ```
80
-
81
- The QR code transfers the encryption key between devices without touching a server. Physical proximity only. Same security model as AirDrop.
82
-
83
- Alternative: store the key in 1Password and pull from both machines.
84
-
85
- ### Use the WIP.computer Relay (Default)
86
-
87
- We host the relay infrastructure. You just need an encryption key (generated by `crystal pair`).
88
-
89
- ```
90
- Open your AI and say:
91
-
92
- I want to set up multi-device sync for Memory Crystal.
93
- Walk me through the setup step by step.
94
- ```
95
-
96
- Your agent generates your encryption key, configures the connection, and tests it.
97
-
98
- **What you need:**
99
- - Memory Crystal installed on both machines (Core + Node)
100
- - An encryption key (your agent generates this, or use `crystal pair`)
101
-
102
- **Pricing:** Free during beta for individual use. When pricing is introduced, your agent will handle it via [AI CASH](https://github.com/wipcomputer/wip-agent-pay/blob/main/CASH.md).
103
-
104
- ### Self-Host Your Own Relay
105
-
106
- Run your own relay on Cloudflare Workers (free tier). Same code, your infrastructure. Full sovereignty.
107
-
108
- **What you need:**
109
- - A Cloudflare account (free tier works)
110
- - About five minutes
111
-
112
- **Steps:**
113
- 1. Clone the repo and deploy the Worker:
114
- ```bash
115
- cd memory-crystal
116
- npm run build:worker
117
- wrangler deploy --config wrangler.toml
118
- ```
119
- 2. Create an R2 bucket: `wrangler r2 bucket create memory-crystal-relay`
120
- 3. Set auth tokens for your devices:
121
- ```bash
122
- wrangler secret put AUTH_TOKEN_CC_MINI
123
- wrangler secret put AUTH_TOKEN_CC_AIR
124
- ```
125
- 4. Configure Memory Crystal to use your Worker:
126
- ```bash
127
- export CRYSTAL_RELAY_URL=https://your-relay.your-domain.workers.dev
128
- export CRYSTAL_RELAY_TOKEN=your-auth-token
129
- ```
130
-
131
- Full deployment details in [Technical Documentation](https://github.com/wipcomputer/memory-crystal/blob/main/TECHNICAL.md).
132
-
133
- No fees. No dependencies on us. The relay code is open source.
134
-
135
- ### Connecting ChatGPT and Claude
136
-
137
- Every node has the full database and file tree. All search is local. There is no cloud search layer.
138
-
139
- ChatGPT and Claude on iOS/web connect via the MCP server running on your local machine. On platforms where a local MCP server isn't possible (iOS without a Mac nearby), the native Apple app (future) will provide local search via MLX Swift.
140
-
141
- The Cloud MCP demo server (`worker-mcp.ts`, D1 + Vectorize) exists for onboarding and testing but is not the production architecture. With full LDM sync, every device that has Memory Crystal installed can search locally.
142
-
143
- ## Encryption
144
-
145
- - **AES-256-GCM** for encryption (authenticated encryption, no padding oracle attacks)
146
- - **HMAC-SHA256** for signing (integrity verification before decryption)
147
- - Shared key generated locally via `crystal pair`, never transmitted to the relay
148
- - Key must be present on all synced devices
149
-
150
- ### Key Management
151
-
152
- | Method | How | Best for |
153
- |--------|-----|----------|
154
- | `crystal pair` | QR code + pairing string | Two devices in the same room |
155
- | 1Password | Store key, pull via SA token on each machine | Headless, multiple machines |
156
- | Manual | `openssl rand -base64 32`, copy to each device | SSH, air-gapped |
157
-
158
- ## Architecture
159
-
160
- ```
161
- Encrypted Relay (device sync):
162
- src/worker.ts Cloudflare Worker, R2 storage, dead drop (3 channels)
163
- src/crypto.ts AES-256-GCM + HMAC-SHA256
164
- src/poller.ts Crystal Core pickup + ingest + staging detection + commands
165
- src/mirror-sync.ts Delta chunk sync + file tree sync to Crystal Nodes
166
- src/file-sync.ts Manifest-based file tree delta sync
167
- src/cc-hook.ts Claude Code hook (relay mode) + sendCommand()
168
- src/cc-poller.ts Continuous capture (cron, primary local path)
169
- src/staging.ts New agent staging pipeline (detect, stage, process)
170
-
171
- Cloud MCP Demo (deprecated for production):
172
- src/worker-mcp.ts OAuth 2.1 + DCR, MCP protocol, 4 tools
173
- src/cloud-crystal.ts D1 + Vectorize backend (demo/onboarding only)
174
- wrangler-mcp.toml Separate Worker config
175
-
176
- Pairing:
177
- crystal pair QR code key sharing
178
- src/pair.ts Pairing logic, QR display, key save
179
- src/crypto.ts Key generation + pairing string encode/decode
180
- ```
181
-
182
- ## More Info
183
-
184
- - [README.md](https://github.com/wipcomputer/memory-crystal/blob/main/README.md) ... What Memory Crystal is and how to install it.
185
- - [Technical Documentation](https://github.com/wipcomputer/memory-crystal/blob/main/TECHNICAL.md) ... Full technical documentation.
186
- - [QR Pairing Spec](ai/plan/2026-02-27--cc-mini--qr-pairing-spec.md) ... Full spec for the `crystal pair` command.
187
-
188
- ---
189
-
190
- ## License
191
-
192
- ```
193
- src/core.ts, cli.ts, mcp-server.ts, skills/ MIT (use anywhere, no restrictions)
194
- src/worker.ts, src/worker-mcp.ts AGPL (relay + cloud server)
195
- ```
196
-
197
- AGPL for personal use is free.
198
-
199
- Built by Parker Todd Brooks, Lēsa (OpenClaw, Claude Opus 4.6), Claude Code CLI (Claude Opus 4.6).
@@ -1,48 +0,0 @@
1
- # Release Notes: Memory Crystal v0.7.23
2
-
3
- **Date:** 2026-03-15
4
-
5
- ## Search Quality v2 + MLX Local LLM
6
-
7
- This release adds six search quality features ported from the QMD v2.0 analysis, plus the complete MLX local LLM infrastructure for Apple Silicon. Deep search is now disambiguatable, cacheable, debuggable, and can run entirely offline on Apple Silicon.
8
-
9
- ### Intent parameter
10
-
11
- Disambiguates queries without adding search terms. `crystal search "security" --intent "1Password"` steers results toward 1Password-related security instead of repo permissions or agent secrets. Intent flows through the expansion prompt (guides LLM variations), disables strong-signal bypass (keyword match might not be what the caller wants), and is prepended to the rerank query. Available via CLI `--intent` and MCP `intent`.
12
-
13
- ### Persistent LLM cache
14
-
15
- Expansion and reranking results are now cached in crystal.db (`llm_cache` table) with a 7-day TTL. Same query = instant on repeat searches. Reranking cache is content-addressable (keyed by query + sorted passage hashes), so identical content from different sessions shares cached scores. Configurable via `CRYSTAL_CACHE_TTL_DAYS` env var.
16
-
17
- ### Explain mode
18
-
19
- Per-result scoring breakdown showing FTS score, vector score, RRF rank, reranker score, recency weight, and final blended score. `crystal search "query" --explain`. Available via CLI `--explain` and MCP `explain`. Makes search quality transparent and debuggable.
20
-
21
- ### candidateLimit
22
-
23
- Tunable rerank pool size. `crystal search "query" --candidates 60`. Default stays 40. More candidates = better recall, slower reranking. Available via CLI `--candidates` and MCP `candidate_limit`.
24
-
25
- ### Structured search API
26
-
27
- `crystal.structuredSearch(queries)` accepts pre-expanded StructuredQuery[] with typed sub-queries (lex, vec, hyde). Skips LLM expansion entirely. Agents construct their own queries when they already know what they want. RRF fusion with first list weighted 2x.
28
-
29
- ### MLX local LLM (Phase 3)
30
-
31
- Complete auto-install infrastructure for running a local LLM on Apple Silicon:
32
-
33
- - `crystal mlx setup` detects Apple Silicon, installs mlx-lm (uv > pip3 > pip3 --user), creates LaunchAgent for always-on server
34
- - Model: `mlx-community/Qwen2.5-3B-Instruct-4bit` (~1.5 GB, fast on M-series)
35
- - Port 18791 (18789 OpenClaw, 18790 Crystal Core, 18791 MLX)
36
- - `crystal mlx status` and `crystal mlx stop` for server management
37
- - `crystal doctor` check #13: MLX health (not installed / down / running)
38
- - `crystal init` detects Apple Silicon and suggests MLX setup
39
- - State file at `~/.ldm/state/mlx-server.json`
40
-
41
- ### Also in this release
42
-
43
- - QMD v2.0 analysis documented (`ai/product/notes/`)
44
- - Search quality plan written (`ai/product/plans-prds/current/`)
45
- - MLX plan moved from upcoming to current
46
- - Stashed roadmap + readme-first updates recovered (PR #74)
47
-
48
- Closes #57, #63, #64.
@@ -1,24 +0,0 @@
1
- # Release Notes: memory-crystal v0.7.25
2
-
3
- Bump SKILL.md version and name to match package branding.
4
-
5
- ## What changed
6
-
7
- - SKILL.md version bumped from 0.4.0 to 0.7.24 (was stuck at the original version)
8
- - SKILL.md name changed from `memory` to `wip-memory-crystal` (matches branded convention)
9
- - Forces deploy to public repo, triggering auto-publish to wip.computer/install/
10
-
11
- ## Why
12
-
13
- The SKILL.md version was out of sync with the package version. The name didn't match the `wip-` branding convention used across all install files on wip.computer.
14
-
15
- ## Issues closed
16
-
17
- - #80
18
-
19
- ## How to verify
20
-
21
- ```bash
22
- crystal --version
23
- head -4 ~/.ldm/extensions/memory-crystal/skills/memory/SKILL.md
24
- ```
@@ -1,7 +0,0 @@
1
- # Memory Crystal v0.7.26
2
-
3
- Add repository field to package.json. GitHub Packages needs this to link packages to the repo.
4
-
5
- ## Issues closed
6
-
7
- - Closes #50
@@ -1,31 +0,0 @@
1
- # Release Notes: memory-crystal v0.7.28
2
-
3
- **Move all log paths from /tmp/ to ~/.ldm/logs/ so logs survive reboots.**
4
-
5
- ## What changed
6
-
7
- - Cron entry for crystal-capture now logs to `~/.ldm/logs/crystal-capture.log` instead of `/tmp/ldm-dev-tools/`
8
- - LaunchAgent plist template for ldm-backup now logs to `~/.ldm/logs/ldm-backup.log`
9
- - `mkdirSync` ensures `~/.ldm/logs/` exists instead of creating `/tmp/ldm-dev-tools/`
10
- - CLI output shows the correct log path
11
-
12
- ## Why
13
-
14
- macOS clears `/tmp/` on every reboot. All cron and LaunchAgent logs were lost after restart, making it impossible to debug issues. `~/.ldm/logs/` persists across reboots and is the correct home for LDM OS logs.
15
-
16
- ## Issues closed
17
-
18
- - wipcomputer/wip-ldm-os#120
19
-
20
- ## How to verify
21
-
22
- ```bash
23
- # After install, check that cron entry points to ~/.ldm/logs/
24
- crystal init --dry-run 2>&1 | grep crystal-capture
25
-
26
- # Check backup setup points to ~/.ldm/logs/
27
- crystal backup setup --dry-run 2>&1 | grep ldm-backup
28
-
29
- # Verify logs land in the right place
30
- ls ~/.ldm/logs/
31
- ```
@@ -1,28 +0,0 @@
1
- # Release Notes: memory-crystal v0.7.29
2
-
3
- **Doc audit: MLX setup, deep search params, log paths, role clarification.**
4
-
5
- ## What changed
6
-
7
- SKILL.md and TECHNICAL.md updated for 2 weeks of undocumented features:
8
-
9
- - **MLX local LLM:** Added as Option A in SKILL.md Step 2. CLI commands (setup, status, stop) added to TECHNICAL.md.
10
- - **Deep search parameters:** `--intent`, `--explain`, `--candidates` documented in both SKILL.md (crystal_search tool) and TECHNICAL.md (CLI reference + new sections for intent, explain, candidate limit, LLM cache).
11
- - **Log paths:** Fixed obsolete `/tmp/ldm-dev-tools/` reference to `~/.ldm/logs/`. Added logs/ to directory structure.
12
- - **Role clarification:** Two-role architecture (Core and Node) explicitly stated. Standalone role was removed in v0.7.22.
13
-
14
- ## Why
15
-
16
- 29 releases in 13 days. Docs didn't keep pace. Agents using crystal_search didn't know about --intent (query disambiguation) or --explain (scoring transparency).
17
-
18
- ## Issues closed
19
-
20
- - #57
21
-
22
- ## How to verify
23
-
24
- ```bash
25
- grep "intent" SKILL.md TECHNICAL.md
26
- grep "mlx" SKILL.md TECHNICAL.md
27
- grep "ldm/logs" TECHNICAL.md
28
- ```
@@ -1,64 +0,0 @@
1
- # Memory Crystal v0.7.4 ... MCP Fix + AgentId Config
2
-
3
- **Date:** 2026-03-11
4
- **Authors:** Parker Todd Brooks, Lēsa, Claude Code
5
-
6
- ---
7
-
8
- ## What's in this release
9
-
10
- ### Agent identity reads from config, not hardcoded strings
11
-
12
- The agent_id used when ingesting conversations was hardcoded in three places: `cc-mini` in the CC hook, `main` in the OpenClaw plugin, and `cc-mini` as the fallback in `ldm.ts`. This caused ID drift. The same agent got recorded under multiple IDs, and we had to manually merge 141K+ chunks in the database.
13
-
14
- Now `getAgentId()` scans `~/.ldm/agents/*/config.json` for a matching harness type. The CC hook passes `'claude-code'`, the OC plugin passes `'openclaw'`, and the config file is the source of truth. `CRYSTAL_AGENT_ID` env var still works as an override.
15
-
16
- New exports: `AgentConfig`, `loadAgentConfig()`, `saveAgentConfig()`. The installer writes `agentId` to config.json during `crystal init`.
17
-
18
- **Closes #33.**
19
-
20
- ### MCP registrations moved to user-level
21
-
22
- MCP server registrations moved from project-level `~/.openclaw/.mcp.json` to user-level `~/.claude.json`. The old file was a Claude Code convention that only loaded when running from `~/.openclaw/`. Now all 4 MCP servers (memory-crystal, lesa-bridge, wip-agent-pay, wip-repos) load from any directory as "User MCPs".
23
-
24
- OpenClaw doesn't read `.mcp.json` at all. It uses its own plugin system. The file was moved to `~/.openclaw/_trash/`.
25
-
26
- ### OPENCLAW_HOME env var fix (v0.7.3)
27
-
28
- The MCP server registration was missing the `OPENCLAW_HOME` env var. Without it, the memory-crystal MCP server couldn't find Lēsa's OpenClaw installation for private-mode checks. Fixed in v0.7.3, deployed in this release.
29
-
30
- ### Branch cleanup
31
-
32
- 33 stale branches renamed with `--merged-` suffix. Zero active branches besides main.
33
-
34
- ### QMD v1.1.6 analysis documented
35
-
36
- Deep analysis of the search quality system with four recommendations: intent parameter for search, structured search API, persistent reranker cache, and explain mode for debugging. See `ai/product/notes/2026-03-09--cc-mini--qmd-v1.1.6-analysis-and-recommendations.md`.
37
-
38
- ---
39
-
40
- ## Files changed
41
-
42
- | File | What |
43
- |------|------|
44
- | `src/ldm.ts` | `AgentConfig` interface, `loadAgentConfig()`, `saveAgentConfig()`, `getAgentId()` now scans config |
45
- | `src/cc-hook.ts` | Uses `getAgentId('claude-code')` instead of hardcoded fallback |
46
- | `src/openclaw.ts` | Uses `OC_AGENT_ID` instead of `'main'` fallback |
47
- | `src/installer.ts` | Writes `agentId` to config.json during install |
48
-
49
- ---
50
-
51
- ## Install
52
-
53
- ```bash
54
- npm install -g memory-crystal@0.7.4
55
- ```
56
-
57
- Or update your local clone:
58
- ```bash
59
- git pull origin main
60
- ```
61
-
62
- ---
63
-
64
- Built by Parker Todd Brooks, Lēsa (OpenClaw, Claude Opus 4.6), Claude Code (Claude Opus 4.6).
@@ -1,19 +0,0 @@
1
- # Release Notes: Memory Crystal v0.7.5
2
-
3
- ## LDM OS Integration
4
-
5
- Memory Crystal now works with LDM OS when it's available.
6
-
7
- ### crystal init delegates to ldm install
8
-
9
- When the `ldm` CLI exists on PATH, `crystal init` delegates generic deployment to it. LDM OS handles the scaffold, interface detection, and extension deployment. Memory Crystal keeps its own setup: database backup, role configuration, pairing, cron jobs.
10
-
11
- When `ldm` isn't available, `crystal init` works standalone like it always has. No new dependencies. No breaking changes.
12
-
13
- ### LDM OS tip
14
-
15
- After install completes, Memory Crystal prints a tip: "Run `ldm install` to see more skills you can add." Helps users discover the rest of the ecosystem.
16
-
17
- ### Part of LDM OS
18
-
19
- README now includes a "Part of LDM OS" section linking back to the LDM OS repo. Memory Crystal installs into LDM OS, the local runtime for AI agents.