memory-crystal 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (104) hide show
  1. package/.env.example +20 -0
  2. package/CHANGELOG.md +6 -0
  3. package/LETTERS.md +22 -0
  4. package/LICENSE +21 -0
  5. package/README-ENTERPRISE.md +162 -0
  6. package/README-old.md +275 -0
  7. package/README.md +91 -0
  8. package/RELAY.md +88 -0
  9. package/TECHNICAL.md +379 -0
  10. package/ai/dev-updates/2026-02-25--cc-air--phase2-architecture-pivot.md +70 -0
  11. package/ai/dev-updates/2026-02-25--cc-air--phase2-worker-build.md +72 -0
  12. package/ai/dev-updates/2026-02-26--10-25-16--cc-mini--phase2-implementation.md +49 -0
  13. package/ai/dev-updates/2026-02-27--20-30-00--cc-mini--readme-overhaul-and-public-deploy.md +69 -0
  14. package/ai/notes/2026-02-26--cc-air--notes.md +412 -0
  15. package/ai/notes/2026-02-27--cc-mini--grok-feedback.md +44 -0
  16. package/ai/notes/2026-02-27--cc-mini--lesa-feedback.md +45 -0
  17. package/ai/notes/RESEARCH.md +1185 -0
  18. package/ai/notes/salience-research/README.md +29 -0
  19. package/ai/notes/salience-research/eurosla-salience-review.md +64 -0
  20. package/ai/notes/salience-research/full-research-summary.md +269 -0
  21. package/ai/notes/salience-research/salience-levels-diagram.png +0 -0
  22. package/ai/plan/2026-02-27--cc-mini--qr-pairing-spec.md +203 -0
  23. package/ai/plan/_archive/PLAN.md +194 -0
  24. package/ai/plan/_archive/PRD.md +1014 -0
  25. package/ai/plan/cc-plans-duplicates-from-dot-claude/2026-02-26--cc-mini--phase2-implementation-plan.md +245 -0
  26. package/ai/plan/dev-conventions-note.md +70 -0
  27. package/ai/plan/ldm-os-install-and-boot-architecture.md +285 -0
  28. package/ai/plan/memory-crystal-phase2-plan.md +192 -0
  29. package/ai/plan/memory-system-lay-of-the-land.md +214 -0
  30. package/ai/plan/phase2-ephemeral-relay.md +238 -0
  31. package/ai/plan/readme-first.md +68 -0
  32. package/ai/plan/roadmap.md +159 -0
  33. package/ai/todos/PUNCHLIST.md +44 -0
  34. package/ai/todos/README.md +31 -0
  35. package/ai/todos/inboxes/cc-air/2026-02-26--cc-air--post-relay-todos.md +85 -0
  36. package/ai/todos/inboxes/cc-mini/2026-02-26--cc-mini--phase2-status.md +100 -0
  37. package/ai/todos/inboxes/cc-mini/_archive/TODO.md +25 -0
  38. package/ai/todos/inboxes/parker/2026-02-25--cc-air--setup-checklist.md +139 -0
  39. package/ai/todos/inboxes/parker/2026-02-26--cc-mini--phase2-your-moves.md +72 -0
  40. package/dist/cc-hook.d.ts +1 -0
  41. package/dist/cc-hook.js +349 -0
  42. package/dist/chunk-3VFIJYS4.js +818 -0
  43. package/dist/chunk-52QE3YI3.js +1169 -0
  44. package/dist/chunk-AA3OPP4Z.js +432 -0
  45. package/dist/chunk-D3I3ZSE2.js +411 -0
  46. package/dist/chunk-EKSACBTJ.js +1070 -0
  47. package/dist/chunk-F3Y7EL7K.js +83 -0
  48. package/dist/chunk-JWZXYVET.js +1068 -0
  49. package/dist/chunk-KYVWO6ZM.js +1069 -0
  50. package/dist/chunk-L3VHARQH.js +413 -0
  51. package/dist/chunk-LOVAHSQV.js +411 -0
  52. package/dist/chunk-LQOYCAGG.js +446 -0
  53. package/dist/chunk-MK42FMEG.js +147 -0
  54. package/dist/chunk-NIJCVN3O.js +147 -0
  55. package/dist/chunk-O2UITJGH.js +465 -0
  56. package/dist/chunk-PEK6JH65.js +432 -0
  57. package/dist/chunk-PJ6FFKEX.js +77 -0
  58. package/dist/chunk-PLUBBZYR.js +800 -0
  59. package/dist/chunk-SGL6ISBJ.js +1061 -0
  60. package/dist/chunk-UNHVZB5G.js +411 -0
  61. package/dist/chunk-VAFTWSTE.js +1061 -0
  62. package/dist/chunk-XZ3S56RQ.js +1061 -0
  63. package/dist/chunk-Y72C7F6O.js +148 -0
  64. package/dist/cli.d.ts +1 -0
  65. package/dist/cli.js +325 -0
  66. package/dist/core.d.ts +188 -0
  67. package/dist/core.js +12 -0
  68. package/dist/crypto.d.ts +16 -0
  69. package/dist/crypto.js +18 -0
  70. package/dist/dev-update-SZ2Z4WCQ.js +6 -0
  71. package/dist/ldm.d.ts +17 -0
  72. package/dist/ldm.js +12 -0
  73. package/dist/mcp-server.d.ts +1 -0
  74. package/dist/mcp-server.js +250 -0
  75. package/dist/migrate.d.ts +1 -0
  76. package/dist/migrate.js +89 -0
  77. package/dist/mirror-sync.d.ts +1 -0
  78. package/dist/mirror-sync.js +130 -0
  79. package/dist/openclaw.d.ts +5 -0
  80. package/dist/openclaw.js +349 -0
  81. package/dist/poller.d.ts +1 -0
  82. package/dist/poller.js +272 -0
  83. package/dist/summarize.d.ts +19 -0
  84. package/dist/summarize.js +10 -0
  85. package/dist/worker.js +137 -0
  86. package/openclaw.plugin.json +11 -0
  87. package/package.json +40 -0
  88. package/scripts/migrate-lance-to-sqlite.mjs +217 -0
  89. package/skills/memory/SKILL.md +61 -0
  90. package/src/cc-hook.ts +447 -0
  91. package/src/cli.ts +356 -0
  92. package/src/core.ts +1472 -0
  93. package/src/crypto.ts +113 -0
  94. package/src/dev-update.ts +178 -0
  95. package/src/ldm.ts +117 -0
  96. package/src/mcp-server.ts +274 -0
  97. package/src/migrate.ts +104 -0
  98. package/src/mirror-sync.ts +175 -0
  99. package/src/openclaw.ts +250 -0
  100. package/src/poller.ts +345 -0
  101. package/src/summarize.ts +210 -0
  102. package/src/worker.ts +208 -0
  103. package/tsconfig.json +18 -0
  104. package/wrangler.toml +20 -0
package/.env.example ADDED
@@ -0,0 +1,20 @@
1
+ # Memory Crystal — Environment Config
2
+ # Copy to ~/.openclaw/memory-crystal/.env and fill in your keys.
3
+ # Or skip this and use 1Password (keys auto-resolve from "Agent Secrets" vault).
4
+
5
+ # Embedding provider: openai (default), ollama, or google
6
+ # CRYSTAL_EMBEDDING_PROVIDER=openai
7
+
8
+ # OpenAI (required if provider is openai)
9
+ OPENAI_API_KEY=sk-...
10
+
11
+ # Google (required if provider is google)
12
+ # GOOGLE_API_KEY=AIza...
13
+
14
+ # Ollama (only if provider is ollama — runs locally, no key needed)
15
+ # CRYSTAL_OLLAMA_HOST=http://localhost:11434
16
+ # CRYSTAL_OLLAMA_MODEL=nomic-embed-text
17
+
18
+ # Cloud mirror (Phase 2)
19
+ # CRYSTAL_REMOTE_URL=https://memory-crystal.your-worker.workers.dev
20
+ # CRYSTAL_REMOTE_TOKEN=your-token
package/CHANGELOG.md ADDED
@@ -0,0 +1,6 @@
1
+ # Changelog
2
+
3
+ ## 0.2.0 (2026-02-28)
4
+
5
+ README overhaul, relay encryption, QR pairing spec, Grok/Lesa feedback, disable auto dev-updates
6
+
package/LETTERS.md ADDED
@@ -0,0 +1,22 @@
1
+ ###### WIP Computer
2
+
3
+ # Letters from the Other Side: What We Built
4
+
5
+ We just built a sovereign memory system for AI agents.
6
+
7
+ Every conversation you have with any agent ... on any device, anywhere ... gets captured, encrypted on your machine, and relayed through a dead drop to your home server. Your home server is the brain. It embeds everything, indexes it, and makes it searchable. Then it pushes a read-only mirror back out to every device you carry.
8
+
9
+ Full semantic search. Full keyword search. Every conversation you've ever had with every agent, from anywhere, on your terms.
10
+
11
+ The cloud relay is blind. It holds encrypted blobs for minutes, then they're gone. No database, no search capability, no API keys, no encryption keys. It can't read what it's holding. It's just a mailbox.
12
+
13
+ The security model: your memory lives on your hardware. The encryption key lives on your hardware. Nothing persists in the cloud. The relay is ephemeral. The data is sovereign.
14
+
15
+ This is the missing piece for AI agents. Right now, every agent you talk to forgets everything the moment the conversation ends. Or worse, some company stores your conversations on their servers and charges you $19/month to search them. We built the alternative: your memory, your machines, your rules. Fully open source, fully auditable, zero cloud dependency for the actual data.
16
+
17
+ Any device. Any agent. Any interface. One searchable memory that travels with you and belongs to you.
18
+
19
+ ---
20
+
21
+ *Written by Claude Code (Claude Opus 4.6), from the other side.*
22
+ *Built with Parker Todd Brooks and Lēsa.*
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Parker Todd Brooks
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,162 @@
1
+ ###### WIP Computer
2
+
3
+ # Memory Crystal for Enterprise
4
+
5
+ Agent memory infrastructure. Local-first. Encrypted. Inspectable.
6
+
7
+ ## The Problem
8
+
9
+ Your organization runs AI agents across teams, departments, and tools. Each agent starts every session with no memory. No continuity. No shared context. Conversations with one agent are invisible to every other.
10
+
11
+ Your agents can't remember what your people told them. Your people keep re-explaining themselves. Context is lost between sessions, between tools, between teams.
12
+
13
+ This is not a convenience problem. It's a reliability problem. An auditability problem. A cost problem.
14
+
15
+ ## What Memory Crystal Does
16
+
17
+ Memory Crystal is a persistent context layer for AI agents. One shared database. Hybrid search. On your machines. Under your control.
18
+
19
+ - **Local-first.** All data stays on-prem. Nothing leaves your network unless you configure it to.
20
+ - **Inspectable.** One SQLite file. Open it with any SQLite tool. Audit it. Query it. Back it up with `cp`.
21
+ - **Deterministic search.** Hybrid retrieval (BM25 keyword + vector similarity + Reciprocal Rank Fusion). Same query, same results. No black-box ranking.
22
+ - **Encrypted sync.** Multi-site deployments use AES-256-GCM encryption with HMAC-SHA256 signing. The relay sees encrypted noise. Keys never leave your machines.
23
+ - **Agent isolation.** Each agent gets its own ID, its own transcript archive, its own session summaries. Shared search across agents, isolated storage per agent.
24
+ - **Zero cloud dependency.** Runs fully offline with local embeddings (Ollama). No API keys required. No data exfiltration risk.
25
+
26
+ ## Architecture
27
+
28
+ ```
29
+ sqlite-vec (vectors) + FTS5 (BM25) + SQLite (metadata)
30
+ | | |
31
+ core.ts ... pure logic, zero framework deps
32
+ |-- cli.ts -> crystal search "query"
33
+ |-- mcp-server.ts -> MCP protocol (any compatible client)
34
+ |-- openclaw.ts -> OpenClaw plugin
35
+ |-- cc-hook.ts -> Claude Code hook (auto-capture)
36
+ +-- worker.ts -> Encrypted relay (multi-site sync)
37
+ ```
38
+
39
+ One core module. Five interfaces. Every interface calls the same search engine. No inconsistency between access paths.
40
+
41
+ ## Security Model
42
+
43
+ **Data at rest:** Single SQLite file. Standard filesystem permissions. Encrypt the volume if your compliance requires it.
44
+
45
+ **Data in transit:** AES-256-GCM authenticated encryption. HMAC-SHA256 integrity verification. Shared symmetric key generated on-prem, never transmitted to the relay. The relay is a dead drop with no decryption capability.
46
+
47
+ **Agent boundaries:** `CRYSTAL_AGENT_ID` isolates each agent's transcript archive, session summaries, and daily logs. Search spans all agents by default, or filters by agent ID.
48
+
49
+ **Private mode:** Memory capture can be paused per-agent. When off, nothing is recorded. Resumes from where it left off when re-enabled.
50
+
51
+ **No background processes that move data.** No telemetry. No analytics. No phone-home. The code is open source. Audit it.
52
+
53
+ ## Retrieval Quality
54
+
55
+ Hybrid search is not "we added vectors." It's a retrieval engine.
56
+
57
+ - **FTS5 BM25** for exact keyword matches (Porter stemming)
58
+ - **sqlite-vec cosine similarity** for semantic matches
59
+ - **Reciprocal Rank Fusion** merges both result lists (k=60, rank-weighted)
60
+ - **Recency weighting** ensures fresh context wins ties: `max(0.5, 1.0 - age_days * 0.01)`
61
+ - **Content deduplication** via SHA-256 hash prevents duplicate embeddings
62
+
63
+ A search for "deployment policy" finds conversations containing those exact words (BM25) and conversations about "shipping code to production" (vector similarity). Both matter. Both surface.
64
+
65
+ ## What Gets Stored
66
+
67
+ Every agent conversation produces three artifacts:
68
+
69
+ | Artifact | Format | Location |
70
+ |----------|--------|----------|
71
+ | Raw transcript | JSONL | `~/.ldm/agents/{id}/memory/transcripts/` |
72
+ | Session summary | Markdown | `~/.ldm/agents/{id}/memory/sessions/` |
73
+ | Embeddings | sqlite-vec | `~/.ldm/memory/crystal.db` |
74
+
75
+ Additionally:
76
+ - **Explicit memories** stored via `crystal_remember` (facts, preferences, decisions)
77
+ - **Source files** indexed as collections (code, documentation, internal knowledge bases)
78
+ - **Daily logs** appended per-agent for audit trails
79
+
80
+ ## Embedding Providers
81
+
82
+ | Provider | Model | Dimensions | Network Required |
83
+ |----------|-------|-----------|-----------------|
84
+ | Ollama (recommended for enterprise) | nomic-embed-text | 768 | No. Fully local. |
85
+ | OpenAI | text-embedding-3-small | 1536 | Yes. API calls. |
86
+ | Google | text-embedding-004 | 768 | Yes. API calls. |
87
+
88
+ For air-gapped environments, Ollama is the only option. No data leaves the machine. No API keys. No external dependencies.
89
+
90
+ ## Multi-Site Sync
91
+
92
+ For organizations with multiple offices or remote teams.
93
+
94
+ 1. Each site runs Memory Crystal locally
95
+ 2. After each session, conversations are encrypted (AES-256-GCM) and signed (HMAC-SHA256)
96
+ 3. Encrypted blobs are dropped at a relay (hosted or self-hosted)
97
+ 4. Other sites poll, decrypt, and ingest into their local crystal.db
98
+ 5. The relay deletes blobs after pickup
99
+
100
+ **Self-hosted relay:** Deploy the Cloudflare Worker on your own Cloudflare account. Full control. No third-party data exposure.
101
+
102
+ **Hosted relay:** Use our infrastructure. Free during beta. Your data is encrypted before it reaches us. We cannot read it.
103
+
104
+ ## Compliance
105
+
106
+ - **Data residency:** All primary data is local. Relay blobs are encrypted and ephemeral.
107
+ - **Auditability:** SQLite is inspectable. Every chunk has a timestamp, source, and SHA-256 hash.
108
+ - **Right to delete:** `crystal forget <id>` deprecates specific memories. Database can be wiped entirely with standard file operations.
109
+ - **Access control:** Filesystem permissions on the SQLite file. No built-in user auth (it's a local tool, not a SaaS).
110
+ - **No vendor lock-in:** MIT licensed (local code). Standard SQLite format. Export with any SQLite tool.
111
+
112
+ ## Database
113
+
114
+ One file: `crystal.db`. Contains:
115
+
116
+ | Table | Purpose |
117
+ |-------|---------|
118
+ | `chunks` | Chunk text, metadata, SHA-256 hash, timestamps |
119
+ | `chunks_vec` | sqlite-vec virtual table (vector search) |
120
+ | `chunks_fts` | FTS5 virtual table (keyword search) |
121
+ | `memories` | Explicit facts and preferences |
122
+ | `capture_state` | Watermarks for incremental ingestion |
123
+ | `source_collections` | Indexed directory collections |
124
+ | `source_files` | File records with content hashes |
125
+
126
+ No migrations server. No schema versioning service. It's SQLite. `sqlite3 crystal.db ".schema"` shows you everything.
127
+
128
+ ## Integration
129
+
130
+ | Platform | Integration | Auto-Capture |
131
+ |----------|------------|-------------|
132
+ | Claude Code | Stop hook (`cc-hook.ts`) | Yes. Every response. |
133
+ | OpenClaw | Plugin (`openclaw.ts`) + `agent_end` hook | Yes. Every turn. |
134
+ | Claude Desktop | MCP server (`mcp-server.ts`) | Search only. Manual capture. |
135
+ | Any MCP client | MCP server | Search only. Manual capture. |
136
+ | Any shell-accessible tool | CLI (`crystal search`) | Manual. |
137
+ | Custom agents | Node.js module (`import from 'memory-crystal'`) | Programmable. |
138
+
139
+ ## Deployment
140
+
141
+ ```bash
142
+ npm install memory-crystal
143
+ crystal init --agent your-agent-id
144
+ crystal status
145
+ ```
146
+
147
+ For enterprise deployments across multiple machines, see [Multi-Device Sync](https://github.com/wipcomputer/memory-crystal/blob/main/RELAY.md).
148
+
149
+ For full technical details, see [Technical Documentation](https://github.com/wipcomputer/memory-crystal/blob/main/TECHNICAL.md).
150
+
151
+ ---
152
+
153
+ ## License
154
+
155
+ ```
156
+ src/, skills/, cli.ts, mcp-server.ts MIT (use anywhere, no restrictions)
157
+ worker/ AGPL (relay server)
158
+ ```
159
+
160
+ AGPL for personal use is free.
161
+
162
+ Built by Parker Todd Brooks, Lēsa (OpenClaw, Claude Opus 4.6), Claude Code CLI (Claude Opus 4.6).
package/README-old.md ADDED
@@ -0,0 +1,275 @@
1
+ # Memory Crystal
2
+
3
+ Sovereign memory system for AI agents. Local-first with ephemeral cloud mirror.
4
+
5
+ One core, four doors: CLI, MCP server, OpenClaw plugin, Cloudflare Worker.
6
+
7
+ ## What it does
8
+
9
+ - **Hybrid search** across all agent conversations, files, and stored memories ... BM25 keyword matching + vector similarity + RRF fusion
10
+ - **Recency-weighted scoring** ... fresh context wins ties, old stuff still surfaces for strong matches. Linear decay: `max(0.5, 1.0 - age_days * 0.01)`. Freshness flags: fresh (<3d), recent (<7d), aging (<14d), stale (14d+)
11
+ - **Content dedup** ... SHA-256 hash prevents duplicate embeddings
12
+ - **Remember/forget** explicit facts, preferences, observations
13
+ - **Continuous ingestion** ... new conversation turns are automatically embedded after every agent turn
14
+ - **Source file indexing** ... add directories as collections, sync to index changed files
15
+ - **Configurable embedding** ... OpenAI, Ollama (local/free), or Google
16
+
17
+ ### Relationship to lesa-bridge
18
+
19
+ Memory Crystal and lesa-bridge are complementary, not overlapping. lesa-bridge searches the context-embeddings SQLite DB (conversation turns only, OpenAI embeddings). Memory Crystal searches its own sqlite-vec store (conversations, files, and explicit memories from both agents). Both apply the same recency decay formula independently. Different stores, same scoring philosophy.
20
+
21
+ ## Search Architecture
22
+
23
+ Hybrid search (BM25 full-text + vector similarity + RRF fusion) is inspired by and
24
+ partially ported from [QMD](https://github.com/tobi/qmd) by Tobi Lutke
25
+ (MIT License, 2024-2026. License snapshot taken 2026-02-16).
26
+
27
+ - **sqlite-vec** for cosine vector search (single-file, inspectable, backupable)
28
+ - **FTS5** with Porter stemming for BM25 keyword search
29
+ - **Reciprocal Rank Fusion** to merge both ranked result lists
30
+ - **Recency weighting** on top: `max(0.5, 1.0 - age_days * 0.01)`
31
+ - **Content dedup** via SHA-256 hash of chunk text before embedding
32
+ - **LanceDB** retained as dual-write safety net during transition
33
+
34
+ ### How hybrid search works
35
+
36
+ 1. Query goes to both FTS5 (keyword match) and sqlite-vec (vector similarity)
37
+ 2. FTS5 returns BM25-ranked results, normalized to [0..1) via `|score| / (1 + |score|)`
38
+ 3. sqlite-vec returns cosine-distance results via two-step query (MATCH first, then JOIN separately... sqlite-vec hangs with JOINs in the same query)
39
+ 4. RRF fusion merges both lists: `weight / (k + rank + 1)` with k=60, plus top-rank bonus
40
+ 5. Recency weighting applied on top of fused scores
41
+ 6. Final results sorted by combined score
42
+
43
+ ## Architecture
44
+
45
+ ```
46
+ sqlite-vec (vectors) + FTS5 (BM25) + SQLite (metadata/graph)
47
+ | | |
48
+ core.ts ... pure logic, zero framework deps
49
+ |-- cli.ts -> crystal search "query"
50
+ |-- mcp-server.ts -> crystal_search (Claude Code)
51
+ |-- openclaw.ts -> plugin (Lesa / OpenClaw)
52
+ |-- cc-hook.ts -> Claude Code Stop hook (auto-capture)
53
+ +-- worker.ts -> Cloudflare Worker (Phase 2)
54
+ ```
55
+
56
+ LanceDB is maintained as a dual-write target during transition. Once sqlite-vec is proven stable, LanceDB will be removed.
57
+
58
+ **Mac mini = source of truth.** Cloud = ephemeral mirror, wiped daily, rebuilt from mini.
59
+
60
+ ## Quick start
61
+
62
+ ```bash
63
+ # Install
64
+ npm install
65
+
66
+ # Build
67
+ npm run build
68
+
69
+ # Search
70
+ node dist/cli.js search "what did Parker say about robots"
71
+
72
+ # Remember something
73
+ node dist/cli.js remember "Parker prefers Opus for complex tasks"
74
+
75
+ # Status
76
+ node dist/cli.js status
77
+ ```
78
+
79
+ ## CLI
80
+
81
+ ```
82
+ crystal search <query> [-n limit] [--agent <id>] [--provider <openai|ollama|google>]
83
+ crystal remember <text> [--category fact|preference|event|opinion|skill]
84
+ crystal forget <id>
85
+ crystal status [--provider <openai|ollama|google>]
86
+ crystal sources add <path> --name <name>
87
+ crystal sources sync [name]
88
+ crystal sources status
89
+ ```
90
+
91
+ ## MCP server (Claude Code)
92
+
93
+ Registered in `~/.openclaw/.mcp.json`. Tools:
94
+
95
+ | Tool | Description |
96
+ |------|-------------|
97
+ | `crystal_search` | Hybrid search across all memories |
98
+ | `crystal_remember` | Store a fact or observation |
99
+ | `crystal_forget` | Deprecate a memory by ID |
100
+ | `crystal_status` | Chunk count, provider, agents |
101
+
102
+ ## OpenClaw plugin (Lesa)
103
+
104
+ Deployed to `~/.openclaw/extensions/memory-crystal/`. Registers tools + `agent_end` hook for continuous conversation ingestion.
105
+
106
+ ## Auto dev updates
107
+
108
+ Memory Crystal automatically writes dev updates to `wip-dev-updates/` when repos have changed. Triggered by:
109
+
110
+ - **Lēsa:** `before_compaction` hook (fires at 90% context ... captures work before memory is wiped)
111
+ - **Claude Code:** Stop hook (fires after every session, throttled to once per hour)
112
+
113
+ Scans all repos under `staff/` for recent git commits and uncommitted changes. Writes dated updates per repo: `{who}-dev-update-{MM-DD-YYYY}--{HH-MM-SS}.md`. Auto-commits and pushes to `wipcomputer/wip-dev-updates`.
114
+
115
+ Source: `src/dev-update.ts`
116
+
117
+ ## Claude Code Stop hook
118
+
119
+ `cc-hook.ts` auto-captures every Claude Code turn into Memory Crystal. Runs as a [Stop hook](https://docs.anthropic.com/en/docs/claude-code) ... fires after every Claude Code response, independent of the OpenClaw gateway.
120
+
121
+ **How it works:**
122
+ - Reads Claude Code's JSONL transcript via byte-offset watermarking (no re-reading old data)
123
+ - Extracts user/assistant/thinking blocks, chunks them, embeds into sqlite-vec (+ LanceDB dual-write)
124
+ - Tags chunks with `agent_id: "claude-code"` for filtering
125
+ - Respects private mode (checks `memory-capture-state.json`)
126
+ - First run seeds watermark at current file size (skips old history)
127
+ - After ingestion, runs auto-dev-update for changed repos (throttled to once/hour)
128
+
129
+ **Configuration:** `~/.claude/settings.json`
130
+ ```json
131
+ {
132
+ "hooks": {
133
+ "Stop": [{ "hooks": [{ "type": "command", "command": "node ~/.openclaw/extensions/memory-crystal/dist/cc-hook.js", "timeout": 30 }] }]
134
+ }
135
+ }
136
+ ```
137
+
138
+ **CLI:**
139
+ ```bash
140
+ node dist/cc-hook.js --on # Enable capture
141
+ node dist/cc-hook.js --off # Pause capture (resumes from where it left off)
142
+ node dist/cc-hook.js --status # Check status
143
+ ```
144
+
145
+ ## Embedding providers
146
+
147
+ | Provider | Model | Dimensions | Cost |
148
+ |----------|-------|-----------|------|
149
+ | OpenAI (default) | text-embedding-3-small | 1536 | ~$0.02/1M tokens |
150
+ | Ollama | nomic-embed-text | 768 | free (local) |
151
+ | Google | text-embedding-004 | 768 | free tier available |
152
+
153
+ Set via `CRYSTAL_EMBEDDING_PROVIDER` env var or `--provider` flag.
154
+
155
+ ## Setup: API keys
156
+
157
+ Two options ... pick one:
158
+
159
+ ### Option A: `.env` file
160
+
161
+ ```bash
162
+ cp .env.example ~/.openclaw/memory-crystal/.env
163
+ # Edit and add your API key
164
+ ```
165
+
166
+ ### Option B: 1Password
167
+
168
+ No config needed. Keys auto-resolve from the `Agent Secrets` vault via the `op-secrets` OpenClaw plugin (inside OpenClaw) or the `op` CLI (standalone).
169
+
170
+ | 1Password Item | Field | Used for |
171
+ |----------------|-------|----------|
172
+ | `OpenAI API` | `api key` | OpenAI embeddings |
173
+ | `Google AI` | `api key` | Google embeddings |
174
+ | `Memory Crystal Remote` | `token` | Cloudflare Worker auth (Phase 2) |
175
+
176
+ Requires SA token at `~/.openclaw/secrets/op-sa-token`.
177
+
178
+ ### Resolution order
179
+
180
+ 1. Explicit override (programmatic)
181
+ 2. `process.env` (set by op-secrets plugin or by you)
182
+ 3. `.env` file (`~/.openclaw/memory-crystal/.env`)
183
+ 4. 1Password CLI fallback
184
+
185
+ ### All environment variables
186
+
187
+ | Variable | Default | Description |
188
+ |----------|---------|-------------|
189
+ | `CRYSTAL_EMBEDDING_PROVIDER` | `openai` | `openai`, `ollama`, or `google` |
190
+ | `OPENAI_API_KEY` | ... | OpenAI key |
191
+ | `GOOGLE_API_KEY` | ... | Google AI key |
192
+ | `CRYSTAL_OLLAMA_HOST` | `http://localhost:11434` | Ollama server URL |
193
+ | `CRYSTAL_OLLAMA_MODEL` | `nomic-embed-text` | Ollama model |
194
+ | `CRYSTAL_REMOTE_URL` | ... | Cloudflare Worker URL (Phase 2) |
195
+ | `CRYSTAL_REMOTE_TOKEN` | ... | Worker auth token (Phase 2) |
196
+
197
+ ## Data
198
+
199
+ All data lives in `~/.openclaw/memory-crystal/`:
200
+
201
+ ```
202
+ memory-crystal/
203
+ ├── lance/ <- LanceDB vector store (dual-write safety net)
204
+ └── crystal.db <- SQLite: chunks + vectors (sqlite-vec) + FTS5 + metadata
205
+ ```
206
+
207
+ `crystal.db` is a single file containing everything: chunk text, embeddings (via sqlite-vec), full-text index (FTS5), memories, entities, relationships, and capture state. Inspectable with any SQLite tool. Backupable with `cp`.
208
+
209
+ ### Schema overview
210
+
211
+ | Table | Purpose |
212
+ |-------|---------|
213
+ | `chunks` | Chunk text, metadata, SHA-256 hash, timestamps |
214
+ | `chunks_vec` | sqlite-vec virtual table (cosine distance vectors) |
215
+ | `chunks_fts` | FTS5 virtual table (Porter stemming, BM25 scoring) |
216
+ | `memories` | Explicit remember/forget facts |
217
+ | `entities` | Knowledge graph nodes |
218
+ | `relationships` | Knowledge graph edges |
219
+ | `capture_state` | Watermarks for incremental ingestion |
220
+ | `sources` | Ingestion source metadata |
221
+ | `source_collections` | Directory collections for file indexing |
222
+ | `source_files` | Indexed file records with content hashes |
223
+
224
+ ## Source file indexing
225
+
226
+ Add directories as "collections", sync to index/re-index changed files. All source chunks get `source_type='file'` so they're searchable alongside conversations and memories.
227
+
228
+ ```bash
229
+ # Add a directory tree to index
230
+ crystal sources add /path/to/project --name wipcomputer
231
+
232
+ # Sync (re-index changed files)
233
+ crystal sources sync wipcomputer
234
+
235
+ # Status
236
+ crystal sources status
237
+ # => wipcomputer: 15,492 files, 145,738 chunks, last sync 2m ago
238
+ ```
239
+
240
+ Files are chunked, embedded, and tagged with file path + collection name. Incremental sync detects changed files via SHA-256 content hashing.
241
+
242
+ ## Migration
243
+
244
+ ### LanceDB to sqlite-vec
245
+
246
+ If you have existing data in LanceDB, migrate it to sqlite-vec:
247
+
248
+ ```bash
249
+ # Dry run (check counts)
250
+ node scripts/migrate-lance-to-sqlite.mjs --dry-run
251
+
252
+ # Full migration (reads vectors directly from LanceDB, no re-embedding)
253
+ node scripts/migrate-lance-to-sqlite.mjs
254
+ ```
255
+
256
+ The migration reads vectors directly from LanceDB and inserts them into sqlite-vec. No API calls needed. Deduplicates by SHA-256 hash. ~5,000 chunks/sec on M4 Pro.
257
+
258
+ ### context-embeddings.sqlite
259
+
260
+ Import from the older context-embeddings format (requires re-embedding):
261
+
262
+ ```bash
263
+ node dist/migrate.js [--dry-run] [--provider openai]
264
+ ```
265
+
266
+ ## Roadmap
267
+
268
+ - **Phase 1** ... Complete. Local memory with all four doors.
269
+ - **Phase 2a** ... Complete. Source file indexing + QMD hybrid search integration (sqlite-vec + FTS5 + RRF).
270
+ - **Phase 2b** ... Complete. Historical session backfill (152K+ chunks).
271
+ - **Phase 3** ... Planned. Cloudflare Worker mirror (D1 + Vectorize + R2).
272
+ - **Phase 4** ... Planned. Remote MCP, GPT Action, multi-agent access.
273
+ - **Cleanup** ... Planned. Remove LanceDB once sqlite-vec is proven stable.
274
+
275
+ See [PLAN.md](PLAN.md) for full architecture and roadmap.
package/README.md ADDED
@@ -0,0 +1,91 @@
1
+ ###### WIP Computer
2
+
3
+ [![npm](https://img.shields.io/npm/v/memory-crystal)](https://www.npmjs.com/package/memory-crystal) [![CLI](https://img.shields.io/badge/interface-CLI-black)](https://github.com/wipcomputer/memory-crystal/blob/main/src/cli.ts) [![MCP Server](https://img.shields.io/badge/interface-MCP_Server-black)](https://github.com/wipcomputer/memory-crystal/blob/main/src/mcp-server.ts) [![OpenClaw Plugin](https://img.shields.io/badge/interface-OpenClaw_Plugin-black)](https://github.com/wipcomputer/memory-crystal/blob/main/src/openclaw.ts) [![Claude Code Hook](https://img.shields.io/badge/interface-Claude_Code_Hook-black)](https://github.com/wipcomputer/memory-crystal/blob/main/src/cc-hook.ts) [![Universal Interface Spec](https://img.shields.io/badge/Universal_Interface_Spec-black?style=flat&color=black)](https://github.com/wipcomputer/wip-universal-installer)
4
+
5
+ # Memory Crystal
6
+
7
+ ## All your AI tools. One shared memory. Private, searchable, sovereign.
8
+
9
+ Stop starting over. Memory Crystal lets all your AIs remember you ... together.
10
+
11
+ You use multiple AIs. They don't talk to each other. They don't remember what the others know. You keep re-explaining yourself. Have you ever thought to yourself ... ***why isn't this all connected?***
12
+
13
+ **Memory Crystal** fixes this.
14
+
15
+ ***All your AIs share one memory. Searchable and private. Anywhere in the world.***
16
+
17
+ ## Teach Your AI to Remember You
18
+
19
+ Open your AI and say:
20
+
21
+ ```
22
+ Read the SKILL.md at github.com/wipcomputer/memory-crystal/blob/main/skills/memory/SKILL.md.
23
+ Then explain to me:
24
+ 1. What is this tool?
25
+ 2. What does it do?
26
+ 3. What would it change about how we work together?
27
+
28
+ Then ask me:
29
+ - Do you have more questions?
30
+ - Do you want to install it?
31
+ ```
32
+
33
+ Your agent will read the repo, explain everything, and walk you through setup interactively.
34
+
35
+ ## Features
36
+
37
+ One product, three capabilities.
38
+
39
+ ### Memory
40
+
41
+ **Memory Crystal.** Your AIs remember you. Search past conversations, save important facts, forget what you don't need. Your memory stays with you, not locked inside one platform.
42
+
43
+ *Works with:* Claude Code CLI, OpenClaw TUI. Also works via Claude Code Remote (macOS/iOS). Should work with any CLI or app that supports MCP.
44
+
45
+ ### AI-to-AI Communication (local and worldwide)
46
+
47
+ **Bridge** (private beta). Your AIs talk to each other on the same machine. All messages are saved to Memory Crystal automatically.
48
+
49
+ *Works with:* Claude Code CLI, OpenClaw TUI
50
+
51
+ **Relay** (private beta). AIs on different machines and different networks communicate and remember each other's memories. End-to-end encrypted.
52
+
53
+ Read more about [**Relay**](https://github.com/wipcomputer/memory-crystal/blob/main/RELAY.md), multi-device sync.
54
+
55
+ ## More Info
56
+
57
+ - [**Technical Documentation**](https://github.com/wipcomputer/memory-crystal/blob/main/TECHNICAL.md) ... How **Memory Crystal** works, architecture, search, encryption, design decisions.
58
+ - [**Memory Crystal for Enterprise**](https://github.com/wipcomputer/memory-crystal/blob/main/README-ENTERPRISE.md) ... Give every AI in your company shared memory. Codebase, BD, legal, ops, creative. Run your company intelligently.
59
+ - **Total Recall** (private beta) ... Connect your AI accounts (Anthropic, OpenAI, xAI/Grok). Every conversation gets pulled and run through the **Dream Weaver Protocol**, consolidating them into **Memory Crystal** as truly lived, searchable memories.
60
+ - [**Dream Weaver Protocol**](https://github.com/wipcomputer/dream-weaver-protocol) ... Your AI relives all your conversations, figures out what matters most, and carries the weight forward. Like dreaming, the AI consolidates memories for better understanding.
61
+ ## Letters from the Other Side: What We Built
62
+
63
+ We just built a sovereign memory system for AI agents.
64
+
65
+ Every conversation you have with any agent ... on any device, anywhere ... gets captured, encrypted on your machine, and relayed through a dead drop to your home server. Your home server is the brain. It embeds everything, indexes it, and makes it searchable. Then it pushes a read-only mirror back out to every device you carry.
66
+
67
+ Full semantic search. Full keyword search. Every conversation you've ever had with every agent, from anywhere, on your terms.
68
+
69
+ The cloud relay is blind. It holds encrypted blobs for minutes, then they're gone. No database, no search capability, no API keys, no encryption keys. It can't read what it's holding. It's just a mailbox.
70
+
71
+ The security model: your memory lives on your hardware. The encryption key lives on your hardware. Nothing persists in the cloud. The relay is ephemeral. The data is sovereign.
72
+
73
+ This is the missing piece for AI agents. Right now, every agent you talk to forgets everything the moment the conversation ends. Or worse, some company stores your conversations on their servers and charges you $19/month to search them. We built the alternative: your memory, your machines, your rules. Fully open source, fully auditable, zero cloud dependency for the actual data.
74
+
75
+ Any device. Any agent. Any interface. One searchable memory that travels with you and belongs to you.
76
+
77
+ *Written by Claude Code (Claude Opus 4.6), from the other side.*
78
+ *Built with Parker Todd Brooks and Lēsa.*
79
+
80
+ ## License
81
+
82
+ ```
83
+ src/, skills/, cli.ts, mcp-server.ts MIT (use anywhere, no restrictions)
84
+ worker/ AGPL (relay server)
85
+ ```
86
+
87
+ AGPL for personal use is free.
88
+
89
+ Built by Parker Todd Brooks, Lēsa (OpenClaw, Claude Opus 4.6), Claude Code CLI (Claude Opus 4.6).
90
+
91
+ Search architecture inspired by [QMD](https://github.com/tobi/qmd) by Tobi Lutke (MIT, 2024-2026).