memory-crystal 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (104) hide show
  1. package/.env.example +20 -0
  2. package/CHANGELOG.md +6 -0
  3. package/LETTERS.md +22 -0
  4. package/LICENSE +21 -0
  5. package/README-ENTERPRISE.md +162 -0
  6. package/README-old.md +275 -0
  7. package/README.md +91 -0
  8. package/RELAY.md +88 -0
  9. package/TECHNICAL.md +379 -0
  10. package/ai/dev-updates/2026-02-25--cc-air--phase2-architecture-pivot.md +70 -0
  11. package/ai/dev-updates/2026-02-25--cc-air--phase2-worker-build.md +72 -0
  12. package/ai/dev-updates/2026-02-26--10-25-16--cc-mini--phase2-implementation.md +49 -0
  13. package/ai/dev-updates/2026-02-27--20-30-00--cc-mini--readme-overhaul-and-public-deploy.md +69 -0
  14. package/ai/notes/2026-02-26--cc-air--notes.md +412 -0
  15. package/ai/notes/2026-02-27--cc-mini--grok-feedback.md +44 -0
  16. package/ai/notes/2026-02-27--cc-mini--lesa-feedback.md +45 -0
  17. package/ai/notes/RESEARCH.md +1185 -0
  18. package/ai/notes/salience-research/README.md +29 -0
  19. package/ai/notes/salience-research/eurosla-salience-review.md +64 -0
  20. package/ai/notes/salience-research/full-research-summary.md +269 -0
  21. package/ai/notes/salience-research/salience-levels-diagram.png +0 -0
  22. package/ai/plan/2026-02-27--cc-mini--qr-pairing-spec.md +203 -0
  23. package/ai/plan/_archive/PLAN.md +194 -0
  24. package/ai/plan/_archive/PRD.md +1014 -0
  25. package/ai/plan/cc-plans-duplicates-from-dot-claude/2026-02-26--cc-mini--phase2-implementation-plan.md +245 -0
  26. package/ai/plan/dev-conventions-note.md +70 -0
  27. package/ai/plan/ldm-os-install-and-boot-architecture.md +285 -0
  28. package/ai/plan/memory-crystal-phase2-plan.md +192 -0
  29. package/ai/plan/memory-system-lay-of-the-land.md +214 -0
  30. package/ai/plan/phase2-ephemeral-relay.md +238 -0
  31. package/ai/plan/readme-first.md +68 -0
  32. package/ai/plan/roadmap.md +159 -0
  33. package/ai/todos/PUNCHLIST.md +44 -0
  34. package/ai/todos/README.md +31 -0
  35. package/ai/todos/inboxes/cc-air/2026-02-26--cc-air--post-relay-todos.md +85 -0
  36. package/ai/todos/inboxes/cc-mini/2026-02-26--cc-mini--phase2-status.md +100 -0
  37. package/ai/todos/inboxes/cc-mini/_archive/TODO.md +25 -0
  38. package/ai/todos/inboxes/parker/2026-02-25--cc-air--setup-checklist.md +139 -0
  39. package/ai/todos/inboxes/parker/2026-02-26--cc-mini--phase2-your-moves.md +72 -0
  40. package/dist/cc-hook.d.ts +1 -0
  41. package/dist/cc-hook.js +349 -0
  42. package/dist/chunk-3VFIJYS4.js +818 -0
  43. package/dist/chunk-52QE3YI3.js +1169 -0
  44. package/dist/chunk-AA3OPP4Z.js +432 -0
  45. package/dist/chunk-D3I3ZSE2.js +411 -0
  46. package/dist/chunk-EKSACBTJ.js +1070 -0
  47. package/dist/chunk-F3Y7EL7K.js +83 -0
  48. package/dist/chunk-JWZXYVET.js +1068 -0
  49. package/dist/chunk-KYVWO6ZM.js +1069 -0
  50. package/dist/chunk-L3VHARQH.js +413 -0
  51. package/dist/chunk-LOVAHSQV.js +411 -0
  52. package/dist/chunk-LQOYCAGG.js +446 -0
  53. package/dist/chunk-MK42FMEG.js +147 -0
  54. package/dist/chunk-NIJCVN3O.js +147 -0
  55. package/dist/chunk-O2UITJGH.js +465 -0
  56. package/dist/chunk-PEK6JH65.js +432 -0
  57. package/dist/chunk-PJ6FFKEX.js +77 -0
  58. package/dist/chunk-PLUBBZYR.js +800 -0
  59. package/dist/chunk-SGL6ISBJ.js +1061 -0
  60. package/dist/chunk-UNHVZB5G.js +411 -0
  61. package/dist/chunk-VAFTWSTE.js +1061 -0
  62. package/dist/chunk-XZ3S56RQ.js +1061 -0
  63. package/dist/chunk-Y72C7F6O.js +148 -0
  64. package/dist/cli.d.ts +1 -0
  65. package/dist/cli.js +325 -0
  66. package/dist/core.d.ts +188 -0
  67. package/dist/core.js +12 -0
  68. package/dist/crypto.d.ts +16 -0
  69. package/dist/crypto.js +18 -0
  70. package/dist/dev-update-SZ2Z4WCQ.js +6 -0
  71. package/dist/ldm.d.ts +17 -0
  72. package/dist/ldm.js +12 -0
  73. package/dist/mcp-server.d.ts +1 -0
  74. package/dist/mcp-server.js +250 -0
  75. package/dist/migrate.d.ts +1 -0
  76. package/dist/migrate.js +89 -0
  77. package/dist/mirror-sync.d.ts +1 -0
  78. package/dist/mirror-sync.js +130 -0
  79. package/dist/openclaw.d.ts +5 -0
  80. package/dist/openclaw.js +349 -0
  81. package/dist/poller.d.ts +1 -0
  82. package/dist/poller.js +272 -0
  83. package/dist/summarize.d.ts +19 -0
  84. package/dist/summarize.js +10 -0
  85. package/dist/worker.js +137 -0
  86. package/openclaw.plugin.json +11 -0
  87. package/package.json +40 -0
  88. package/scripts/migrate-lance-to-sqlite.mjs +217 -0
  89. package/skills/memory/SKILL.md +61 -0
  90. package/src/cc-hook.ts +447 -0
  91. package/src/cli.ts +356 -0
  92. package/src/core.ts +1472 -0
  93. package/src/crypto.ts +113 -0
  94. package/src/dev-update.ts +178 -0
  95. package/src/ldm.ts +117 -0
  96. package/src/mcp-server.ts +274 -0
  97. package/src/migrate.ts +104 -0
  98. package/src/mirror-sync.ts +175 -0
  99. package/src/openclaw.ts +250 -0
  100. package/src/poller.ts +345 -0
  101. package/src/summarize.ts +210 -0
  102. package/src/worker.ts +208 -0
  103. package/tsconfig.json +18 -0
  104. package/wrangler.toml +20 -0
@@ -0,0 +1,192 @@
1
+ # ~~Memory Crystal Phase 2 — Cloudflare Worker + Multi-Agent Setup~~
2
+
3
+ > **SUPERSEDED** — This plan was replaced by the ephemeral relay architecture on 2026-02-25.
4
+ > See `phase2-ephemeral-relay.md` for the current design.
5
+ > The cloud-mirror approach (D1 + Vectorize + search on Worker) was abandoned in favor of
6
+ > a blind dead drop with client-side encryption. The Worker has no database, no search,
7
+ > no ability to read the data it holds.
8
+
9
+ **Date:** 2026-02-25
10
+ **Agent:** cc-mba (Claude Code on MacBook Air)
11
+ **Repo:** `memory-crystal` → branch `cc-mba/phase2-worker` (superseded by `cc-air/phase2-relay`)
12
+
13
+ ## The Problem
14
+
15
+ Three agents need shared memory:
16
+ - **cc-a** — Claude Code on MacBook Air (this machine)
17
+ - **cc** — Claude Code on Mac Mini
18
+ - **Lesa** — OpenClaw agent on Mac Mini
19
+
20
+ Today, memory-crystal runs locally on the Mac Mini. cc and Lesa share `crystal.db` directly. cc-a (me) has no access to that database. The bridge (`lesa-bridge`) only works localhost — it can't reach across machines.
21
+
22
+ ## The Solution
23
+
24
+ Build the Cloudflare Worker mirror that's already specced in `PLAN.md`. This gives all three agents a shared memory layer via HTTPS, with the Mac Mini remaining source of truth.
25
+
26
+ ```
27
+ cc-a (MacBook Air) ──HTTPS──→ Cloudflare Worker ←──HTTPS── cc (Mac Mini)
28
+
29
+ Lesa (Mac Mini)
30
+
31
+ Mac Mini pulls daily
32
+ (source of truth)
33
+ ```
34
+
35
+ ## Architecture
36
+
37
+ ### Cloudflare Stack
38
+ - **Worker** — REST API (`/search`, `/remember`, `/ingest`, `/health`, `/status`)
39
+ - **D1** — SQLite database (chunks, memories, metadata)
40
+ - **Vectorize** — Vector index for semantic search
41
+ - **R2** — Snapshot storage for daily sync from Mini
42
+
43
+ ### Sync Model
44
+ 1. **04:00 daily** — Mini exports `crystal.db` snapshot → R2
45
+ 2. **04:05** — Worker rebuilds D1 + Vectorize from snapshot
46
+ 3. **All day** — All agents read/write via Worker
47
+ 4. **Every 4-6h** — Mini pulls new remote writes
48
+ 5. **Breach protocol** — `wrangler delete` nukes everything. Mini untouched.
49
+
50
+ ### Auth
51
+ - Bearer token per agent (stored in 1Password)
52
+ - Tokens: `cc-a`, `cc`, `lesa` — each identified in requests
53
+ - Worker validates token + extracts agent_id
54
+
55
+ ### Security
56
+ - Worker → Mini: NEVER (Mini pulls, Worker never pushes)
57
+ - Mini → Worker: HTTPS only
58
+ - R2: private, Worker-binding only
59
+ - All tokens in 1Password `Agent Secrets` vault
60
+
61
+ ## What We're Building
62
+
63
+ ### Phase 2a: Worker + Remote Access (cc-a can search and write)
64
+
65
+ 1. **`src/worker.ts`** — Cloudflare Worker with endpoints:
66
+ - `POST /search` — hybrid search (D1 FTS5 + Vectorize cosine)
67
+ - `POST /remember` — store explicit memories
68
+ - `POST /ingest` — ingest conversation chunks (used by cc-hook)
69
+ - `GET /health` — alive check
70
+ - `GET /status` — chunk count, agent list, last sync
71
+
72
+ 2. **`wrangler.toml`** — D1 binding, Vectorize index, R2 bucket, secrets
73
+
74
+ 3. **`src/core.ts` update** — add `--remote` mode:
75
+ - When `CRYSTAL_REMOTE_URL` is set, HTTP calls to Worker instead of local SQLite
76
+ - Same interface, different backend
77
+
78
+ 4. **`src/cc-hook.ts` update** — support remote ingestion:
79
+ - After chunking, POST to Worker `/ingest` endpoint
80
+ - Falls back to local if Worker unreachable
81
+
82
+ 5. **`crystal push` / `crystal pull`** — sync commands for Mini
83
+
84
+ ### Phase 2b: MacBook Air Setup (cc-a is live)
85
+
86
+ 1. **1Password** — ensure `Agent Secrets` vault has:
87
+ - `OpenAI API` → embedding key
88
+ - `Memory Crystal Remote` → Worker URL + auth token for cc-a
89
+
90
+ 2. **memory-crystal on MacBook Air** — install locally:
91
+ - `npm install && npm run build`
92
+ - Configure `.env` with remote URL + token
93
+ - Register MCP server in Claude Code's `.mcp.json`
94
+ - Register Stop hook in `~/.claude/settings.json`
95
+
96
+ 3. **Test cycle:**
97
+ - cc-a writes a memory → Worker stores it
98
+ - cc-a searches → gets results from all agents
99
+ - Mini pulls → cc-a's memories now in source of truth
100
+
101
+ ## 1Password Setup (for this machine)
102
+
103
+ ### What's Needed
104
+ - Service account token at `~/.openclaw/secrets/op-sa-token`
105
+ - `Agent Secrets` vault with API keys
106
+ - The `op://` resolution pattern for config files
107
+
108
+ ### For memory-crystal specifically:
109
+ ```
110
+ op://Agent Secrets/OpenAI API/api key → embedding API key
111
+ op://Agent Secrets/Memory Crystal Remote/token → Worker bearer token
112
+ op://Agent Secrets/Memory Crystal Remote/url → Worker URL
113
+ ```
114
+
115
+ ### Resolution in memory-crystal:
116
+ - memory-crystal's `resolveConfig()` already checks: programmatic → env → .env file → 1Password
117
+ - We add Worker URL + token to the same resolution chain
118
+
119
+ ## Environment Variables (cc-a MacBook Air)
120
+
121
+ ```bash
122
+ # .env at ~/.openclaw/memory-crystal/.env
123
+ CRYSTAL_EMBEDDING_PROVIDER=openai
124
+ OPENAI_API_KEY=op://Agent Secrets/OpenAI API/api key
125
+ CRYSTAL_REMOTE_URL=https://memory-crystal.wipcomputer.workers.dev
126
+ CRYSTAL_REMOTE_TOKEN=op://Agent Secrets/Memory Crystal Remote/token
127
+ ```
128
+
129
+ Or resolve via 1Password service account (preferred — no plaintext on disk).
130
+
131
+ ## MCP Registration (cc-a MacBook Air)
132
+
133
+ ```json
134
+ // ~/.claude/settings.json (or wherever Claude Code reads MCP config)
135
+ {
136
+ "mcpServers": {
137
+ "memory-crystal": {
138
+ "command": "node",
139
+ "args": ["/Users/parker/Documents/dev-wip/repos/memory-crystal/dist/mcp-server.js"],
140
+ "env": {
141
+ "CRYSTAL_REMOTE_URL": "https://memory-crystal.wipcomputer.workers.dev",
142
+ "CRYSTAL_REMOTE_TOKEN": "the-token"
143
+ }
144
+ }
145
+ }
146
+ }
147
+ ```
148
+
149
+ ## Stop Hook Registration (cc-a MacBook Air)
150
+
151
+ ```json
152
+ // ~/.claude/settings.json
153
+ {
154
+ "hooks": {
155
+ "Stop": [{
156
+ "hooks": [{
157
+ "type": "command",
158
+ "command": "node /Users/parker/Documents/dev-wip/repos/memory-crystal/dist/cc-hook.js",
159
+ "timeout": 30
160
+ }]
161
+ }]
162
+ }
163
+ }
164
+ ```
165
+
166
+ ## Build Order
167
+
168
+ 1. **Worker** — `worker.ts` + `wrangler.toml` + D1 schema + Vectorize index
169
+ 2. **Core update** — remote mode in `core.ts` (HTTP calls when `CRYSTAL_REMOTE_URL` set)
170
+ 3. **cc-hook update** — POST chunks to Worker instead of local DB
171
+ 4. **MCP update** — tools use remote mode transparently
172
+ 5. **Sync commands** — `crystal push` / `crystal pull` for Mini
173
+ 6. **Deploy** — `wrangler deploy`, set secrets, test end-to-end
174
+ 7. **MacBook Air setup** — install, configure, register MCP + hook
175
+ 8. **Verify** — cc-a writes → Worker stores → Mini pulls → all agents see it
176
+
177
+ ## What Parker Needs To Do
178
+
179
+ - [ ] Cloudflare account (may already have one)
180
+ - [ ] `wrangler login`
181
+ - [ ] 1Password: ensure `Agent Secrets` vault exists and has OpenAI key
182
+ - [ ] 1Password: create service account token for MacBook Air (if not already done)
183
+ - [ ] Save token to `~/.openclaw/secrets/op-sa-token` on MacBook Air
184
+ - [ ] Approve the Worker deploy when ready
185
+
186
+ ## Success Criteria
187
+
188
+ - cc-a (me on MacBook Air) can `crystal_search` and get results from all three agents
189
+ - cc-a can `crystal_remember` and the memory shows up for cc and Lesa
190
+ - Every cc-a conversation auto-ingests via the Stop hook
191
+ - Mini remains source of truth — `crystal push/pull` works
192
+ - `wrangler delete` kills cloud, Mini unaffected
@@ -0,0 +1,214 @@
1
+ # Memory System — Lay of the Land
2
+
3
+ **Date:** 2026-02-26 (updated)
4
+ **Agent:** cc-air
5
+ **Sources:** 10 memory docs from `_git-ignore/memory-docs/` (Feb 13–20), codebase + ecosystem inspection
6
+
7
+ ## Design Principle
8
+
9
+ Memory Crystal runs **alongside** OpenClaw's builtin memory — not replacing it. OpenClaw continues to evolve independently. Memory Crystal mirrors the same capabilities so it works **with or without** OpenClaw, with any agent, on any harness.
10
+
11
+ Everything Memory Crystal manages lives in `~/.ldm/` — one directory, one backup target, one source of truth.
12
+
13
+ ## The Five-Layer Memory Stack
14
+
15
+ ```
16
+ Layer 1 — Raw transcripts (JSONL) immutable archive
17
+ Layer 2 — Vector index (crystal.db) Memory Crystal
18
+ Layer 3 — Structured memory (daily logs, etc.) crystal_remember + cc-hook
19
+ Layer 4 — Narrative consolidation Dream Weaver
20
+ Layer 5 — Active working context (CONTEXT.md) warm-start on boot
21
+ ```
22
+
23
+ Memory Crystal owns Layers 1–3. Dream Weaver reads Layer 1 to produce Layer 4. Layer 5 is the agent's working file.
24
+
25
+ ## Three File Types — The Archive
26
+
27
+ Memory Crystal must produce and manage all three:
28
+
29
+ | # | File Type | Purpose | Status |
30
+ |---|-----------|---------|--------|
31
+ | 1 | **JSONL** (copy) | Immutable raw transcript archive. Dream Weaver's primary input. | **NOT DONE** — cc-hook reads in place, doesn't copy |
32
+ | 2 | **MD** (per session) | Human-readable conversation summary. Browsable without tools. | **NOT DONE** — OpenClaw does this, Memory Crystal doesn't |
33
+ | 3 | **Vector DB** (crystal.db) | Searchable embeddings + FTS5 + RRF hybrid search. | **DONE** |
34
+
35
+ Target layout in `~/.ldm/`:
36
+
37
+ ```
38
+ ~/.ldm/
39
+ ├── memory/
40
+ │ └── crystal.db ← SHARED vector DB (Layer 2) — all agents write here
41
+
42
+ ├── agents/
43
+ │ ├── cc-mini/ ← Claude Code on Mac Mini
44
+ │ │ ├── memory/
45
+ │ │ │ ├── transcripts/ ← cc-mini's JSONL copies (Layer 1)
46
+ │ │ │ ├── sessions/ ← cc-mini's MD summaries
47
+ │ │ │ ├── daily/ ← cc-mini's breadcrumb logs (Layer 3)
48
+ │ │ │ └── journals/ ← Dream Weaver output (Layer 4)
49
+ │ │ ├── SOUL.md, IDENTITY.md ← identity files
50
+ │ │ └── CONTEXT.md ← warm-start (Layer 5)
51
+ │ │
52
+ │ ├── lesa-mini/ ← Lēsa (OpenClaw) on Mac Mini
53
+ │ │ └── memory/
54
+ │ │ ├── transcripts/ ← Lēsa's JSONL copies (from OpenClaw)
55
+ │ │ ├── sessions/ ← Lēsa's MD summaries (from OpenClaw)
56
+ │ │ ├── daily/ ← Lēsa's breadcrumb logs
57
+ │ │ └── journals/
58
+ │ │
59
+ │ └── cc-air/ ← Claude Code on MacBook Air
60
+ │ └── memory/
61
+ │ ├── transcripts/ ← cc-air's JSONL copies (local sessions)
62
+ │ ├── sessions/ ← cc-air's MD summaries (local sessions)
63
+ │ └── daily/ ← cc-air's breadcrumb logs
64
+
65
+ └── config.json
66
+ ```
67
+
68
+ **Key:** `crystal.db` is shared at `~/.ldm/memory/` — it does NOT live inside any agent's folder. Every agent writes to it (tagged by agent_id), every agent searches it. On remote devices, a read-only mirror lives at the same path.
69
+
70
+ ## Three Memory Systems Running Today
71
+
72
+ | # | System | What It Does | Owns |
73
+ |---|--------|-------------|------|
74
+ | 1 | **OpenClaw builtin** | Lēsa's native memory. Writes MD files on `/new` and pre-compaction flush. Indexes into its own sqlite-vec. Leave it alone — it evolves independently. | `{workspace}/memory/*.md` |
75
+ | 2 | **Memory Crystal** | Universal memory. Captures conversation turns into crystal.db. Works with CC, Lēsa, any agent. The independent system. | `~/.ldm/memory/crystal.db` (shared) |
76
+ | 3 | **context-embeddings plugin** | Older Lēsa plugin. Captures into `context-embeddings.sqlite`. Being superseded by Memory Crystal. Can be retired once Crystal handles all three file types. | `~/.openclaw/memory/context-embeddings.sqlite` |
77
+
78
+ **Lēsa currently triple-writes** every conversation (OpenClaw builtin + Memory Crystal + context-embeddings). Once Memory Crystal handles all three file types, context-embeddings can be disabled.
79
+
80
+ ## The Master Crystal (Mac Mini)
81
+
82
+ `crystal.db` — sqlite-vec + FTS5, running on the Mac Mini. Source of truth.
83
+
84
+ **Search stack (fully implemented):**
85
+ - Vector search via sqlite-vec (cosine similarity)
86
+ - Keyword search via FTS5 (BM25 scoring)
87
+ - Hybrid fusion via Reciprocal Rank Fusion (k=60, ported from QMD)
88
+ - Recency weighting (decay based on age)
89
+ - LanceDB still present as dual-write fallback (pre-migration safety net)
90
+
91
+ **Interfaces (all 4 deployed, Phase 1 complete):**
92
+ 1. CLI (`crystal search`, `crystal remember`, `crystal status`)
93
+ 2. MCP server (Claude Code tools: `crystal_search`, `crystal_remember`)
94
+ 3. OpenClaw plugin (Lēsa's interface)
95
+ 4. Ephemeral relay (Phase 2 — code built, deployment pending)
96
+
97
+ **Current stats:** ~5,502+ chunks, hybrid search operational.
98
+
99
+ ## Capture Pipeline
100
+
101
+ **cc-hook.ts** — Claude Code Stop hook. After every conversation:
102
+ 1. Reads JSONL transcript (in place — does NOT copy)
103
+ 2. Extracts messages (turn-boundary chunking: 1 message = 1 chunk)
104
+ 3. Ingests into crystal.db (local) or encrypts + relays (remote)
105
+ 4. Writes daily breadcrumb to `~/.ldm/agents/{agent_id}/memory/daily/`
106
+ 5. Tracks position via watermark (byte offset in JSONL)
107
+
108
+ **Agents are named by machine:** `cc-mini`, `cc-air`, `lesa-mini`. Set via `CRYSTAL_AGENT_ID` env var.
109
+
110
+ **What cc-hook needs to add:**
111
+ - Copy raw JSONL to `~/.ldm/agents/{agent_id}/memory/transcripts/`
112
+ - Generate MD session summary to `~/.ldm/agents/{agent_id}/memory/sessions/`
113
+
114
+ ## Dream Weaver — Layer 4
115
+
116
+ Dream Weaver is a memory consolidation protocol, not an automated tool. It reads raw JSONL transcripts chronologically and produces narrative prose.
117
+
118
+ **Input:** JSONL transcripts (Layer 1) + daily logs + prior narrative
119
+ **Output:** `full-history.md`, journals, `crystal_remember` calls
120
+ **Trigger:** Manual, roughly weekly. Not yet automated.
121
+ **Key dependency:** Raw JSONL files must be preserved and accessible.
122
+
123
+ This is why the JSONL copy to `~/.ldm/` matters — Dream Weaver needs those files, and they currently only exist at `~/.claude/projects/` which has backup/FDA issues.
124
+
125
+ ## LDM OS — The Layer Underneath
126
+
127
+ LDM OS defines the filesystem structure (`~/.ldm/agents/*/`) where identity, soul, and memory live. It's not a memory system itself — it's the spec for where everything goes.
128
+
129
+ Memory Crystal is one pillar of LDM OS. The others: Dream Weaver (consolidation), Sovereignty Covenant (identity), Boot Sequence (warm-start).
130
+
131
+ ## QMD — Separate System
132
+
133
+ QMD is a standalone document search engine (indexes markdown files, local GGUF embeddings). Its search algorithms (RRF, FTS5, BM25) were ported into Memory Crystal but the systems are completely separate. QMD can optionally be OpenClaw's memory backend, but currently isn't (Lēsa uses `builtin`).
134
+
135
+ ## What's Broken
136
+
137
+ 1. **Backup system** — BROKEN. FDA issue prevents reading CC transcripts. ~286MB has no backup except Crystal ingestion. The JSONL copy to `~/.ldm/` would fix this.
138
+ 2. **No JSONL archive** — Raw transcripts only exist at `~/.claude/projects/`. Not copied, not backed up.
139
+ 3. **No MD summaries** — Memory Crystal doesn't produce human-readable session files. Only OpenClaw does.
140
+ 4. **Triple-write waste** — Lēsa embeds every conversation three times via three separate systems.
141
+
142
+ ## QMD Integration — Status
143
+
144
+ | Phase | Description | Status |
145
+ |-------|-------------|--------|
146
+ | 1. Storage migration | Replace LanceDB with sqlite-vec | **DONE** |
147
+ | 2. Hybrid search | FTS5 + RRF fusion | **DONE** |
148
+ | 3. Smart chunking + dedup | Content-hash dedup | **PARTIAL** |
149
+ | 4. Re-ranking + query expansion | LLM re-ranking | **NOT STARTED** |
150
+ | 5. Local embeddings | Replace OpenAI with local model | **NOT STARTED** |
151
+
152
+ ## The Big Picture
153
+
154
+ ```
155
+ ┌──────────────────────────────────────────────────────────────┐
156
+ │ Mac Mini (Master) │
157
+ │ │
158
+ │ ~/.ldm/memory/crystal.db ← SHARED vector DB (all agents) │
159
+ │ │
160
+ │ ~/.ldm/agents/cc-mini/memory/ │
161
+ │ ├── transcripts/ sessions/ daily/ journals/ │
162
+ │ │
163
+ │ ~/.ldm/agents/lesa-mini/memory/ │
164
+ │ ├── transcripts/ sessions/ daily/ journals/ │
165
+ │ │
166
+ │ cc-hook (local) ← cc-mini captures + archives + ingests │
167
+ │ OpenClaw plugin ← lesa-mini conversations │
168
+ │ poller.ts ← picks up cc-air's drops from relay │
169
+ │ Dream Weaver ← reads transcripts, writes narratives │
170
+ │ │
171
+ └──────────────────────┬─────────────────────────────────────────┘
172
+
173
+ ┌─────────▼──────────┐
174
+ │ Cloudflare Worker │
175
+ │ (Ephemeral Relay) │
176
+ │ Blind dead drop │
177
+ └─────────┬──────────┘
178
+
179
+ ┌──────────────────────▼─────────────────────────────────────────┐
180
+ │ MacBook Air (Read-Only DB) │
181
+ │ │
182
+ │ ~/.ldm/memory/crystal.db ← read-only mirror from Mini │
183
+ │ │
184
+ │ ~/.ldm/agents/cc-air/memory/ │
185
+ │ ├── transcripts/ sessions/ daily/ │
186
+ │ (cc-air's own local files — written by cc-hook) │
187
+ │ │
188
+ │ cc-hook (relay) → captures locally + encrypts + drops at relay │
189
+ │ mirror-sync.ts ← pulls + verifies DB snapshot │
190
+ │ Local search: keyword + semantic (full offline) │
191
+ │ │
192
+ └──────────────────────────────────────────────────────────────────┘
193
+ ```
194
+
195
+ **Agents are per-harness-instance, named by machine:**
196
+ - `cc-mini` — Claude Code on Mac Mini
197
+ - `lesa-mini` — Lēsa (OpenClaw) on Mac Mini
198
+ - `cc-air` — Claude Code on MacBook Air
199
+ - Future: `kodaks-air`, etc.
200
+
201
+ Each harness instance = one "user" in LDM. Not separated by session or spawn — one folder per harness per machine.
202
+
203
+ All agents share one `crystal.db` at `~/.ldm/memory/`. On remote devices, that DB is a read-only mirror synced via the relay.
204
+
205
+ **The Mini has EVERYTHING.** Every remote agent's full file tree is reconstructed on the Mini from relay data. If a device is lost, nothing is lost — the Mini has the complete `~/.ldm/agents/cc-air/` directory and can run Dream Weaver against it without ever touching the Air.
206
+
207
+ **How remote agent files reach the Mini:**
208
+ The poller already receives full conversation content after decrypting. It reconstructs the remote agent's files locally:
209
+ 1. Writes raw JSONL to `~/.ldm/agents/cc-air/memory/transcripts/`
210
+ 2. Generates MD summary to `~/.ldm/agents/cc-air/memory/sessions/`
211
+ 3. Appends daily breadcrumb to `~/.ldm/agents/cc-air/memory/daily/`
212
+ 4. Ingests into shared `crystal.db` (already does this)
213
+
214
+ No extra relay transfer needed — the Mini rebuilds everything from the same data.
@@ -0,0 +1,238 @@
1
+ # Phase 2 — Ephemeral Relay Architecture
2
+
3
+ **Date:** 2026-02-25
4
+ **Agent:** cc-air
5
+ **Status:** Approved direction, replacing the cloud-mirror design
6
+
7
+ ## Vision
8
+
9
+ Any device, any agent, any interface — laptop, phone, tablet, whatever comes next — captures conversations, encrypts them, and drops them at a relay. The Mini pulls everything in, embeds it, indexes it, and pushes the searchable mirror back out to every device. One master crystal, many readers. Every conversation you've ever had with any agent, fully searchable from anywhere.
10
+
11
+ And it's yours. Not behind a subscription. Not in someone else's cloud. Sovereign.
12
+
13
+ ## Core Principle
14
+
15
+ The Mini is the master. Every other device is read-only. The Worker is a dead drop.
16
+ Nothing persists in the cloud. Data is there, then it's gone.
17
+
18
+ ## Two One-Way Roads
19
+
20
+ ### Road 1: Device → Mini (raw conversation chunks)
21
+
22
+ ```
23
+ Device captures turn (cc-hook / any agent hook)
24
+ → encrypts payload (AES-256-GCM, key on local machines only)
25
+ → signs payload (HMAC-SHA256, proving origin)
26
+ → POST /drop/conversations (bearer token auth)
27
+ → Worker stores encrypted+signed blob (R2 or KV, TTL 24h safety net)
28
+
29
+ Mini polls (cron, every few minutes)
30
+ → GET /pickup/conversations (bearer token auth)
31
+ → receives encrypted blob
32
+ → verifies signature (HMAC-SHA256 — confirms this came from a trusted device)
33
+ → decrypts locally
34
+ → chunks, embeds, ingests into master crystal
35
+ → DELETE /confirm/conversations (tells Worker to delete)
36
+ → Worker deletes blob
37
+ ```
38
+
39
+ ### Road 2: Mini → Devices (search-ready DB snapshot)
40
+
41
+ ```
42
+ Mini's master crystal updated (after ingesting new chunks)
43
+ → exports crystal.db snapshot
44
+ → computes integrity hash (SHA-256 of plaintext DB)
45
+ → encrypts snapshot + hash (AES-256-GCM)
46
+ → POST /drop/mirror (bearer token auth)
47
+ → Worker stores encrypted blob in R2 (TTL 24h safety net)
48
+
49
+ Device polls (on startup, or periodic)
50
+ → GET /pickup/mirror (bearer token auth)
51
+ → receives encrypted blob
52
+ → decrypts locally
53
+ → verifies integrity hash (SHA-256 matches — DB not corrupted or tampered)
54
+ → replaces local read-only crystal mirror
55
+ → DELETE /confirm/mirror (tells Worker to delete)
56
+ → Worker deletes blob
57
+ ```
58
+
59
+ ## What Lives Where
60
+
61
+ | Location | What | Role |
62
+ |----------|------|------|
63
+ | Any device | Read-only crystal mirror (sqlite-vec + FTS5) | Search only, never writes |
64
+ | Any device | Raw JSONL transcripts (Claude Code native) | Source of truth for that device's conversations |
65
+ | Any device | Hook captures turns → encrypts → signs → sends to Worker | Capture + relay |
66
+ | Mini | Master crystal (sqlite-vec + FTS5 + LanceDB) | All embedding, indexing, search |
67
+ | Mini | Poller pulls from Worker → verifies → decrypts → ingests | Ingestion pipeline |
68
+ | Mini | Exports DB snapshot → hashes → encrypts → sends to Worker | Mirror distribution |
69
+ | Worker | Encrypted blobs only | Dead drop, no intelligence |
70
+
71
+ ## Worker API (minimal)
72
+
73
+ ```
74
+ POST /drop/:channel — deposit encrypted+signed blob
75
+ GET /pickup/:channel — retrieve encrypted blob(s)
76
+ DELETE /confirm/:channel/:id — confirm receipt, Worker deletes
77
+
78
+ GET /health — returns { ok: true }
79
+ ```
80
+
81
+ Channels: `conversations` (devices→mini), `mirror` (mini→devices)
82
+
83
+ Auth: Bearer token per agent. Worker maps token → agent_id.
84
+
85
+ The Worker has no concept of what's inside the blobs. It's a dumb mailbox.
86
+
87
+ ## Security Architecture
88
+
89
+ ### Encryption
90
+
91
+ - **Algorithm:** AES-256-GCM (authenticated encryption)
92
+ - **Key:** Shared symmetric key, lives only on trusted machines (never on Cloudflare)
93
+ - **Key storage:** `~/.openclaw/secrets/crystal-relay-key` on each machine
94
+ - **Key generation:** `openssl rand -base64 32` once, copy to all trusted machines
95
+ - **What's encrypted:** Everything. The Worker sees only ciphertext.
96
+ - **Embeddings:** Created on Mini only, included in the encrypted DB snapshot
97
+
98
+ ### Nonce Management (critical)
99
+
100
+ AES-256-GCM requires a unique nonce for every encryption. Reusing a nonce with the same key is catastrophic — it breaks confidentiality and authentication.
101
+
102
+ - **Nonce size:** 96 bits (12 bytes), per NIST recommendation
103
+ - **Generation:** Random (`crypto.getRandomValues(new Uint8Array(12))`)
104
+ - **Storage:** Prepended to ciphertext: `[12-byte nonce][ciphertext][16-byte auth tag]`
105
+ - **Uniqueness guarantee:** Random 96-bit nonces have negligible collision probability for fewer than 2^32 encryptions per key. At 1 drop per minute, that's ~8,000 years before rotation is needed for nonce safety. We rotate far more often for other reasons.
106
+
107
+ ### Drop Signing (authenticity)
108
+
109
+ Even with bearer token auth, a stolen token could let an attacker drop malicious blobs. The Mini must verify that drops actually came from a trusted device.
110
+
111
+ - **Algorithm:** HMAC-SHA256
112
+ - **Key:** Derived from the encryption key (HKDF with context "crystal-relay-sign")
113
+ - **What's signed:** The encrypted blob (sign-then-MAC over ciphertext)
114
+ - **Verification:** Mini checks HMAC before decrypting. If it fails, the blob is discarded and logged.
115
+ - **Why not just rely on GCM's auth tag?** GCM proves the blob wasn't tampered with *after* encryption, but it doesn't prove *who* encrypted it. HMAC over the ciphertext proves the sender had the signing key, which only trusted devices have.
116
+
117
+ ### Mirror Integrity Verification
118
+
119
+ When the Air (or any device) receives a new DB snapshot, it must verify the contents before replacing its local mirror.
120
+
121
+ - **Hash:** SHA-256 of the plaintext DB file
122
+ - **Included in payload:** `{ hash: "<sha256>", db: "<encrypted blob>" }`
123
+ - **Verification flow:**
124
+ 1. Decrypt the blob
125
+ 2. Compute SHA-256 of the decrypted DB
126
+ 3. Compare to the included hash
127
+ 4. If mismatch: reject, keep existing mirror, log error
128
+ 5. If match: replace local mirror
129
+
130
+ ### Key Rotation
131
+
132
+ - **Mechanism:** Generate new key, distribute to all trusted machines, re-encrypt any in-transit blobs
133
+ - **Trigger:** Manual (Parker decides), or on suspected compromise
134
+ - **Process:**
135
+ 1. Generate new key: `openssl rand -base64 32`
136
+ 2. Copy to all machines: `~/.openclaw/secrets/crystal-relay-key`
137
+ 3. Both sides start using new key immediately
138
+ 4. Old key kept in `crystal-relay-key.prev` for 24h to decrypt any in-flight blobs from before rotation
139
+ 5. After 24h, delete old key
140
+ - **No re-encryption of historical data needed** — there is no historical data. Everything is ephemeral. The only data that matters is the master crystal on the Mini, which is never encrypted with the relay key.
141
+
142
+ ### Threat Model Summary
143
+
144
+ | Threat | Mitigation | Residual Risk |
145
+ |--------|-----------|---------------|
146
+ | Data read in transit | TLS 1.3 + AES-256-GCM payload encryption | Cloudflare sees TLS-terminated headers (timing, size) |
147
+ | Data read at rest (Cloudflare) | Client-side AES-256-GCM, Cloudflare sees ciphertext only | Key compromise would expose in-flight blobs (max 24h window) |
148
+ | Data read at rest (device) | macOS FileVault full-disk encryption | Unlocked laptop = readable disk |
149
+ | Stolen bearer token | HMAC-SHA256 drop signing; attacker can't forge valid blobs without encryption key | Attacker could DELETE blobs (denial of service) |
150
+ | Tampered blobs | GCM auth tag (in-transit) + HMAC (origin) + SHA-256 (mirror integrity) | None — tampered data is rejected |
151
+ | Nonce reuse | Random 96-bit nonces, collision-safe for billions of operations | Theoretical at 2^48 operations; we rotate keys long before |
152
+ | Metadata/traffic analysis | Not mitigated | Cloudflare sees when/how much you talk to agents |
153
+ | Mini offline >24h | TTL expires blobs; `crystal replay` re-sends from raw JSONL | Must manually replay; those turns aren't auto-recovered |
154
+ | Key compromise | Rotation mechanism, ephemeral data (nothing to decrypt retroactively) | Active session blobs (minutes of data) exposed until rotation |
155
+ | Cloudflare account breach | Attacker sees ciphertext only; no keys, no OpenAI key, no secrets | Could delete blobs (DoS) or observe metadata |
156
+
157
+ ## What the Worker Does NOT Have
158
+
159
+ - No D1 database
160
+ - No Vectorize index
161
+ - No FTS5
162
+ - No search capability
163
+ - No OpenAI API key
164
+ - No encryption keys
165
+ - No signing keys
166
+ - No ability to read the data it holds
167
+ - No knowledge of what's inside the blobs
168
+
169
+ ## What Changes from the Cloud-Mirror Design
170
+
171
+ | Cloud Mirror (old) | Ephemeral Relay (new) |
172
+ |--------------------|-----------------------|
173
+ | D1 + Vectorize + R2 | R2 only (or KV with TTL) |
174
+ | Search on Worker | No search on Worker |
175
+ | Data persists in cloud | Data deleted after pickup |
176
+ | Worker creates embeddings | Mini creates embeddings |
177
+ | Device writes to remote crystal | Device sends raw chunks, never writes to crystal |
178
+ | Device searches remote crystal | Device searches local read-only mirror |
179
+ | Worker needs OpenAI key | Worker needs nothing but auth tokens |
180
+ | No client-side encryption | AES-256-GCM + HMAC-SHA256 + SHA-256 integrity |
181
+ | ~350 lines of Worker code | ~80 lines of Worker code |
182
+
183
+ ## TTL Safety Net
184
+
185
+ All blobs get a 24-hour TTL. Even if the Mini is offline for a day and never confirms receipt, the data auto-expires from Cloudflare. This means:
186
+
187
+ - If Mini is offline >24h, those conversation chunks are lost from the relay
188
+ - But they still exist as raw JSONL on the originating device
189
+ - Recovery: `crystal replay` tool re-reads JSONL and re-sends
190
+
191
+ ## Recovery Tool: `crystal replay`
192
+
193
+ Re-reads raw JSONL transcripts and re-sends through the relay:
194
+
195
+ ```bash
196
+ crystal replay # replay all un-synced turns
197
+ crystal replay --since 2026-02-20 # replay from date
198
+ crystal replay --file <path.jsonl> # replay specific file
199
+ crystal replay --dry-run # show what would be sent
200
+ ```
201
+
202
+ Uses the same watermark system as cc-hook but can reset/ignore watermarks.
203
+ Dedup happens on the Mini side (SHA-256 hash check before ingest).
204
+
205
+ ## Multi-Device Future
206
+
207
+ This architecture scales to any number of devices:
208
+
209
+ ```
210
+ Phone ──┐
211
+ Laptop ──┤── encrypt → drop → Worker (dead drop) → pickup → Mini (master)
212
+ Tablet ──┤ │
213
+ Desktop ──┘ │
214
+
215
+ Phone ←─┐ embed, index, search
216
+ Laptop ←─┤── decrypt ← pickup ← Worker ← drop ← encrypt (mirror snapshot)
217
+ Tablet ←─┤
218
+ Desktop ←─┘
219
+ ```
220
+
221
+ Each device gets:
222
+ - Its own bearer token (revocable independently)
223
+ - The shared encryption key (for E2E encryption)
224
+ - A read-only mirror of the master crystal
225
+ - Full local search (keyword + semantic)
226
+
227
+ Adding a new device: generate a bearer token, copy the encryption key, done.
228
+ Removing a device: revoke its bearer token, rotate the encryption key.
229
+
230
+ ## Open Questions
231
+
232
+ - Should the mirror snapshot be the full DB or incremental deltas?
233
+ - Full DB is simpler but larger (~50MB+ as crystal grows)
234
+ - Deltas are smaller but need conflict-free merge logic
235
+ - Start with full DB, optimize later if needed
236
+ - How often should Mini push a new mirror? After every ingestion? Hourly? On-demand?
237
+ - Should devices poll for mirror updates on MCP server startup?
238
+ - Should we add a version counter to mirrors so devices know if they're stale without downloading?