memory-crystal 0.2.0

package/RELAY.md

@@ -0,0 +1,88 @@
+###### WIP Computer
+
+# Relay: Multi-Device Sync
+
+Memory Crystal works on one machine out of the box. Multi-device sync lets your agent's memory follow you across machines. Conversations captured on your laptop are available on your desktop and vice versa.
+
+Everything is encrypted before it leaves your machine. The relay never sees your data unencrypted.
+
+## Two Options
+
+### Use Our Relay (Default)
+
+We host the relay infrastructure. You just set an encryption key.
+
+```
+Open your AI and say:
+
+I want to set up multi-device sync for Memory Crystal.
+Walk me through the setup step by step.
+```
+
+Your agent generates your encryption key, configures the connection, and tests it. Takes about two minutes.
+
+**What you need:**
+- Memory Crystal installed on both machines
+- An encryption key (your agent generates this)
+
+**Pricing:** Free during beta. When pricing is introduced, your agent will handle it via [AI CASH](https://github.com/wipcomputer/wip-agent-pay/blob/main/CASH.md).
+
+### Self-Host Your Own Relay
+
+Run your own relay on Cloudflare Workers (free tier). Same code, your infrastructure. Full control.
+
+**What you need:**
+- A Cloudflare account (free tier works)
+- About five minutes
+
+**Steps:**
+1. Deploy the Worker from `worker/index.js` to Cloudflare
+2. Create a KV namespace called `PAY_TOKENS`, bind it as `KV`
+3. Set a `WORKER_SECRET` via `wrangler secret put WORKER_SECRET`
+4. Point Memory Crystal at your Worker:
+
+```bash
+crystal relay --self-hosted --worker-url https://your-relay.your-domain.com
+```
+
+Full deployment details in [Technical Documentation](https://github.com/wipcomputer/memory-crystal/blob/main/TECHNICAL.md).
+
+No fees. No dependencies on us. The relay code is open source (MIT).
+
+## How Sync Works
+
+1. You work with your agent on Machine A
+2. After each session, Memory Crystal encrypts the conversation (AES-256-GCM) and drops it at the relay
+3. Machine B polls the relay, downloads the encrypted blob, decrypts it locally, and ingests it into its crystal.db
+4. The relay deletes the blob after pickup
+
+Two one-way roads:
+- **Device to home machine** ... encrypted conversation chunks
+- **Home machine to devices** ... search-ready DB snapshot via mirror-sync
+
+The relay is a dead drop. It stores encrypted blobs temporarily and serves them on request. It has no decryption capability. If someone compromises the relay, they get encrypted noise.
+
+## Encryption
+
+- **AES-256-GCM** for encryption (authenticated encryption, no padding oracle attacks)
+- **HMAC-SHA256** for signing (integrity verification before decryption)
+- Shared key generated locally, never transmitted to the relay
+- Key must be present on both machines (store in 1Password, AirDrop between Macs, or transfer manually)
+
+## More Info
+
+- [README.md](https://github.com/wipcomputer/memory-crystal/blob/main/README.md) ... What Memory Crystal is and how to install it.
+- [Technical Documentation](https://github.com/wipcomputer/memory-crystal/blob/main/TECHNICAL.md) ... Full technical documentation.
+
+---
+
+## License
+
+```
+src/, skills/, cli.ts, mcp-server.ts   MIT    (use anywhere, no restrictions)
+worker/                                AGPL   (relay server)
+```
+
+AGPL for personal use is free.
+
+Built by Parker Todd Brooks, Lēsa (OpenClaw, Claude Opus 4.6), Claude Code CLI (Claude Opus 4.6).

package/TECHNICAL.md

@@ -0,0 +1,379 @@
+###### WIP Computer
+
+# Technical Documentation
+
+How Memory Crystal works. Architecture, design decisions, integrations, encryption, search, and everything else the open source community is going to ask about.
+
+## How Does It Work?
+
+Memory Crystal captures every conversation you have with any AI tool, embeds it into a local SQLite database, and makes it searchable with hybrid search (keyword + semantic). One database file. Runs on your machine. Nothing leaves your device unless you set up multi-device sync.
+
+Every conversation produces three artifacts:
+1. **JSONL transcript** ... the raw session, archived to disk
+2. **Markdown summary** ... title, summary, key topics (generated by LLM or simple extraction)
+3. **Vector embeddings** ... chunked, embedded, and stored in crystal.db for search
+
+## How Does It Work with Claude?
+
+Memory Crystal runs as a [Claude Code Stop hook](https://docs.anthropic.com/en/docs/claude-code). After every Claude Code response, the hook fires automatically:
+
+1. Reads Claude Code's JSONL transcript file via byte-offset watermarking (only reads new data since last capture)
+2. Extracts user, assistant, and thinking blocks
+3. Chunks them, embeds into sqlite-vec
+4. Archives the full JSONL transcript
+5. Generates a markdown session summary
+6. Appends a daily breadcrumb log
+
+**Configuration:**
+```json
+{
+  "hooks": {
+    "Stop": [{ "hooks": [{ "type": "command", "command": "node ~/.openclaw/extensions/memory-crystal/dist/cc-hook.js", "timeout": 30 }] }]
+  }
+}
+```
+
+```bash
+node dist/cc-hook.js --on       # Enable capture
+node dist/cc-hook.js --off      # Pause capture
+node dist/cc-hook.js --status   # Check status
+```
+
+Respects private mode. When capture is off, nothing is recorded. First run seeds the watermark at current file size so it doesn't replay your entire history.
+
+## How Does It Work with OpenClaw?
+
+Memory Crystal is an OpenClaw plugin. It registers tools (`crystal_search`, `crystal_remember`, `crystal_forget`, `crystal_status`) and an `agent_end` hook that captures conversations after every agent turn.
+
+Deployed to `~/.openclaw/extensions/memory-crystal/`. The plugin uses the same `core.ts` as every other interface. Same search, same database, same embeddings.
+
+## How Does It Work with ChatGPT, Codex, and Other Tools?
+
+Any tool that can run shell commands or call an MCP server can use Memory Crystal.
+
+- **MCP Server** ... `mcp-server.ts` exposes `crystal_search`, `crystal_remember`, `crystal_forget`, `crystal_status`, `crystal_sources_add`, `crystal_sources_sync`, `crystal_sources_status`. Works with Claude Desktop, Claude Code, or any MCP-compatible client.
+- **CLI** ... `crystal search "query"` from any terminal. Any tool with shell access can call it.
+- **Module** ... `import { MemoryCrystal } from 'memory-crystal'` for Node.js integration.
+
+For tools that don't have hook support (like ChatGPT), you can manually export conversations and import them. Automatic capture requires hook support (Claude Code, OpenClaw).
+
+## Architecture
+
+One core, five interfaces.
+
+```
+sqlite-vec (vectors) + FTS5 (BM25) + SQLite (metadata/graph)
+         |                 |                    |
+    core.ts ... pure logic, zero framework deps
+    |-- cli.ts          -> crystal search "query"
+    |-- mcp-server.ts   -> crystal_search (Claude Code, Claude Desktop)
+    |-- openclaw.ts     -> plugin (OpenClaw agents)
+    |-- cc-hook.ts      -> Claude Code Stop hook (auto-capture)
+    +-- worker.ts       -> Cloudflare Worker (encrypted relay)
+```
+
+Every interface calls the same `core.ts`. The core has zero framework dependencies. It talks to SQLite and nothing else.
+
+## Search: How Does It Work?
+
+Hybrid search. Two engines run in parallel and their results are fused.
+
+### The Pipeline
+
+1. Query goes to both FTS5 (keyword match) and sqlite-vec (vector similarity)
+2. FTS5 returns BM25-ranked results, normalized to [0..1) via `|score| / (1 + |score|)`
+3. sqlite-vec returns cosine-distance results via two-step query (MATCH first, then JOIN separately ... sqlite-vec hangs with JOINs in the same query)
+4. Reciprocal Rank Fusion merges both lists: `weight / (k + rank + 1)` with k=60, plus top-rank bonus
+5. Recency weighting applied on top: `max(0.5, 1.0 - age_days * 0.01)`
+6. Final results sorted by combined score
+
+Inspired by and partially ported from [QMD](https://github.com/tobi/qmd) by Tobi Lutke (MIT, 2024-2026).
+
+### Why Hybrid?
+
+Vector search alone misses exact matches. Keyword search alone misses semantic similarity. Hybrid catches both. A search for "deployment process" will find conversations that use the word "deployment" (BM25) and conversations about "shipping code to production" (vector similarity).
+
+### Recency Decay
+
+Linear decay from 1.0 to 0.5 over 50 days. Fresh context wins ties. Old stuff still surfaces for strong matches.
+
+Freshness flags: fresh (<3 days), recent (<7 days), aging (<14 days), stale (14+ days).
+
+### Content Dedup
+
+SHA-256 hash of chunk text before embedding. Duplicate content is never re-embedded. This matters when the same conversation is captured by multiple hooks (e.g., Claude Code hook and OpenClaw plugin running simultaneously).
+
+## Database
+
+Everything lives in one file: `crystal.db`. Inspectable with any SQLite tool. Backupable with `cp`.
+
+### Schema
+
+| Table | Purpose |
+|-------|---------|
+| `chunks` | Chunk text, metadata, SHA-256 hash, timestamps |
+| `chunks_vec` | sqlite-vec virtual table (cosine distance vectors) |
+| `chunks_fts` | FTS5 virtual table (Porter stemming, BM25 scoring) |
+| `memories` | Explicit remember/forget facts |
+| `entities` | Knowledge graph nodes |
+| `relationships` | Knowledge graph edges |
+| `capture_state` | Watermarks for incremental ingestion |
+| `sources` | Ingestion source metadata |
+| `source_collections` | Directory collections for file indexing |
+| `source_files` | Indexed file records with content hashes |
+
+### Why SQLite?
+
+One file. No server. No Docker. No connection strings. Works on every platform. Inspectable with standard tools. Backupable with `cp`. Ships with every OS.
+
+sqlite-vec adds vector search as a virtual table. FTS5 adds full-text search. Both are SQLite extensions that work within the same database file.
+
+### DB Location
+
+`resolveConfig()` in `core.ts` checks in order:
+
+1. Explicit override (programmatic)
+2. `CRYSTAL_DATA_DIR` env var
+3. `~/.ldm/memory/crystal.db` (if it exists)
+4. `~/.openclaw/memory-crystal/` (legacy fallback)
+
+## Directory Structure
+
+Memory Crystal manages `~/.ldm/` ... the universal agent home directory for [LDM OS](https://github.com/wipcomputer/dream-weaver-protocol).
+
+```
+~/.ldm/
+  config.json                              version, registered agents array
+  memory/
+    crystal.db                             shared vector DB (all agents)
+  agents/{agent_id}/
+    memory/
+      transcripts/                         full JSONL session transcripts
+      sessions/                            markdown session summaries
+      daily/                               daily breadcrumb logs
+      journals/                            agent journals
+```
+
+Path resolution is centralized in `src/ldm.ts`:
+- `getAgentId()` ... resolves from `CRYSTAL_AGENT_ID` env var, default `cc-mini`
+- `ldmPaths(agentId?)` ... returns all paths as an object
+- `scaffoldLdm(agentId?)` ... creates the full directory tree
+- `ensureLdm(agentId?)` ... idempotent check, scaffolds if needed
+
+## Encryption: How Does It Work?
+
+For multi-device sync. All encryption happens on-device before anything touches the network.
+
+- **AES-256-GCM** for encryption. Authenticated encryption ... ciphertext tampering is detected.
+- **HMAC-SHA256** for signing. Integrity verification before decryption. If the signature doesn't match, the blob is rejected.
+- **Shared symmetric key** generated locally with `openssl rand -hex 32`. Never transmitted to the relay.
+- The relay stores and serves encrypted blobs. It has no decryption capability. Compromising the relay yields encrypted noise.
+
+### Key Management
+
+The same encryption key must be present on all devices. Options:
+- **1Password** ... store the key, both machines pull from 1Password via SA token
+- **AirDrop** ... direct transfer between Macs
+- **Manual** ... copy the key securely between machines
+
+### Relay Architecture
+
+1. **Device side** (`cc-hook.ts` relay mode): Encrypts JSONL with AES-256-GCM, signs with HMAC-SHA256, drops at Cloudflare Worker
+2. **Worker** (`worker.ts`): Stores encrypted blobs in KV. Pure dead drop. No decryption.
+3. **Home machine** (`poller.ts`): Polls Worker, downloads blobs, decrypts, ingests into crystal.db. Reconstructs remote agent's file tree (JSONL, MD summary, daily breadcrumb).
+4. **Mirror sync** (`mirror-sync.ts`): Pulls crystal.db snapshot from home machine to devices for local search.
+
+Two one-way roads:
+- **Device -> Home machine** ... encrypted conversation chunks (ephemeral)
+- **Home machine -> Devices** ... search-ready DB snapshot
+
+## Session Summaries
+
+`src/summarize.ts` generates markdown summaries. Two modes:
+
+**LLM mode** (default): Calls gpt-4o-mini with a condensed transcript. Returns title, slug, summary, key topics.
+
+**Simple mode**: First user message becomes the title. First 10 messages as preview. No API call.
+
+Controlled by `CRYSTAL_SUMMARY_MODE` env var (`llm` or `simple`).
+
+## Embedding Providers
+
+| Provider | Model | Dimensions | Cost |
+|----------|-------|-----------|------|
+| OpenAI (default) | text-embedding-3-small | 1536 | ~$0.02/1M tokens |
+| Ollama | nomic-embed-text | 768 | Free (local) |
+| Google | text-embedding-004 | 768 | Free tier available |
+
+Set via `CRYSTAL_EMBEDDING_PROVIDER` env var or `--provider` flag.
+
+### Why These Three?
+
+- **OpenAI** ... best quality, lowest friction. Most people already have an API key.
+- **Ollama** ... fully offline. Zero cost. Privacy-first. No data leaves your machine.
+- **Google** ... free tier is generous. Good alternative if you don't want OpenAI.
+
+## Source File Indexing
+
+Add directories as "collections". Files are chunked, embedded, and tagged with file path + collection name. Searchable alongside conversations and memories.
+
+```bash
+crystal sources add /path/to/project --name my-project
+crystal sources sync my-project
+crystal sources status
+```
+
+Incremental sync detects changed files via SHA-256 content hashing. Only re-embeds what changed.
+
+## Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `CRYSTAL_EMBEDDING_PROVIDER` | `openai` | `openai`, `ollama`, or `google` |
+| `CRYSTAL_AGENT_ID` | `cc-mini` | Agent identifier for LDM paths |
+| `CRYSTAL_SUMMARY_MODE` | `llm` | `llm` or `simple` |
+| `CRYSTAL_SUMMARY_PROVIDER` | `openai` | Summary LLM provider |
+| `CRYSTAL_SUMMARY_MODEL` | `gpt-4o-mini` | Summary LLM model |
+| `CRYSTAL_DATA_DIR` | (auto) | Override DB location |
+| `CRYSTAL_RELAY_KEY` | ... | Shared encryption key for relay |
+| `CRYSTAL_RELAY_URL` | ... | Cloudflare Worker URL |
+| `CRYSTAL_REMOTE_URL` | ... | Remote Worker URL |
+| `CRYSTAL_REMOTE_TOKEN` | ... | Worker auth token |
+| `OPENAI_API_KEY` | ... | OpenAI key |
+| `GOOGLE_API_KEY` | ... | Google AI key |
+| `CRYSTAL_OLLAMA_HOST` | `http://localhost:11434` | Ollama server URL |
+| `CRYSTAL_OLLAMA_MODEL` | `nomic-embed-text` | Ollama model |
+
+### API Key Resolution
+
+1. Explicit override (programmatic)
+2. `process.env` (set by plugin or manually)
+3. `.env` file (`~/.openclaw/memory-crystal/.env`)
+4. 1Password CLI fallback
+
+## CLI Reference
+
+```bash
+# Search
+crystal search <query> [-n limit] [--agent <id>] [--provider <openai|ollama|google>]
+
+# Remember / forget
+crystal remember <text> [--category fact|preference|event|opinion|skill]
+crystal forget <id>
+
+# Status
+crystal status [--provider <openai|ollama|google>]
+
+# Source file indexing
+crystal sources add <path> --name <name>
+crystal sources sync [name]
+crystal sources status
+
+# LDM management
+crystal init [--agent <id>]
+crystal migrate-db
+```
+
+## MCP Tools
+
+| Tool | Description |
+|------|-------------|
+| `crystal_search` | Hybrid search across all memories |
+| `crystal_remember` | Store a fact or observation |
+| `crystal_forget` | Deprecate a memory by ID |
+| `crystal_status` | Chunk count, provider, agents |
+| `crystal_sources_add` | Add a directory for indexing |
+| `crystal_sources_sync` | Re-index changed files |
+| `crystal_sources_status` | Collection stats |
+
+## Migration
+
+### Legacy DB to LDM
+
+```bash
+crystal migrate-db
+```
+
+Copies the database to `~/.ldm/memory/crystal.db`. Verifies chunk count. Creates symlinks at the old path.
+
+### LanceDB to sqlite-vec
+
+```bash
+node scripts/migrate-lance-to-sqlite.mjs --dry-run   # check counts
+node scripts/migrate-lance-to-sqlite.mjs              # full migration
+```
+
+Reads vectors directly from LanceDB. No re-embedding needed. ~5,000 chunks/sec on M4 Pro.
+
+### context-embeddings.sqlite
+
+```bash
+node dist/migrate.js [--dry-run] [--provider openai]
+```
+
+Import from the older context-embeddings format (requires re-embedding).
+
+## Project Structure
+
+```
+memory-crystal/
+  src/
+    core.ts           Pure logic, zero framework deps
+    cli.ts            CLI wrapper (crystal command)
+    mcp-server.ts     MCP server (Claude Code, Claude Desktop)
+    openclaw.ts       OpenClaw plugin wrapper
+    cc-hook.ts        Claude Code Stop hook (auto-capture)
+    ldm.ts            LDM scaffolding and path resolution
+    summarize.ts      Markdown session summary generation
+    crypto.ts         AES-256-GCM + HMAC-SHA256 encryption
+    worker.ts         Cloudflare Worker (encrypted dead drop)
+    poller.ts         Relay poller (home machine side)
+    mirror-sync.ts    DB mirror sync (device side)
+    migrate.ts        Legacy migration tools
+    dev-update.ts     Auto dev-update generation
+  skills/
+    memory/SKILL.md   Agent skill definition
+  scripts/
+    migrate-lance-to-sqlite.mjs
+  dist/               Built output
+  ai/                 Plans, dev updates, todos
+```
+
+## Design Decisions
+
+**Why sqlite-vec over pgvector, Pinecone, Weaviate, etc.?**
+No server. No Docker. No cloud dependency. One file. Works offline. Backupable with `cp`. The tradeoff is scale ... sqlite-vec works great up to ~500K vectors. Beyond that, consider dedicated vector stores.
+
+**Why FTS5 + vectors instead of just vectors?**
+Vectors alone miss exact keyword matches. "error code 403" should match conversations containing "403", not just semantically similar conversations about HTTP errors. Hybrid search catches both.
+
+**Why RRF for fusion?**
+Reciprocal Rank Fusion is simple, robust, and doesn't require score calibration between the two engines. Each engine ranks results independently. RRF merges based on rank position, not raw scores.
+
+**Why recency weighting?**
+Without it, old conversations dominate. A conversation from 3 days ago about your current project should outrank a conversation from 3 months ago about a different project, even if the old one is a slightly better semantic match.
+
+**Why AES-256-GCM for relay encryption?**
+Authenticated encryption. Ciphertext tampering is detected. No padding oracle attacks. Standard, auditable, widely implemented. Combined with HMAC-SHA256 signing for belt-and-suspenders integrity verification.
+
+**Why a dead drop instead of direct device-to-device sync?**
+Devices aren't always online at the same time. A dead drop decouples sender and receiver. Your laptop drops encrypted blobs whenever it captures. Your desktop picks them up whenever it polls. No NAT traversal, no port forwarding, no peer discovery.
+
+**Why LanceDB dual-write?**
+Safety net during the sqlite-vec transition. Once sqlite-vec is proven stable at scale, LanceDB will be removed. Until then, both stores receive writes so rollback is possible.
+
+## Roadmap
+
+- **Phase 1** ... Complete. Local memory with CLI, MCP, OpenClaw plugin, Claude Code hook.
+- **Phase 2a** ... Complete. Source file indexing + QMD hybrid search (sqlite-vec + FTS5 + RRF).
+- **Phase 2b** ... Complete. Historical session backfill (152K+ chunks).
+- **Phase 2c** ... Complete. LDM scaffolding, JSONL archive, markdown summaries, relay merge.
+- **Phase 3** ... Planned. Hosted relay deployment.
+- **Phase 4** ... Planned. Remote MCP, GPT Action, multi-agent access.
+- **Cleanup** ... Planned. Remove LanceDB once sqlite-vec is proven stable.
+
+---
+
+Built by Parker Todd Brooks, Lēsa (OpenClaw, Claude Opus 4.6), Claude Code CLI (Claude Opus 4.6).
+
+Search architecture inspired by [QMD](https://github.com/tobi/qmd) by Tobi Lutke (MIT, 2024-2026).

package/ai/dev-updates/2026-02-25--cc-air--phase2-architecture-pivot.md

@@ -0,0 +1,70 @@
+# 2026-02-25 — Phase 2 Architecture Pivot: Cloud Mirror → Ephemeral Relay
+
+**Agent:** cc-air (Claude Code on MacBook Air)
+**Repo:** `memory-crystal`
+**Branch:** `cc-air/phase2-relay` (new, replaces `cc-mba/phase2-worker`)
+
+## What Happened
+
+Built the Phase 2 Cloudflare Worker as a cloud database mirror (D1 + Vectorize + R2 + FTS5). Full search on the Worker, persistent data in Cloudflare, ~350 lines of Worker code. It worked — built clean, types passed.
+
+Then Parker asked the right questions:
+1. What's the security risk of data in transit?
+2. What's the security risk of data sitting on Cloudflare?
+3. What happens when sync breaks and we lose context?
+
+Read the Dream Weaver Protocol architecture. The entire memory system is designed around sovereignty — "your memory, your machine, your rules." Putting conversation data persistently on Cloudflare violates that principle. And it's not just code conversations — it's everything said to every agent. Personal.
+
+Parker's core requirement: "I gotta be able to leave the house." Take the laptop, talk to cc-air, and have those conversations make it back to the master crystal on the Mini. But the data shouldn't *live* in the cloud. It should pass through.
+
+## The Pivot
+
+**Cloud Mirror (old):** Worker is a searchable database. Data persists on Cloudflare. Air searches the cloud. Cloudflare sees plaintext. OpenAI API key on Cloudflare.
+
+**Ephemeral Relay (new):** Worker is a dead drop. Data passes through encrypted, gets picked up, gets deleted. Nothing persists. Worker can't read what it holds. ~80 lines of code.
+
+### Architecture
+
+```
+Any Device → encrypt → Worker (dead drop) → Mini picks up → decrypt → embed → master crystal
+                                                                    │
+Mini → encrypt mirror snapshot → Worker → Device picks up → decrypt → local read-only mirror
+```
+
+- **Mini is the master.** All embedding, indexing, and search intelligence lives there.
+- **Devices are read-only.** They capture raw conversations and search against a local mirror. They never write to the crystal.
+- **Worker is a mailbox.** Encrypted blobs in, encrypted blobs out. TTL auto-expires everything in 24h. No D1, no Vectorize, no search, no API keys.
+
+### Security Hardening
+
+- **AES-256-GCM** client-side encryption. Key lives only on trusted machines, never on Cloudflare.
+- **HMAC-SHA256 drop signing.** Mini verifies every blob came from a trusted device.
+- **Random 96-bit nonces.** Prepended to ciphertext, collision-safe for billions of operations.
+- **SHA-256 mirror integrity.** Devices verify DB snapshots before replacing their local mirror.
+- **Key rotation mechanism.** 24h overlap for in-flight blobs, no re-encryption needed (ephemeral data).
+- **Explicit threat model.** Every risk documented with mitigation and residual risk.
+
+## Why This Matters Beyond Us
+
+This isn't just cc-air talking to the Mini. Any device, any agent, any interface can use the same pattern. Phone, laptop, tablet. Every conversation captured, encrypted, relayed, embedded, searchable. One master crystal, many readers. Sovereign memory that travels with you.
+
+Open source. Auditable. No subscription. No cloud lock-in.
+
+## Files
+
+| File | Status |
+|------|--------|
+| `ai/plan/phase2-ephemeral-relay.md` | NEW — full architecture spec with security model |
+| `ai/plan/dev-conventions-note.md` | NEW — documented existing dev conventions |
+| `ai/dev-updates/2026-02-25--cc-air--phase2-worker-build.md` | EXISTING — documents the original cloud mirror build |
+| `src/worker.ts` | TO BE REBUILT — ephemeral relay (~80 lines replacing ~350) |
+| `src/crypto.ts` | TO BE CREATED — AES-256-GCM + HMAC-SHA256 + HKDF |
+| `src/cc-hook.ts` | TO BE MODIFIED — encrypt + relay instead of direct ingest |
+| `src/poller.ts` | TO BE CREATED — Mini-side pickup + ingest |
+| `src/mirror-sync.ts` | TO BE CREATED — Device-side mirror pull |
+| `wrangler.toml` | TO BE SIMPLIFIED — R2 only, no D1/Vectorize |
+| `schema.sql` | TO BE REMOVED — no database on Worker |
+
+## Previous Branch
+
+`cc-mba/phase2-worker` — contains the cloud mirror build. Preserved so people can see the progression from "persistent cloud DB" to "ephemeral encrypted relay." The thinking is the product.

package/ai/dev-updates/2026-02-25--cc-air--phase2-worker-build.md

@@ -0,0 +1,72 @@
+# 2026-02-25 — Phase 2 Worker Build
+
+**Agent:** cc-a (Claude Code on MacBook Air)
+**Repo:** `memory-crystal`
+**Branch:** `cc-mba/phase2-worker`
+
+## What Was Done
+
+### 1. Read & Understood the Entire Codebase
+- `core.ts` (1,346 lines) — hybrid search: sqlite-vec + FTS5 + RRF fusion + recency weighting
+- `mcp-server.ts` — 7 MCP tools for Claude Code (search, remember, forget, status, sources)
+- `cc-hook.ts` — Stop hook with byte-offset watermarking, batched ingestion
+- `openclaw.ts` — Lesa's agent_end hook with compaction detection
+- `resolveConfig()` — 4-level config resolution (params → env → .env → 1Password)
+
+### 2. Built the Cloudflare Worker (`src/worker.ts`)
+- REST API endpoints: `/search`, `/ingest`, `/remember`, `/forget`, `/status`, `/health`
+- Sync endpoints: `/sync/push` (Mini uploads snapshot to R2), `/sync/pull` (Mini pulls new remote writes)
+- Auth: bearer token per agent (cc-a, cc, lesa) mapped to agent_id
+- Search: FTS5 on D1 + Vectorize cosine similarity + RRF fusion + recency weighting
+- Deduplication: SHA-256 hash check before ingest
+- Embedding: OpenAI text-embedding-3-small via fetch
+
+### 3. Created D1 Schema (`schema.sql`)
+- `chunks` table with hash-based dedup, agent_id, source tracking
+- `chunks_fts` virtual table (FTS5 with Porter stemming)
+- Triggers to keep FTS in sync with chunks
+- `memories` table for explicit remember/forget
+- `capture_state` and `sync_log` for sync tracking
+
+### 4. Created Wrangler Config (`wrangler.toml`)
+- D1 binding (needs database_id after creation)
+- Vectorize binding (memory-crystal-chunks, 1536 dimensions)
+- R2 bucket binding (memory-crystal-snapshots)
+- Secrets: AUTH_TOKEN_CC_A, AUTH_TOKEN_CC, AUTH_TOKEN_LESA, OPENAI_API_KEY
+
+### 5. Added Remote Mode to Core (`core.ts`)
+- New `RemoteCrystal` class — same interface as `Crystal`, talks to Worker via HTTP
+- `createCrystal()` factory — returns `RemoteCrystal` when `remoteUrl` + `remoteToken` are set, otherwise local `Crystal`
+- `chunkText()` method on RemoteCrystal for cc-hook compatibility
+
+### 6. Updated MCP Server (`mcp-server.ts`)
+- Uses `createCrystal()` — auto-detects remote mode
+- Shows "(REMOTE)" in status when using cloud mirror
+- Source indexing tools gracefully return "not available in remote mode"
+
+### 7. Updated CC Hook (`cc-hook.ts`)
+- Uses `createCrystal()` for remote ingestion support
+- Configurable `CRYSTAL_AGENT_ID` env var (defaults to 'claude-code', set to 'cc-a' on MacBook Air)
+- Type-safe with `Crystal | RemoteCrystal` union
+
+### 8. Build Verification
+- `npm run build:local` — succeeds, all types pass
+- `npm run build:worker` — succeeds, worker.js = 12 KB
+
+## Current Status
+
+**Code: COMPLETE.** All source files written and building clean.
+
+**Not yet deployed.** Needs Cloudflare resources created and secrets set.
+
+## Files Changed/Created
+
+| File | Action |
+|------|--------|
+| `src/worker.ts` | NEW — Cloudflare Worker |
+| `schema.sql` | NEW — D1 database schema |
+| `wrangler.toml` | NEW — Cloudflare config |
+| `src/core.ts` | MODIFIED — added RemoteCrystal + createCrystal() |
+| `src/mcp-server.ts` | MODIFIED — remote mode support |
+| `src/cc-hook.ts` | MODIFIED — remote mode + configurable agent_id |
+| `package.json` | MODIFIED — added build:worker and build:local scripts |

package/ai/dev-updates/2026-02-26--10-25-16--cc-mini--phase2-implementation.md

@@ -0,0 +1,49 @@
+# Phase 2 Implementation
+
+*Dev update by cc-mini, 2026-02-26 10:25 PST*
+
+**Branch:** `mini/phase2-relay`
+
+## What happened
+
+Built and merged Memory Crystal Phase 2. The system now produces all three file types (JSONL transcripts, MD session summaries, vector DB) and manages the full `~/.ldm/` directory tree. Also merged cc-air's ephemeral relay code and expanded the poller to reconstruct remote agent file trees.
+
+## New modules
+
+- **`src/ldm.ts`** -- Central LDM path resolution and scaffolding. `getAgentId()`, `ldmPaths()`, `scaffoldLdm()`, `ensureLdm()`. Every other file imports paths from here.
+- **`src/summarize.ts`** -- MD session summary generation. Two modes: `simple` (no API call, first message becomes title) and `llm` (calls gpt-4o-mini for title + summary + topics). Controlled by `CRYSTAL_SUMMARY_MODE` env var.
+
+## Changes to existing modules
+
+- **`src/core.ts`** -- `resolveConfig()` checks `~/.ldm/memory/crystal.db` first, falls back to legacy path. Added `RemoteCrystal` class and `createCrystal()` factory from relay merge.
+- **`src/cli.ts`** -- Added `crystal init [--agent]` and `crystal migrate-db` subcommands. Init scaffolds the full directory tree. Migrate copies the DB, verifies chunk count, creates symlink.
+- **`src/cc-hook.ts`** -- Three new behaviors after each capture: (1) archive JSONL transcript to `~/.ldm/agents/{id}/transcripts/`, (2) generate MD session summary to `sessions/`, (3) relay mode from cc-air merge (encrypt + drop at Worker when env vars set).
+- **`src/poller.ts`** -- After decrypting relay drops, now reconstructs the remote agent's full file tree: JSONL transcript, MD summary, daily breadcrumb. This means the Mini has everything even if a device is lost.
+- **`src/mirror-sync.ts`** -- Updated paths to use `ldmPaths()` instead of hardcoded `~/.openclaw/memory-crystal/`.
+- **`src/dev-update.ts`** -- Writes to each repo's own `ai/` folder instead of centralized `wip-dev-updates`. Disabled the git add/commit/push to the old repo.
+- **`src/mcp-server.ts`** -- Uses `createCrystal()` factory. Remote mode guards on source indexing operations.
+
+## Portability fix
+
+All hardcoded `/Users/lesa` and `/Users/parker` HOME fallbacks replaced with `process.env.HOME || ''` across every source file. The old fallbacks were cc-air writing `/Users/parker` and the Mini having `/Users/lesa`. Neither matters in practice (HOME is always set), but it was wrong for anyone else installing.
+
+## Relay merge
+
+Merged `origin/cc-air/phase2-relay` into the branch. cc-air built the ephemeral encrypted relay: `src/crypto.ts` (AES-256-GCM + HMAC), `src/worker.ts` (Cloudflare dead drop), `src/poller.ts` (Mini-side ingest), `src/mirror-sync.ts` (device-side DB pull). One conflict in cc-hook.ts (imports + dev-update section), resolved to keep both relay mode and the new summary/archive code.
+
+## ai/ folder cleanup
+
+Established the inbox convention: `ai/todos/inboxes/{recipient}/`. Each agent or human gets an inbox. Moved salience-research from `artifacts/` to `ai/notes/`. Added `PUNCHLIST.md` for blockers to ship. Updated `wip-dev-resources/DEVELOPMENT-PROCESS.md` with the new convention.
+
+## Verified
+
+- `npm run build` (local + worker): zero errors, 12 entry points
+- `crystal status`: 159,574 chunks from existing DB
+- `crystal init --agent cc-mini`: scaffolds `~/.ldm/` correctly
+- `crystal search "test"`: returns results
+- `node poller.js --status`: "not configured" (correct)
+- `node mirror-sync.js --status`: shows state
+
+## What's next
+
+Parker's moves: run `crystal migrate-db`, deploy plugin, merge to main, release. Relay setup (Cloudflare) when ready. See `ai/todos/inboxes/parker/` and `ai/todos/PUNCHLIST.md`.