@wipcomputer/memory-crystal 0.7.28 → 0.7.30

package/.worktrees/memory-crystal-private--cc-mini-release-notes-v0.7.30/TECHNICAL.md

@@ -0,0 +1,656 @@
+###### WIP Computer
+
+# Technical Documentation
+
+How Memory Crystal works. Architecture, design decisions, integrations, encryption, search, and everything else the open source community is going to ask about.
+
+## How Does It Work?
+
+Memory Crystal captures every conversation you have with any AI tool, embeds it into a local SQLite database, and makes it searchable with hybrid search (keyword + semantic). One database file. Runs on your machine. Nothing leaves your device unless you set up multi-device sync.
+
+### Five-Layer Memory Stack
+
+| Layer | What | How |
+|-------|------|-----|
+| L1: Raw Transcripts | Every conversation archived as JSONL | Automatic capture (cron, hooks, plugins) |
+| L2: Vector Index | Chunks embedded into crystal.db | Automatic. Hybrid search (BM25 + vector + RRF) |
+| L3: Structured Memory | Facts, preferences, decisions | `crystal_remember` / `crystal_forget` |
+| L4: Narrative Consolidation | Dream Weaver journals, identity, soul | `crystal dream-weave` (via Dream Weaver Protocol) |
+| L5: Active Working Context | Boot sequence files, shared context | Agent reads on startup |
+
+Every conversation produces three artifacts:
+1. **JSONL transcript** ... the raw session, archived to disk
+2. **Markdown summary** ... title, summary, key topics (generated by LLM or simple extraction)
+3. **Vector embeddings** ... chunked, embedded, and stored in crystal.db for search
+
+## How Does It Work with Claude Code CLI?
+
+Two capture paths work together. The poller is primary. The Stop hook is redundancy.
+
+### Continuous Capture (Primary)
+
+A cron job runs `cc-poller.ts` every minute. It reads Claude Code's JSONL transcript files via byte-offset watermarking (only reads new data since last capture) and produces all three artifacts in a single pass:
+
+1. Extracts user, assistant, and thinking blocks
+2. Chunks them, embeds into sqlite-vec
+3. Archives the full JSONL transcript
+4. Generates a markdown session summary
+5. Appends a daily breadcrumb log
+
+**Install:**
+```bash
+crystal init                    # Scaffolds ~/.ldm/, deploys capture script, installs cron
+```
+
+This copies `crystal-capture.sh` to `~/.ldm/bin/` and installs a cron entry:
+```
+* * * * * ~/.ldm/bin/crystal-capture.sh >> ~/.ldm/logs/crystal-capture.log 2>&1
+```
+
+The script calls `node ~/.ldm/extensions/memory-crystal/dist/cc-poller.js`. The poller fetches the OpenAI API key internally via `opRead()` (1Password SA token). No secrets in the shell script.
+
+### Stop Hook (Redundancy)
+
+The Claude Code Stop hook runs `cc-hook.ts` after every response. It checks the watermark and flushes anything the poller missed. If the poller already captured everything, the Stop hook is a no-op.
+
+```json
+{
+  "hooks": {
+    "Stop": [{ "hooks": [{ "type": "command", "command": "node ~/.ldm/extensions/memory-crystal/dist/cc-hook.js", "timeout": 30 }] }]
+  }
+}
+```
+
+```bash
+node dist/cc-hook.js --on       # Enable capture
+node dist/cc-hook.js --off      # Pause capture
+node dist/cc-hook.js --status   # Check status
+```
+
+Respects private mode. When capture is off, nothing is recorded.
+
+### Why Both?
+
+The Stop hook only fires when a session ends. Long sessions, remote disconnects, and context compactions never trigger Stop. A 72-hour session produced zero captures with Stop-only. The poller decouples capture from the session lifecycle entirely.
+
+## How Does It Work with OpenClaw?
+
+Memory Crystal is an OpenClaw plugin. It registers tools (`crystal_search`, `crystal_remember`, `crystal_forget`, `crystal_status`) and an `agent_end` hook that captures conversations after every agent turn.
+
+After embedding, the plugin also syncs raw data to LDM:
+- Session JSONLs from `~/.openclaw/agents/main/sessions/`
+- Workspace .md files from `~/.openclaw/workspace/`
+- Daily logs from `~/.openclaw/workspace/memory/`
+
+This ensures LDM has a complete copy of all agent data, not just embeddings. Raw data is never modified. Copies are idempotent (skip if same size).
+
+Deployed to `~/.openclaw/extensions/memory-crystal/`. The plugin uses the same `core.ts` as every other interface. Same search, same database, same embeddings.
+
+## How Does It Work with ChatGPT and Claude (iOS, web, macOS)?
+
+Memory Crystal runs a remote MCP server (`worker-mcp.ts`) on Cloudflare Workers. ChatGPT and Claude connect via OAuth 2.1 on any surface: macOS app, iOS app, or web.
+
+Four tools: `memory_search`, `memory_remember`, `memory_forget`, `memory_status`.
+
+**Tier 1 (Sovereign):** Memories are encrypted and relayed to your Crystal Core. No cloud search. The cloud MCP tells the client "search is available on your local devices."
+
+**Tier 2 (Convenience):** Memories are stored in D1 + Vectorize for cloud search. Same hybrid search algorithm as local Crystal (BM25 + vector + RRF). Your Crystal Core is still the source of truth.
+
+**Source:** `src/worker-mcp.ts` (OAuth + MCP server), `src/cloud-crystal.ts` (D1 + Vectorize backend)
+
+## How Does It Work with Other Tools?
+
+Any tool that can run shell commands or call an MCP server can use Memory Crystal.
+
+- **MCP Server** ... `mcp-server.ts` exposes `crystal_search`, `crystal_remember`, `crystal_forget`, `crystal_status`, `crystal_sources_add`, `crystal_sources_sync`, `crystal_sources_status`. Works with Claude Desktop, Claude Code, or any MCP-compatible client.
+- **CLI** ... `crystal search "query"` from any terminal. Any tool with shell access can call it.
+- **Module** ... `import { MemoryCrystal } from 'memory-crystal'` for Node.js integration.
+
+## Crystal Core and Crystal Node
+
+Memory Crystal uses a Core/Node architecture for multi-device setups:
+
+- **Crystal Core** ... your master memory. All conversations, all embeddings, all memories. This is the database you cannot lose. Install it on something permanent: a desktop, a home server, a Mac mini
+- **Crystal Node** ... a synced copy on any other device. Captures conversations, sends them to the Core via encrypted relay. Gets a mirror back for local search. If a node dies, nothing is lost. The Core has everything
+
+One Core, many Nodes. The Core does embeddings. Nodes just capture and sync.
+
+**Role management:**
+- `crystal role` ... show current role (Core or Node) and what it's connected to
+- `crystal promote` ... make this machine the Crystal Core
+- `crystal demote` ... make this machine a Crystal Node (connects to an existing Core)
+
+You can move the Core later. Start on a laptop, get a desktop, run `crystal promote` on the desktop. The old Core becomes a Node. No data loss.
+
+**If you install the Core on a laptop:** set up automated backups. iCloud Drive, external drive, wherever you trust. Your Core is your memory. Back it up.
+
+## Architecture
+
+One core, multiple interfaces. Two Workers. Three relay channels.
+
+```
+Local:
+  sqlite-vec (vectors) + FTS5 (BM25) + SQLite (metadata)
+           |                 |                    |
+      core.ts ... pure logic, zero framework deps
+      |-- cli.ts            -> crystal search, dream-weave, backfill, serve
+      |-- mcp-server.ts     -> crystal_search (Claude Code, Claude Desktop) + MCP sampling
+      |-- openclaw.ts       -> plugin (OpenClaw agents) + raw data sync to LDM
+      |-- llm.ts            -> LLM provider cascade, query expansion, re-ranking
+      |-- search-pipeline.ts -> deep search pipeline (expand, search, RRF, rerank, blend)
+      |-- cc-hook.ts        -> Claude Code Stop hook + relay commands
+      |-- crystal-serve.ts  -> Crystal Core gateway (localhost:18790)
+      |-- dream-weaver.ts   -> Dream Weaver integration (narrative consolidation)
+      |-- staging.ts        -> New agent staging pipeline
+
+Cloud:
+  D1 (SQL + FTS5) + Vectorize (vectors)
+           |                 |
+      cloud-crystal.ts ... same search algorithm, Cloudflare backends
+      |-- worker-mcp.ts     -> OAuth 2.1 + MCP (ChatGPT, Claude)
+
+Relay:
+      worker.ts             -> Encrypted dead drop (R2, 3 channels)
+      poller.ts             -> Crystal Core pickup + staging + commands
+      mirror-sync.ts        -> DB mirror to devices
+      crypto.ts             -> AES-256-GCM + HMAC-SHA256
+
+Init + Backfill:
+      discover.ts           -> Harness auto-detection (CC + OpenClaw)
+      bulk-copy.ts          -> Raw file copy to LDM (idempotent)
+      oc-backfill.ts        -> OpenClaw JSONL parser
+```
+
+Every local interface calls the same `core.ts`. The cloud MCP calls `cloud-crystal.ts` which implements the same search algorithm against D1 + Vectorize. The relay Worker (`worker.ts`) is intentionally separate and blind.
+
+## Search: How Does It Work?
+
+Two-tier search system. Fast path (hybrid search) runs by default. Deep search adds LLM-powered query expansion and re-ranking for higher quality results. Falls back to fast path silently if no LLM provider is available.
+
+### Fast Path (Hybrid Search)
+
+1. Query goes to both FTS5 (keyword match) and sqlite-vec (vector similarity)
+2. FTS5 returns BM25-ranked results, normalized to [0..1) via `|score| / (1 + |score|)`
+3. sqlite-vec returns cosine-distance results via two-step query (MATCH first, then JOIN separately ... sqlite-vec hangs with JOINs in the same query)
+4. Reciprocal Rank Fusion merges both lists: `weight / (k + rank + 1)` with k=60, tiered weights (BM25 2x, vector 1x)
+5. Recency weighting applied on top: `max(0.3, exp(-age_days * 0.1))`
+6. Final results sorted by combined score
+
+### Deep Search (LLM-Powered, default)
+
+Deep search wraps the fast path with LLM intelligence. Implemented in `search-pipeline.ts`:
+
+1. **Strong signal detection:** BM25 probe first. If top score >= 0.85 with gap >= 0.15 to #2, skip expansion (answer already found).
+2. **Query expansion:** LLM generates 3 variations ... lexical (keyword-focused), vector (semantic rephrase), HyDE (hypothetical answer document). Each variation runs through the fast path.
+3. **RRF merge:** All results from original + expanded queries fused via Reciprocal Rank Fusion.
+4. **LLM re-ranking:** Top 40 RRF candidates scored by LLM for relevance to the original query.
+5. **Position-aware blending:** Top 3: 75% RRF + 25% reranker. Results 4-10: 60/40. Results 11+: 40/60. Trusts RRF for top positions, lets the reranker fix ordering in the tail.
+
+Deep search is the default. No flags needed. Falls back silently to fast path if no LLM provider is available.
+
+### LLM Provider Cascade
+
+The deep search pipeline tries providers in order. First available wins:
+
+| Priority | Provider | Cost | Speed |
+|----------|----------|------|-------|
+| 0 | **MCP Sampling** (if client supports it) | Included in Max subscription | Fast |
+| 1 | **MLX** (local, Apple Silicon) | Free | Fastest |
+| 2 | **Ollama** (local) | Free | Fast |
+| 3 | **OpenAI API** | ~$0.001/search | Network-dependent |
+| 4 | **Anthropic API** (direct key only, not OAuth) | ~$0.001/search | Network-dependent |
+| 5 | **None** | Free | N/A (fast path only) |
+
+Local-first by default. API keys are the fallback, not the primary path. MCP Sampling (priority 0) is designed and coded but waiting on Claude Code to implement it (Anthropic Issue #1785).
+
+Provider detection in `llm.ts`:
+- Check MCP sampling capability from connected client
+- Check `http://localhost:8080/v1/models` for MLX server
+- Check `http://localhost:11434/api/tags` for Ollama (filters out embedding-only models)
+- Check env var or 1Password for OpenAI key
+- Check env var for Anthropic key (skips OAuth tokens `sk-ant-oat01-`)
+- None found: log once, use fast path
+
+### Time-Filtered Search
+
+Search can be filtered by time: `--since 24h`, `--since 7d`, `--since 30d`, or an ISO date. Applied as a SQL WHERE clause before search. Available on CLI and MCP (`time_filter` parameter).
+
+### Intent Parameter
+
+`--intent <description>` disambiguates queries without adding search terms. Example: `crystal search "security" --intent "1Password"` steers toward 1Password-specific context, not repo permissions or agent secrets.
+
+Intent flows through: expansion prompt (guides LLM variations), disables strong-signal bypass, prepended to rerank query for LLM relevance scoring.
+
+### Explain Mode
+
+`--explain` shows per-result scoring breakdown: FTS (keyword) score, vector (semantic) score, RRF rank after fusion, reranker score from LLM, recency weight, final blended score. Makes search quality transparent and debuggable.
+
+### Candidate Limit
+
+`--candidates N` tunes the rerank pool size (default 40). Higher values give the LLM more results to evaluate. Lower values are faster.
+
+### LLM Cache
+
+Expansion + reranking results cached in `llm_cache` table with 7-day TTL. Same query returns instantly on second search. Cache is per-query, per-intent.
+
+### Recency Decay
+
+Exponential decay from 1.0 to floor 0.3. Day 0: 1.0, Day 1: 0.90, Day 3: 0.74, Day 7: 0.50, Day 14+: 0.3 (floor). Fresh context wins decisively. Old content still surfaces for strong matches but doesn't bury recent results.
+
+Freshness flags: fresh (<3 days), recent (<7 days), aging (<14 days), stale (14+ days).
+
+Inspired by and partially ported from [QMD](https://github.com/tobi/qmd) by Tobi Lutke (MIT, 2024-2026).
+
+### Why Hybrid?
+
+Vector search alone misses exact matches. Keyword search alone misses semantic similarity. Hybrid catches both. A search for "deployment process" will find conversations that use the word "deployment" (BM25) and conversations about "shipping code to production" (vector similarity).
+
+### Content Dedup
+
+SHA-256 hash of chunk text before embedding. Duplicate content is never re-embedded. This matters when the same conversation is captured by multiple hooks (e.g., Claude Code hook and OpenClaw plugin running simultaneously).
+
+## Database
+
+Everything lives in one file: `crystal.db`. Inspectable with any SQLite tool. Backupable with `cp`.
+
+### Schema
+
+| Table | Purpose |
+|-------|---------|
+| `chunks` | Chunk text, metadata, SHA-256 hash, timestamps |
+| `chunks_vec` | sqlite-vec virtual table (cosine distance vectors) |
+| `chunks_fts` | FTS5 virtual table (Porter stemming, BM25 scoring) |
+| `memories` | Explicit remember/forget facts |
+| `entities` | Knowledge graph nodes |
+| `relationships` | Knowledge graph edges |
+| `capture_state` | Watermarks for incremental ingestion |
+| `sources` | Ingestion source metadata |
+| `source_collections` | Directory collections for file indexing |
+| `source_files` | Indexed file records with content hashes |
+
+### Why SQLite?
+
+One file. No server. No Docker. No connection strings. Works on every platform. Inspectable with standard tools. Backupable with `cp`. Ships with every OS.
+
+sqlite-vec adds vector search as a virtual table. FTS5 adds full-text search. Both are SQLite extensions that work within the same database file.
+
+### DB Location
+
+`resolveConfig()` in `core.ts` checks in order:
+
+1. Explicit override (programmatic)
+2. `CRYSTAL_DATA_DIR` env var
+3. `~/.ldm/memory/crystal.db` (if it exists)
+4. `~/.openclaw/memory-crystal/` (legacy fallback)
+
+## Directory Structure
+
+Memory Crystal manages `~/.ldm/` ... the universal agent home directory for [LDM OS](https://github.com/wipcomputer/dream-weaver-protocol).
+
+```
+~/.ldm/
+  config.json                              version, registered agents array
+  bin/
+    crystal-capture.sh                     cron job script (deployed by crystal init)
+  logs/
+    crystal-capture.log                    cron output (persists across reboots)
+    ldm-backup.log                         backup script output
+  memory/
+    crystal.db                             shared vector DB (all agents)
+  extensions/
+    memory-crystal/dist/                   deployed JS (cc-poller.js, cc-hook.js, etc.)
+  state/                                   watermarks, poller state, capture state
+  staging/{agent_id}/                      new agent staging (before live ingest)
+    transcripts/                           staged transcripts
+    READY                                  marker file (triggers processing)
+  agents/{agent_id}/
+    SOUL.md                                agent soul (Dream Weaver L4)
+    IDENTITY.md                            agent identity (Dream Weaver L4)
+    CONTEXT.md                             agent context (Dream Weaver L4)
+    REFERENCE.md                           agent reference (Dream Weaver L4)
+    memory/
+      transcripts/                         full JSONL session transcripts
+      sessions/                            markdown session summaries
+      daily/                               daily breadcrumb logs
+      journals/                            Dream Weaver journals
+      workspace/                           synced workspace files (OpenClaw .md)
+```
+
+Path resolution is centralized in `src/ldm.ts`:
+- `getAgentId()` ... resolves from `CRYSTAL_AGENT_ID` env var, default `cc-mini`
+- `ldmPaths(agentId?)` ... returns all paths as an object (including workspace)
+- `scaffoldLdm(agentId?)` ... creates the full directory tree
+- `ensureLdm(agentId?)` ... idempotent check, scaffolds if needed
+- `deployCaptureScript()` ... copies crystal-capture.sh to `~/.ldm/bin/`
+- `installCron()` ... installs the every-minute cron entry (idempotent)
+- `removeCron()` ... removes the crystal-capture cron entry
+- `resolveStatePath()` ... resolves state file path (watermarks, poller state)
+- `stateWritePath()` ... writable state file path
+
+## Encryption: How Does It Work?
+
+For multi-device sync. All encryption happens on-device before anything touches the network.
+
+- **AES-256-GCM** for encryption. Authenticated encryption ... ciphertext tampering is detected.
+- **HMAC-SHA256** for signing. Integrity verification before decryption. If the signature doesn't match, the blob is rejected.
+- **Shared symmetric key** generated locally with `openssl rand -hex 32`. Never transmitted to the relay.
+- The relay stores and serves encrypted blobs. It has no decryption capability. Compromising the relay yields encrypted noise.
+
+### Key Management
+
+The same encryption key must be present on all devices. Options:
+- **1Password** ... store the key, both machines pull from 1Password via SA token
+- **AirDrop** ... direct transfer between Macs
+- **Manual** ... copy the key securely between machines
+
+### Relay Architecture
+
+1. **Crystal Node** (`cc-hook.ts` relay mode): Encrypts JSONL with AES-256-GCM, signs with HMAC-SHA256, drops at Cloudflare Worker. Can also send commands via `sendCommand()`.
+2. **Worker** (`worker.ts`): Stores encrypted blobs in R2 across three channels (`conversations`, `mirror`, `commands`). Pure dead drop. No decryption. Auto-cleans after 24h.
+3. **Crystal Core** (`poller.ts`): Polls Worker, downloads blobs, verifies HMAC, decrypts, ingests into crystal.db. Reconstructs remote agent's file tree (JSONL, MD summary, daily breadcrumb). Detects new agent IDs and routes to staging pipeline. Also polls commands channel and delivers to Crystal Core gateway.
+4. **Mirror sync** (`mirror-sync.ts`): Pushes delta chunks (new embeddings since last sync) + file tree deltas from Crystal Core to relay. Crystal Nodes pull, decrypt, and insert. Cold start gets a full export; after that, delta only.
+5. **Staging** (`staging.ts`): New agents from relay are staged before live ingest. Transcripts are written to `~/.ldm/staging/{agent_id}/`, then backfill + Dream Weaver full mode runs before promoting to live capture.
+
+### Sync Model
+
+**Core is the only embedder.** All embeddings happen on the Core machine. Nodes never embed locally. This prevents split-brain where a node has embeddings that Core doesn't due to network issues.
+
+**Delta sync, not full mirror.** The mirror channel sends only new chunks since last sync, not the entire crystal.db (1.9 GB+). Payload size is proportional to activity, not corpus size. Cold start (new node) gets a one-time full export, then delta only.
+
+**Full LDM tree sync.** The relay syncs the entire `~/.ldm/` file tree, not just the database. Embeddings are pointers to artifacts (files, images, videos). If the artifact isn't on the node, the embedding is an orphan. Every file that an embedding references must exist on every device.
+
+**No cloud search.** Every node has the full database + full file tree. All search is local. The Cloud MCP server (D1 + Vectorize) exists as a demo/onboarding tool but is not the production architecture.
+
+Three relay channels:
+- **conversations** (Node -> Core) ... raw conversation chunks for Core to embed
+- **mirror** (Core -> Nodes) ... delta chunks (pre-embedded) + file tree deltas
+- **commands** (bidirectional) ... Nodes send commands ("run Dream Weaver"), Core sends results
+
+### Future: Native Apple Sync
+
+For Apple-to-Apple devices, a native app replaces the relay entirely. CloudKit handles encrypted sync. MLX Swift handles on-device search quality LLM. No Cloudflare Worker needed between Apple devices. Same delta model. The relay stays for non-Apple and cross-platform setups.
+
+## Session Summaries
+
+`src/summarize.ts` generates markdown summaries. Two modes:
+
+**LLM mode** (default): Calls gpt-4o-mini with a condensed transcript. Returns title, slug, summary, key topics.
+
+**Simple mode**: First user message becomes the title. First 10 messages as preview. No API call.
+
+Controlled by `CRYSTAL_SUMMARY_MODE` env var (`llm` or `simple`).
+
+## Embedding Providers
+
+| Provider | Model | Dimensions | Cost |
+|----------|-------|-----------|------|
+| OpenAI (default) | text-embedding-3-small | 1536 | ~$0.02/1M tokens |
+| Ollama | nomic-embed-text | 768 | Free (local) |
+| Google | text-embedding-004 | 768 | Free tier available |
+
+Set via `CRYSTAL_EMBEDDING_PROVIDER` env var or `--provider` flag.
+
+### Why These Three?
+
+- **OpenAI** ... best quality, lowest friction. Most people already have an API key.
+- **Ollama** ... fully offline. Zero cost. Privacy-first. No data leaves your machine.
+- **Google** ... free tier is generous. Good alternative if you don't want OpenAI.
+
+## Source File Indexing
+
+Add directories as "collections". Files are chunked, embedded, and tagged with file path + collection name. Searchable alongside conversations and memories.
+
+```bash
+crystal sources add /path/to/project --name my-project
+crystal sources sync my-project
+crystal sources status
+```
+
+Incremental sync detects changed files via SHA-256 content hashing. Only re-embeds what changed.
+
+## Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `CRYSTAL_EMBEDDING_PROVIDER` | `openai` | `openai`, `ollama`, or `google` |
+| `CRYSTAL_AGENT_ID` | `cc-mini` | Agent identifier for LDM paths |
+| `CRYSTAL_SUMMARY_MODE` | `llm` | `llm` or `simple` |
+| `CRYSTAL_SUMMARY_PROVIDER` | `openai` | Summary LLM provider |
+| `CRYSTAL_SUMMARY_MODEL` | `gpt-4o-mini` | Summary LLM model |
+| `CRYSTAL_DATA_DIR` | (auto) | Override DB location |
+| `CRYSTAL_RELAY_KEY` | ... | Shared encryption key for relay |
+| `CRYSTAL_RELAY_URL` | ... | Cloudflare Worker URL |
+| `CRYSTAL_REMOTE_URL` | ... | Remote Worker URL |
+| `CRYSTAL_REMOTE_TOKEN` | ... | Worker auth token |
+| `OPENAI_API_KEY` | ... | OpenAI key |
+| `GOOGLE_API_KEY` | ... | Google AI key |
+| `CRYSTAL_OLLAMA_HOST` | `http://localhost:11434` | Ollama server URL |
+| `CRYSTAL_OLLAMA_MODEL` | `nomic-embed-text` | Ollama model |
+| `CRYSTAL_SERVE_PORT` | `18790` | Crystal Core gateway port |
+| `CRYSTAL_SERVE_TOKEN` | ... | Optional bearer token for gateway auth |
+
+### API Key Resolution
+
+1. Explicit override (programmatic)
+2. `process.env` (set by plugin or manually)
+3. `.env` file (`~/.ldm/memory/.env`)
+4. 1Password CLI fallback
+
+## CLI Reference
+
+```bash
+# Search
+crystal search <query> [-n limit] [--agent <id>] [--since <24h|7d|30d>]
+  [--intent <description>] [--candidates N] [--explain]
+  [--provider <openai|ollama|google>]
+
+# Remember / forget
+crystal remember <text> [--category fact|preference|event|opinion|skill]
+crystal forget <id>
+
+# Status
+crystal status [--provider <openai|ollama|google>]
+
+# MLX local LLM (Apple Silicon)
+crystal mlx setup                # auto-install Qwen2.5-3B, create LaunchAgent
+crystal mlx status               # check server health
+crystal mlx stop                 # stop the server
+
+# Source file indexing
+crystal sources add <path> --name <name>
+crystal sources sync [name]
+crystal sources status
+
+# Pairing (relay key sharing)
+crystal pair                          # Show QR code (generate key if none)
+crystal pair --code mc1:<base64>      # Receive key from another device
+
+# LDM management
+crystal init [--agent <id>]           # Scaffold LDM, discover sessions, copy to LDM
+crystal migrate-db
+
+# Crystal Core / Node management
+crystal role                          # Show current role (Core or Node) and connections
+crystal promote                       # Make this machine the Crystal Core
+crystal demote                        # Make this machine a Crystal Node
+
+# Backfill + migration
+crystal backfill [--agent <id>] [--dry-run] [--limit <n>]   # Embed raw transcripts from LDM
+crystal migrate-embeddings [--dry-run]                        # Migrate context-embeddings into crystal.db
+
+# Dream Weaver (narrative consolidation)
+crystal dream-weave [--agent <id>] [--mode full|incremental] [--dry-run]
+
+# Crystal Core gateway
+crystal serve [--port <n>]            # Start HTTP gateway (default: 18790)
+
+# Health + maintenance
+crystal doctor                        # Health check
+crystal backup                        # Backup crystal.db
+```
+
+## MCP Tools
+
+| Tool | Description |
+|------|-------------|
+| `crystal_search` | Hybrid search across all memories |
+| `crystal_remember` | Store a fact or observation |
+| `crystal_forget` | Deprecate a memory by ID |
+| `crystal_status` | Chunk count, provider, agents |
+| `crystal_sources_add` | Add a directory for indexing |
+| `crystal_sources_sync` | Re-index changed files |
+| `crystal_sources_status` | Collection stats |
+
+## Migration
+
+### Legacy DB to LDM
+
+```bash
+crystal migrate-db
+```
+
+Copies the database to `~/.ldm/memory/crystal.db`. Verifies chunk count. Creates symlinks at the old path.
+
+### LanceDB to sqlite-vec
+
+```bash
+node scripts/migrate-lance-to-sqlite.mjs --dry-run   # check counts
+node scripts/migrate-lance-to-sqlite.mjs              # full migration
+```
+
+Reads vectors directly from LanceDB. No re-embedding needed. ~5,000 chunks/sec on M4 Pro.
+
+### context-embeddings.sqlite (legacy migrate.ts)
+
+```bash
+node dist/migrate.js [--dry-run] [--provider openai]
+```
+
+Import from the older context-embeddings format (requires re-embedding).
+
+### context-embeddings.sqlite (direct copy, v0.5.0+)
+
+```bash
+crystal migrate-embeddings --dry-run   # Show what would migrate
+crystal migrate-embeddings              # Copy embeddings directly ($0)
+```
+
+Copies ~3,108 unique conversation chunks from context-embeddings.sqlite into crystal.db. Embeddings are directly compatible (same model: text-embedding-3-small, 1536d, float32). Zero API calls. SHA-256 dedup skips chunks already in crystal.
+
+### Backfill raw transcripts (v0.5.0+)
+
+```bash
+crystal backfill --agent cc-mini --dry-run    # Show file count, estimated tokens
+crystal backfill --agent cc-mini              # Embed all transcripts
+crystal backfill --agent lesa-mini            # Embed OpenClaw transcripts
+```
+
+Scans `~/.ldm/agents/{agentId}/memory/transcripts/*.jsonl`, auto-detects format (Claude Code vs OpenClaw), extracts messages, embeds into crystal.db. Watermark tracking prevents re-embedding. On Node devices, relays to Core instead of local embedding.
+
+## Project Structure
+
+```
+memory-crystal/
+  src/
+    core.ts           Pure logic, zero framework deps
+    cli.ts            CLI wrapper (crystal command)
+    mcp-server.ts     MCP server (Claude Code, Claude Desktop)
+    openclaw.ts       OpenClaw plugin wrapper + raw data sync to LDM
+    cc-poller.ts      Continuous capture (cron job, primary)
+    cc-hook.ts        Claude Code Stop hook (redundancy) + relay commands
+    ldm.ts            LDM scaffolding, path resolution, script deployment, cron
+    summarize.ts      Markdown session summary generation
+    crypto.ts         AES-256-GCM + HMAC-SHA256 encryption
+    role.ts           Core/Node role detection
+    doctor.ts         Health check (crystal doctor)
+    bridge.ts         Bridge detection (lesa-bridge, etc.)
+    discover.ts       Harness auto-detection (Claude Code + OpenClaw)
+    bulk-copy.ts      Raw file copy to LDM (idempotent)
+    oc-backfill.ts    OpenClaw JSONL parser
+    dream-weaver.ts   Dream Weaver integration (imports from dream-weaver-protocol)
+    crystal-serve.ts  Crystal Core gateway (localhost:18790)
+    staging.ts        New agent staging pipeline
+    llm.ts            LLM provider cascade (MLX > Ollama > OpenAI > Anthropic), query expansion, re-ranking
+    search-pipeline.ts Deep search pipeline (expand, search, RRF, rerank, blend)
+    worker.ts         Cloudflare Worker relay (encrypted dead drop, R2, 3 channels)
+    worker-mcp.ts     Cloud MCP server (OAuth 2.1 + DCR, ChatGPT/Claude)
+    cloud-crystal.ts  D1 + Vectorize backend (cloud search)
+    poller.ts         Relay poller (Crystal Core side) + staging + commands
+    mirror-sync.ts    DB mirror sync (device side)
+    migrate.ts        Legacy migration tools
+    pair.ts           QR code pairing logic
+  migrations/
+    0001_init.sql     OAuth tables (clients, codes, tokens, users)
+    0002_cloud_storage.sql  Cloud chunks + memories + FTS5
+  skills/
+    memory/SKILL.md   Agent skill definition
+  scripts/
+    crystal-capture.sh              Cron job script (source of truth, deployed to ~/.ldm/bin/)
+    ldm-backup.sh                   LDM backup script
+    deploy-cloud.sh                 1Password-driven Cloudflare deployment
+    migrate-lance-to-sqlite.mjs
+  wrangler.toml       Relay Worker config
+  wrangler-mcp.toml   Cloud MCP Worker config
+  dist/               Built output
+  ai/                 Plans, dev updates, todos (private repo only)
+```
+
+## Design Decisions
+
+**Why sqlite-vec over pgvector, Pinecone, Weaviate, etc.?**
+No server. No Docker. No cloud dependency. One file. Works offline. Backupable with `cp`. The tradeoff is scale ... sqlite-vec works great up to ~500K vectors. Beyond that, consider dedicated vector stores.
+
+**Why FTS5 + vectors instead of just vectors?**
+Vectors alone miss exact keyword matches. "error code 403" should match conversations containing "403", not just semantically similar conversations about HTTP errors. Hybrid search catches both.
+
+**Why RRF for fusion?**
+Reciprocal Rank Fusion is simple, robust, and doesn't require score calibration between the two engines. Each engine ranks results independently. RRF merges based on rank position, not raw scores.
+
+**Why recency weighting?**
+Without it, old conversations dominate. A conversation from 3 days ago about your current project should outrank a conversation from 3 months ago about a different project, even if the old one is a slightly better semantic match.
+
+**Why AES-256-GCM for relay encryption?**
+Authenticated encryption. Ciphertext tampering is detected. No padding oracle attacks. Standard, auditable, widely implemented. Combined with HMAC-SHA256 signing for belt-and-suspenders integrity verification.
+
+**Why a dead drop instead of direct device-to-device sync?**
+Devices aren't always online at the same time. A dead drop decouples sender and receiver. Your laptop drops encrypted blobs whenever it captures. Your desktop picks them up whenever it polls. No NAT traversal, no port forwarding, no peer discovery.
+
+**Why Dream Weaver as a separate protocol library?**
+The consolidation engine (prompts, parsing, orchestration) lives in the `dream-weaver-protocol` package. Memory Crystal imports it and provides hooks for crystal.db integration (embedding journals, extracting memories). This prevents bifurcation. The protocol repo is the canonical source for HOW to consolidate. Memory Crystal is WHERE the results go.
+
+**Why D1 + Vectorize for the cloud instead of sqlite-vec?**
+sqlite-vec runs inside a single SQLite file. Cloudflare Workers don't have persistent local filesystems. D1 provides serverless SQL with FTS5. Vectorize provides serverless vector search. Same search algorithm (BM25 + vector + RRF), different backends.
+
+## Roadmap
+
+- **Phase 1** ... Complete. Local memory with CLI, MCP, OpenClaw plugin, Claude Code hook.
+- **Phase 2a** ... Complete. Source file indexing + QMD hybrid search (sqlite-vec + FTS5 + RRF).
+- **Phase 2b** ... Complete. Historical session backfill (159K+ chunks).
+- **Phase 2c** ... Complete. LDM scaffolding, JSONL archive, markdown summaries, relay merge.
+- **Phase 3** ... Complete. Encrypted relay (Cloudflare Worker + R2), poller, mirror sync, QR pairing (`crystal pair`).
+- **Phase 4** ... Complete. Cloud MCP server (OAuth 2.1 + DCR, ChatGPT + Claude on all surfaces), D1 + Vectorize backend.
+- **Phase 5** ... Complete. Core/Node architecture, crystal doctor, crystal backup, crystal bridge.
+- **Phase 6** ... Complete. Init discovery, bulk copy, OpenClaw parser, backfill, CE migration.
+- **Phase 7** ... Complete. Dream Weaver integration (via dream-weaver-protocol), Crystal Core gateway, staging pipeline, commands channel.
+- **Phase 8** ... Complete. Search quality: exponential recency decay, time-filtered search, LLM query expansion + re-ranking (deep search), provider cascade (MLX > Ollama > OpenAI > Anthropic), MCP sampling integration (designed, waiting on Claude Code).
+- **Next** ... MLX auto-install during `crystal init`, local embeddings (zero API key default), LanceDB retirement.
+
+## More Info
+
+- [README.md](https://github.com/wipcomputer/memory-crystal/blob/main/README.md) ... What Memory Crystal is and how to install it.
+- [RELAY.md](https://github.com/wipcomputer/memory-crystal/blob/main/RELAY.md) ... Relay: Memory Sync, QR pairing, delta sync, file tree sync.
+
+---
+
+## License
+
+```
+src/core.ts, cli.ts, mcp-server.ts, skills/   MIT    (use anywhere, no restrictions)
+src/worker.ts, src/worker-mcp.ts               AGPL   (relay + cloud server)
+```
+
+AGPL for personal use is free.
+
+Built by Parker Todd Brooks, Lēsa (OpenClaw, Claude Opus 4.6), Claude Code CLI (Claude Opus 4.6).
+
+Search architecture inspired by [QMD](https://github.com/tobi/qmd) by Tobi Lutke (MIT, 2024-2026).