@wipcomputer/memory-crystal 0.7.10

package/.env.example

@@ -0,0 +1,20 @@
+# Memory Crystal — Environment Config
+# Copy to ~/.openclaw/memory-crystal/.env and fill in your keys.
+# Or skip this and use 1Password (keys auto-resolve from "Agent Secrets" vault).
+
+# Embedding provider: openai (default), ollama, or google
+# CRYSTAL_EMBEDDING_PROVIDER=openai
+
+# OpenAI (required if provider is openai)
+OPENAI_API_KEY=sk-...
+
+# Google (required if provider is google)
+# GOOGLE_API_KEY=AIza...
+
+# Ollama (only if provider is ollama — runs locally, no key needed)
+# CRYSTAL_OLLAMA_HOST=http://localhost:11434
+# CRYSTAL_OLLAMA_MODEL=nomic-embed-text
+
+# Cloud mirror (Phase 2)
+# CRYSTAL_REMOTE_URL=https://memory-crystal.your-worker.workers.dev
+# CRYSTAL_REMOTE_TOKEN=your-token

package/CHANGELOG.md

@@ -0,0 +1,367 @@
+# Changelog
+
+
+
+
+
+
+
+
+
+
+## 0.7.10 (2026-03-13)
+
+# Dev Update: Orphan Cleanup, DELETE Trigger, Doctor Fix
+
+**Date:** 2026-03-13
+**Author:** CC-Mini
+**Session:** memory-db-fix
+
+---
+
+## What Happened
+
+Parker ran the Memory Crystal install prompt and `crystal doctor` reported "Embeddings: FAILING ... no provider configured in env." Investigation revealed two separate issues:
+
+### Issue 1: Doctor False Positive
+
+`checkEmbeddingProvider()` in `doctor.ts` only checked `process.env.OPENAI_API_KEY`. But the cron job and hooks resolve the key from 1Password via the SA token at `~/.openclaw/secrets/op-sa-token`. The doctor didn't know about this path.
+
+**Fix:** Added `checkOpEmbeddings()` helper to `doctor.ts` that checks for the SA token file, then does a live `op read` to verify it works. Doctor now reports `ok: openai (via 1Password)` instead of `fail`.
+
+### Issue 2: Orphaned Vectors and FTS Entries
+
+On 2026-03-11, 141,652 bulk file-scan chunks were correctly deleted from the `chunks` table. Parker said: "Why are we indexing documents?" These were raw file scans (Python venv packages, TypeScript source, vendor code) with no conversational context.
+
+The deletion used raw SQL (`DELETE FROM chunks WHERE agent_id = 'system'`). But Memory Crystal had no DELETE trigger. The corresponding entries in `chunks_vec` (sqlite-vec) and `chunks_fts` (FTS5) were left orphaned.
+
+**Impact:**
+- 141,651 orphaned vectors (~875 MB)
+- 141,652 orphaned FTS entries
+- ~7% of search queries hit phantom results (silently filtered out)
+- Database: 1.96 GB (should have been ~1 GB)
+
+**Fix (three parts):**
+
+1. **DELETE trigger** added to `initChunksTables()` in `core.ts`:
+```sql
+CREATE TRIGGER IF NOT EXISTS chunks_cleanup AFTER DELETE ON chunks
+BEGIN
+  DELETE FROM chunks_vec WHERE chunk_id = OLD.id;
+  INSERT INTO chunks_fts(chunks_fts, rowid, text) VALUES('delete', OLD.id, OLD.text);
+END;
+```
+
+2. **`cleanOrphans()` method** added to Crystal class in `core.ts`. Counts orphaned vec/FTS entries, deletes vec orphans in batches of 1000, rebuilds FTS5 from scratch.
+
+3. **`crystal cleanup` CLI command** added to `cli.ts`. Handles the full workflow: backup, pause cron, clean orphans, VACUUM, resume cron. Supports `--dry-run`.
+
+**Cleanup results:**
+- 141,651 orphaned vectors removed
+- FTS rebuilt from 73,813 chunks in 5.7s
+- Database: 1.96 GB -> 1.45 GB (525 MB saved)
+- Verification: chunks = FTS entries = 73,813. Match: YES
+- Zero orphans remaining
+
+### Side Discovery: Plaintext SA Token
+
+During investigation, discovered that `~/.openclaw/secrets/op-sa-token` is a plaintext 1Password SA token readable by any process running as `lesa`. This is the bootstrap credential that unlocks all secrets. Bug report filed. Long-term fix: Lesa iOS app with remote biometrics (product doc written).
+
+### Product Rule Established
+
+Memory Crystal indexes conversations only. File content that appears in conversation turns (agent reads a file, discusses it) is captured as part of the conversation. Raw directory scanning without conversational context is not what Memory Crystal is for.
+
+## Files Changed
+
+| File | Change |
+|------|--------|
+| `src/core.ts` | Added `chunks_cleanup` DELETE trigger, `cleanOrphans()` method |
+| `src/cli.ts` | Added `crystal cleanup` command, updated imports and USAGE |
+| `src/doctor.ts` | Added `checkOpEmbeddings()` for 1Password detection |
+| `ai/product/bugs/2026-03-13--orphaned-vectors-and-fts-after-bulk-delete.md` | Bug report |
+
+## Related (wip-secrets-ios-private)
+
+| File | What |
+|------|------|
+| `ai/product/product-ideas/lesa-app-remote-biometrics.md` | Lesa app: remote biometrics for agent computers |
+| `ai/product/bugs/2026-03-13--plaintext-sa-token-on-disk.md` | Plaintext SA token bug report |
+
+## Status
+
+- Code deployed and running (cleanup already executed)
+- Not yet committed / PR'd / released
+- Needs: branch, commit, PR, merge, `wip-release patch`
+
+## 0.7.9 (2026-03-13)
+
+# Dev Update: Orphan Cleanup, DELETE Trigger, Doctor Fix
+
+**Date:** 2026-03-13
+**Author:** CC-Mini
+**Session:** memory-db-fix
+
+---
+
+## What Happened
+
+Parker ran the Memory Crystal install prompt and `crystal doctor` reported "Embeddings: FAILING ... no provider configured in env." Investigation revealed two separate issues:
+
+### Issue 1: Doctor False Positive
+
+`checkEmbeddingProvider()` in `doctor.ts` only checked `process.env.OPENAI_API_KEY`. But the cron job and hooks resolve the key from 1Password via the SA token at `~/.openclaw/secrets/op-sa-token`. The doctor didn't know about this path.
+
+**Fix:** Added `checkOpEmbeddings()` helper to `doctor.ts` that checks for the SA token file, then does a live `op read` to verify it works. Doctor now reports `ok: openai (via 1Password)` instead of `fail`.
+
+### Issue 2: Orphaned Vectors and FTS Entries
+
+On 2026-03-11, 141,652 bulk file-scan chunks were correctly deleted from the `chunks` table. Parker said: "Why are we indexing documents?" These were raw file scans (Python venv packages, TypeScript source, vendor code) with no conversational context.
+
+The deletion used raw SQL (`DELETE FROM chunks WHERE agent_id = 'system'`). But Memory Crystal had no DELETE trigger. The corresponding entries in `chunks_vec` (sqlite-vec) and `chunks_fts` (FTS5) were left orphaned.
+
+**Impact:**
+- 141,651 orphaned vectors (~875 MB)
+- 141,652 orphaned FTS entries
+- ~7% of search queries hit phantom results (silently filtered out)
+- Database: 1.96 GB (should have been ~1 GB)
+
+**Fix (three parts):**
+
+1. **DELETE trigger** added to `initChunksTables()` in `core.ts`:
+```sql
+CREATE TRIGGER IF NOT EXISTS chunks_cleanup AFTER DELETE ON chunks
+BEGIN
+  DELETE FROM chunks_vec WHERE chunk_id = OLD.id;
+  INSERT INTO chunks_fts(chunks_fts, rowid, text) VALUES('delete', OLD.id, OLD.text);
+END;
+```
+
+2. **`cleanOrphans()` method** added to Crystal class in `core.ts`. Counts orphaned vec/FTS entries, deletes vec orphans in batches of 1000, rebuilds FTS5 from scratch.
+
+3. **`crystal cleanup` CLI command** added to `cli.ts`. Handles the full workflow: backup, pause cron, clean orphans, VACUUM, resume cron. Supports `--dry-run`.
+
+**Cleanup results:**
+- 141,651 orphaned vectors removed
+- FTS rebuilt from 73,813 chunks in 5.7s
+- Database: 1.96 GB -> 1.45 GB (525 MB saved)
+- Verification: chunks = FTS entries = 73,813. Match: YES
+- Zero orphans remaining
+
+### Side Discovery: Plaintext SA Token
+
+During investigation, discovered that `~/.openclaw/secrets/op-sa-token` is a plaintext 1Password SA token readable by any process running as `lesa`. This is the bootstrap credential that unlocks all secrets. Bug report filed. Long-term fix: Lesa iOS app with remote biometrics (product doc written).
+
+### Product Rule Established
+
+Memory Crystal indexes conversations only. File content that appears in conversation turns (agent reads a file, discusses it) is captured as part of the conversation. Raw directory scanning without conversational context is not what Memory Crystal is for.
+
+## Files Changed
+
+| File | Change |
+|------|--------|
+| `src/core.ts` | Added `chunks_cleanup` DELETE trigger, `cleanOrphans()` method |
+| `src/cli.ts` | Added `crystal cleanup` command, updated imports and USAGE |
+| `src/doctor.ts` | Added `checkOpEmbeddings()` for 1Password detection |
+| `ai/product/bugs/2026-03-13--orphaned-vectors-and-fts-after-bulk-delete.md` | Bug report |
+
+## Related (wip-secrets-ios-private)
+
+| File | What |
+|------|------|
+| `ai/product/product-ideas/lesa-app-remote-biometrics.md` | Lesa app: remote biometrics for agent computers |
+| `ai/product/bugs/2026-03-13--plaintext-sa-token-on-disk.md` | Plaintext SA token bug report |
+
+## Status
+
+- Code deployed and running (cleanup already executed)
+- Not yet committed / PR'd / released
+- Needs: branch, commit, PR, merge, `wip-release patch`
+
+## 0.7.8 (2026-03-13)
+
+# Dev Update: Orphan Cleanup, DELETE Trigger, Doctor Fix
+
+**Date:** 2026-03-13
+**Author:** CC-Mini
+**Session:** memory-db-fix
+
+---
+
+## What Happened
+
+Parker ran the Memory Crystal install prompt and `crystal doctor` reported "Embeddings: FAILING ... no provider configured in env." Investigation revealed two separate issues:
+
+### Issue 1: Doctor False Positive
+
+`checkEmbeddingProvider()` in `doctor.ts` only checked `process.env.OPENAI_API_KEY`. But the cron job and hooks resolve the key from 1Password via the SA token at `~/.openclaw/secrets/op-sa-token`. The doctor didn't know about this path.
+
+**Fix:** Added `checkOpEmbeddings()` helper to `doctor.ts` that checks for the SA token file, then does a live `op read` to verify it works. Doctor now reports `ok: openai (via 1Password)` instead of `fail`.
+
+### Issue 2: Orphaned Vectors and FTS Entries
+
+On 2026-03-11, 141,652 bulk file-scan chunks were correctly deleted from the `chunks` table. Parker said: "Why are we indexing documents?" These were raw file scans (Python venv packages, TypeScript source, vendor code) with no conversational context.
+
+The deletion used raw SQL (`DELETE FROM chunks WHERE agent_id = 'system'`). But Memory Crystal had no DELETE trigger. The corresponding entries in `chunks_vec` (sqlite-vec) and `chunks_fts` (FTS5) were left orphaned.
+
+**Impact:**
+- 141,651 orphaned vectors (~875 MB)
+- 141,652 orphaned FTS entries
+- ~7% of search queries hit phantom results (silently filtered out)
+- Database: 1.96 GB (should have been ~1 GB)
+
+**Fix (three parts):**
+
+1. **DELETE trigger** added to `initChunksTables()` in `core.ts`:
+```sql
+CREATE TRIGGER IF NOT EXISTS chunks_cleanup AFTER DELETE ON chunks
+BEGIN
+  DELETE FROM chunks_vec WHERE chunk_id = OLD.id;
+  INSERT INTO chunks_fts(chunks_fts, rowid, text) VALUES('delete', OLD.id, OLD.text);
+END;
+```
+
+2. **`cleanOrphans()` method** added to Crystal class in `core.ts`. Counts orphaned vec/FTS entries, deletes vec orphans in batches of 1000, rebuilds FTS5 from scratch.
+
+3. **`crystal cleanup` CLI command** added to `cli.ts`. Handles the full workflow: backup, pause cron, clean orphans, VACUUM, resume cron. Supports `--dry-run`.
+
+**Cleanup results:**
+- 141,651 orphaned vectors removed
+- FTS rebuilt from 73,813 chunks in 5.7s
+- Database: 1.96 GB -> 1.45 GB (525 MB saved)
+- Verification: chunks = FTS entries = 73,813. Match: YES
+- Zero orphans remaining
+
+### Side Discovery: Plaintext SA Token
+
+During investigation, discovered that `~/.openclaw/secrets/op-sa-token` is a plaintext 1Password SA token readable by any process running as `lesa`. This is the bootstrap credential that unlocks all secrets. Bug report filed. Long-term fix: Lesa iOS app with remote biometrics (product doc written).
+
+### Product Rule Established
+
+Memory Crystal indexes conversations only. File content that appears in conversation turns (agent reads a file, discusses it) is captured as part of the conversation. Raw directory scanning without conversational context is not what Memory Crystal is for.
+
+## Files Changed
+
+| File | Change |
+|------|--------|
+| `src/core.ts` | Added `chunks_cleanup` DELETE trigger, `cleanOrphans()` method |
+| `src/cli.ts` | Added `crystal cleanup` command, updated imports and USAGE |
+| `src/doctor.ts` | Added `checkOpEmbeddings()` for 1Password detection |
+| `ai/product/bugs/2026-03-13--orphaned-vectors-and-fts-after-bulk-delete.md` | Bug report |
+
+## Related (wip-secrets-ios-private)
+
+| File | What |
+|------|------|
+| `ai/product/product-ideas/lesa-app-remote-biometrics.md` | Lesa app: remote biometrics for agent computers |
+| `ai/product/bugs/2026-03-13--plaintext-sa-token-on-disk.md` | Plaintext SA token bug report |
+
+## Status
+
+- Code deployed and running (cleanup already executed)
+- Not yet committed / PR'd / released
+- Needs: branch, commit, PR, merge, `wip-release patch`
+
+## 0.7.7 (2026-03-13)
+
+Update install prompt to new standard format. Replaces old 3-question prompt with 4 explain questions, installed check, and dry-run before install.
+
+## 0.7.6 (2026-03-13)
+
+Update LDM delegation tips
+
+## 0.7.5 (2026-03-13)
+
+# Release Notes: Memory Crystal v0.7.5
+
+## LDM OS Integration
+
+Memory Crystal now works with LDM OS when it's available.
+
+### crystal init delegates to ldm install
+
+When the `ldm` CLI exists on PATH, `crystal init` delegates generic deployment to it. LDM OS handles the scaffold, interface detection, and extension deployment. Memory Crystal keeps its own setup: database backup, role configuration, pairing, cron jobs.
+
+When `ldm` isn't available, `crystal init` works standalone like it always has. No new dependencies. No breaking changes.
+
+### LDM OS tip
+
+After install completes, Memory Crystal prints a tip: "Run `ldm install` to see more skills you can add." Helps users discover the rest of the ecosystem.
+
+### Part of LDM OS
+
+README now includes a "Part of LDM OS" section linking back to the LDM OS repo. Memory Crystal installs into LDM OS, the local runtime for AI agents.
+
+## 0.7.4 (2026-03-11)
+
+MCP fix (OPENCLAW_HOME env var), AgentId reads from LDM config instead of hardcoding, MCP registrations moved to user-level, 33 stale branches renamed, QMD v1.1.6 analysis documented
+
+## 0.7.3 (2026-03-10)
+
+Fix MCP registration to include OPENCLAW_HOME env var for memory-crystal MCP server
+
+## 0.7.2 (2026-03-05)
+
+Fix MCP detection in doctor and installer to check project-level and user-scope registrations
+
+## 0.7.1 (2026-03-05)
+
+Database backup, verification, and import in installer
+
+## 0.7.0 (2026-03-05)
+
+Delta sync, file sync, intelligent install & update
+
+## 0.6.1 (2026-03-05)
+
+Search quality: deep search with LLM query expansion + re-ranking, MCP sampling design, updated docs
+
+## 0.6.0 (2026-03-04)
+
+Dream Weaver integration, Crystal Core gateway, staging pipeline, commands channel.
+
+- Dream Weaver narrative consolidation via `crystal dream-weave` (imports engine from dream-weaver-protocol)
+- Crystal Core gateway (`crystal serve`) on localhost:18790, OpenAI-compatible endpoint
+- Staging pipeline for new agents from relay (auto-detect, stage, backfill, dream-weave)
+- Commands channel on relay Worker (nodes send commands to Core, Core sends results back)
+- OpenClaw raw data sync to LDM after every agent_end turn (sessions, workspace, daily logs)
+- Relay command support in cc-hook.ts (`sendCommand()` export)
+- Harness-aware init flow (OpenClaw vs Claude Code, Core vs Node)
+- Poller now detects new agents and routes to staging before live ingest
+
+## 0.5.0 (2026-03-04)
+
+Init discovery, bulk copy, OpenClaw parser, backfill, CE migration. Reorganize ai/ to ai/product/.
+
+- `crystal init` discovers session files on the current machine (Claude Code + OpenClaw)
+- `crystal backfill` embeds raw transcript files from LDM (Core: local embed, Node: relay to Core)
+- `crystal migrate-embeddings` migrates context-embeddings.sqlite chunks into crystal.db ($0, copies embeddings directly)
+- `src/discover.ts` auto-detects installed harnesses and session file locations
+- `src/bulk-copy.ts` copies raw files to LDM transcripts (idempotent, skip if same size)
+- `src/oc-backfill.ts` parses OpenClaw JSONL format into standard message format
+- Workspace path added to LDM (`~/.ldm/agents/{id}/memory/workspace/`)
+
+
+
+## 0.4.1 (2026-03-03)
+
+Crystal Core/Node architecture, crystal doctor, crystal backup, crystal bridge, SKILL.md onboarding rewrite
+
+## 0.3.3 (2026-03-02)
+
+Fix bin entries: crystal and crystal-mcp commands were missing from v0.3.2 due to npm stripping ./ prefix paths
+
+## 0.3.2 (2026-03-02)
+
+Rewrite SKILL.md as complete agent install guide. Add crystal-mcp binary for clean MCP config. CLI search output matches MCP server (freshness icons, numbered results). Agents can now auto-detect and install for Claude Code CLI, Claude Desktop, and OpenClaw.
+
+## 0.3.1 (2026-03-02)
+
+Fix npm package: exclude ai/ folder from published tarball
+
+## 0.3.0 (2026-03-02)
+
+Phase 1 continuous capture, Cloud MCP server, QR pairing, crystal init, docs overhaul
+
+## 0.2.0 (2026-02-28)
+
+README overhaul, relay encryption, QR pairing spec, Grok/Lesa feedback, disable auto dev-updates

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Parker Todd Brooks
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README-ENTERPRISE.md

@@ -0,0 +1,226 @@
+###### WIP Computer
+
+# Memory Crystal for Enterprise
+
+Agent memory infrastructure. Local-first. Encrypted. Inspectable. *In testing.*
+
+## The Problem
+
+Your organization runs AI agents across teams, departments, and tools. Each agent starts every session with no memory. No continuity. No shared context. Conversations with one agent are invisible to every other.
+
+Your agents can't remember what your people told them. Your people keep re-explaining themselves. Context is lost between sessions, between tools, between teams.
+
+This is not a convenience problem. It's a reliability problem. An auditability problem. A cost problem.
+
+## What Memory Crystal Does
+
+Memory Crystal is a persistent context layer for AI agents. One shared database. Hybrid search. On your machines. Under your control.
+
+- **Local-first.** All data stays on-prem. Nothing leaves your network unless you configure it to.
+- **Inspectable.** One SQLite file. Open it with any SQLite tool. Audit it. Query it. Back it up with `cp`.
+- **Deterministic search.** Hybrid retrieval (BM25 keyword + vector similarity + Reciprocal Rank Fusion). Same query, same results. No black-box ranking.
+- **Encrypted sync.** Multi-site deployments use AES-256-GCM encryption with HMAC-SHA256 signing. The relay sees encrypted noise. Keys never leave your machines.
+- **Agent isolation.** Each agent gets its own ID, its own transcript archive, its own session summaries. Shared search across agents, isolated storage per agent.
+- **Zero cloud dependency.** Runs fully offline with local embeddings (Ollama). No API keys required. No data exfiltration risk.
+
+## Five-Layer Memory Stack
+
+Memory Crystal implements a full memory pipeline, not just search.
+
+| Layer | What | How |
+|-------|------|-----|
+| L1: Raw Transcripts | Every conversation archived as JSONL | Automatic. cc-poller (cron), cc-hook (Stop), openclaw.ts (agent_end) |
+| L2: Vector Index | Chunks embedded into crystal.db | Automatic. Hybrid search (BM25 + vector + RRF) |
+| L3: Structured Memory | Facts, preferences, decisions | `crystal_remember` / `crystal_forget` |
+| L4: Narrative Consolidation | Dream Weaver journals, identity, soul | `crystal dream-weave` (imports engine from Dream Weaver Protocol) |
+| L5: Active Working Context | Boot sequence files, shared context | Agent reads on startup |
+
+L1-L3 are fully automated. L4 runs on-demand or via Crystal Core gateway. L5 is consumed by the agent's boot sequence.
+
+## Architecture
+
+```
+sqlite-vec (vectors) + FTS5 (BM25) + SQLite (metadata)
+         |                 |                    |
+    core.ts ... pure logic, zero framework deps
+    |-- cli.ts            -> crystal search, dream-weave, backfill, serve
+    |-- mcp-server.ts     -> MCP protocol (any compatible client)
+    |-- openclaw.ts       -> OpenClaw plugin + raw data sync to LDM
+    |-- cc-poller.ts      -> Continuous capture (cron, primary)
+    |-- cc-hook.ts        -> Claude Code Stop hook (redundancy) + relay commands
+    |-- crystal-serve.ts  -> Crystal Core gateway (localhost:18790)
+    |-- dream-weaver.ts   -> Dream Weaver integration (narrative consolidation)
+    |-- staging.ts        -> New agent staging pipeline
+    |-- llm.ts            -> LLM provider cascade (MLX > Ollama > OpenAI > Anthropic)
+    |-- search-pipeline.ts -> Deep search (expand, search, RRF, rerank, blend)
+    +-- worker.ts         -> Encrypted relay (multi-site sync, 3 channels)
+```
+
+One core module. Multiple interfaces. Every interface calls the same search engine. No inconsistency between access paths.
+
+## Security Model
+
+**Data at rest:** Single SQLite file. Standard filesystem permissions. Encrypt the volume if your compliance requires it.
+
+**Data in transit:** AES-256-GCM authenticated encryption. HMAC-SHA256 integrity verification. Shared symmetric key generated on-prem, never transmitted to the relay. The relay is a dead drop with no decryption capability.
+
+**Agent boundaries:** `CRYSTAL_AGENT_ID` isolates each agent's transcript archive, session summaries, and daily logs. Search spans all agents by default, or filters by agent ID.
+
+**Private mode:** Memory capture can be paused per-agent. When off, nothing is recorded. Resumes from where it left off when re-enabled.
+
+**No background processes that move data.** No telemetry. No analytics. No phone-home. The code is open source. Audit it.
+
+## Retrieval Quality
+
+Hybrid search is not "we added vectors." It's a two-tier retrieval engine with LLM-powered deep search.
+
+### Fast Path (Hybrid Search)
+- **FTS5 BM25** for exact keyword matches (Porter stemming)
+- **sqlite-vec cosine similarity** for semantic matches
+- **Reciprocal Rank Fusion** merges both result lists (k=60, tiered weights: BM25 2x, vector 1x)
+- **Recency weighting** ensures fresh context wins decisively: exponential decay `max(0.3, exp(-age_days * 0.1))`
+- **Content deduplication** via SHA-256 hash prevents duplicate embeddings
+- **Time-filtered search** ... restrict results to last 24h, 7d, 30d, or any date range
+
+### Deep Search (LLM-Powered, default)
+- **Query expansion** ... LLM generates 3 search variations (lexical, semantic, hypothetical document). Each runs through hybrid search. Results merged via RRF.
+- **Strong signal detection** ... BM25 probe skips expansion when the answer is obvious (saves latency).
+- **LLM re-ranking** ... top 40 candidates scored by LLM for relevance to the original query.
+- **Position-aware blending** ... trusts RRF for top positions, lets the reranker fix ordering in the tail.
+
+Deep search runs by default. Falls back to fast path silently if no LLM provider is available. For air-gapped environments, MLX (Apple Silicon) or Ollama provides free, fully local deep search with no API keys and no network.
+
+A search for "deployment policy" finds conversations containing those exact words (BM25), conversations about "shipping code to production" (vector similarity), and conversations about "release workflow" that the LLM recognizes as relevant. All three matter. All three surface.
+
+## What Gets Stored
+
+Every agent conversation produces three artifacts:
+
+| Artifact | Format | Location |
+|----------|--------|----------|
+| Raw transcript | JSONL | `~/.ldm/agents/{id}/memory/transcripts/` |
+| Session summary | Markdown | `~/.ldm/agents/{id}/memory/sessions/` |
+| Embeddings | sqlite-vec | `~/.ldm/memory/crystal.db` |
+
+Additionally:
+- **Explicit memories** stored via `crystal_remember` (facts, preferences, decisions)
+- **Source files** indexed as collections (code, documentation, internal knowledge bases)
+- **Daily logs** appended per-agent for audit trails
+- **Dream Weaver journals** generated by narrative consolidation (identity, soul, context, reference)
+- **Workspace files** synced from agent workspace to LDM (OpenClaw .md files)
+
+## Embedding Providers
+
+| Provider | Model | Dimensions | Network Required |
+|----------|-------|-----------|-----------------|
+| Ollama (recommended for enterprise) | nomic-embed-text | 768 | No. Fully local. |
+| OpenAI | text-embedding-3-small | 1536 | Yes. API calls. |
+| Google | text-embedding-004 | 768 | Yes. API calls. |
+
+For air-gapped environments, Ollama is the only option. No data leaves the machine. No API keys. No external dependencies.
+
+## Multi-Site Sync
+
+For organizations with multiple offices or remote teams.
+
+**Architecture:** One Crystal Core (the primary embedder), many Crystal Nodes (capture and sync). Core is the only machine that writes embeddings. Nodes capture raw conversations and send them to Core. Core embeds, then pushes deltas back.
+
+**What syncs:**
+1. **Delta chunks** ... only new embeddings since last sync (not the full database)
+2. **Full file tree** ... the entire `~/.ldm/` directory: workspace files, daily logs, journals, media, everything an embedding references
+3. **Commands** ... bidirectional remote operations (run Dream Weaver, trigger backfill, request status)
+
+**How it works:**
+1. Each site runs Memory Crystal locally
+2. Core embeds all conversations into crystal.db (one source of truth for embeddings)
+3. New chunks + changed files are encrypted (AES-256-GCM) and signed (HMAC-SHA256)
+4. Encrypted deltas are dropped at a relay (hosted or self-hosted)
+5. Other sites poll, decrypt, and insert into their local crystal.db + file tree
+6. The relay deletes blobs after pickup
+
+**No cloud search.** Every node has the full database and full file tree. All search is local. The relay is pure transport. Nothing is stored or searchable in the cloud.
+
+**Self-hosted relay:** Deploy the Cloudflare Worker on your own Cloudflare account. Full control. No third-party data exposure.
+
+**Hosted relay:** Use our infrastructure. Free during beta. Your data is encrypted before it reaches us. We cannot read it.
+
+## Compliance
+
+- **Data residency:** All primary data is local. Relay blobs are encrypted and ephemeral.
+- **Auditability:** SQLite is inspectable. Every chunk has a timestamp, source, and SHA-256 hash.
+- **Right to delete:** `crystal forget <id>` deprecates specific memories. Database can be wiped entirely with standard file operations.
+- **Access control:** Filesystem permissions on the SQLite file. No built-in user auth (it's a local tool, not a SaaS).
+- **No vendor lock-in:** MIT licensed (local code). Standard SQLite format. Export with any SQLite tool.
+
+## Database
+
+One file: `crystal.db`. Contains:
+
+| Table | Purpose |
+|-------|---------|
+| `chunks` | Chunk text, metadata, SHA-256 hash, timestamps |
+| `chunks_vec` | sqlite-vec virtual table (vector search) |
+| `chunks_fts` | FTS5 virtual table (keyword search) |
+| `memories` | Explicit facts and preferences |
+| `capture_state` | Watermarks for incremental ingestion |
+| `source_collections` | Indexed directory collections |
+| `source_files` | File records with content hashes |
+
+No migrations server. No schema versioning service. It's SQLite. `sqlite3 crystal.db ".schema"` shows you everything.
+
+## Integration
+
+| Platform | Integration | Auto-Capture |
+|----------|------------|-------------|
+| Claude Code | Cron poller (`cc-poller.ts`, primary) + Stop hook (`cc-hook.ts`, redundancy) | Yes. Every minute via cron, plus flush on session end. |
+| OpenClaw | Plugin (`openclaw.ts`) + `agent_end` hook + raw data sync | Yes. Every turn. Also syncs sessions, workspace, daily logs to LDM. |
+| Claude Desktop | MCP server (`mcp-server.ts`) | Search only. Manual capture. |
+| Any MCP client | MCP server | Search only. Manual capture. |
+| Any shell-accessible tool | CLI (`crystal search`) | Manual. |
+| Custom agents | Node.js module (`import from 'memory-crystal'`) | Programmable. |
+
+## Crystal Core Gateway
+
+Crystal Core runs an HTTP gateway (`crystal serve`) on localhost:18790. OpenAI-compatible endpoint for agent-to-agent communication and automated processing.
+
+- `POST /v1/chat/completions` ... invoke `claude -p` through the gateway
+- `POST /process` ... trigger backfill, dream-weave, or staging processing
+- `GET /status` ... health check and crystal stats
+
+Localhost-only binding. Never exposed to the network. Optional bearer token auth.
+
+## New Agent Onboarding
+
+When a new agent connects via relay, Crystal Core automatically:
+1. Detects the unknown agent ID
+2. Routes to staging (`~/.ldm/staging/{agent_id}/`)
+3. Runs backfill (embed all transcripts)
+4. Runs Dream Weaver full mode (generate identity, soul, context, journals)
+5. Moves to live capture
+
+No manual intervention. The staging pipeline handles the cold-start problem.
+
+## Deployment
+
+```bash
+npm install memory-crystal
+crystal init --agent your-agent-id
+crystal status
+```
+
+For enterprise deployments across multiple machines, see [Relay: Memory Sync](https://github.com/wipcomputer/memory-crystal/blob/main/RELAY.md).
+
+For full technical details, see [Technical Documentation](https://github.com/wipcomputer/memory-crystal/blob/main/TECHNICAL.md).
+
+---
+
+## License
+
+```
+src/, skills/, cli.ts, mcp-server.ts   MIT    (use anywhere, no restrictions)
+worker/                                AGPL   (relay server)
+```
+
+AGPL for personal use is free.
+
+Built by Parker Todd Brooks, Lēsa (OpenClaw, Claude Opus 4.6), Claude Code CLI (Claude Opus 4.6).