th-memory-mcp 1.2.1 → 2.0.0

package/design.md

@@ -1,308 +1,98 @@
-# Design: Adaptive Memory MCP — behavior-learning memory system for OpenCode
-
-> Project: D:\Coding_Project\mcp
-> Date: 2026-08-26 (rev.3 — as-built updated after all phases implemented)
-> Status: **implementation complete** — server v1.1.0, 9 tools, tests passing 70/70 assertions
-
-## 1. Overview
-
-A system that lets OpenCode "remember and adapt" to the user, composed of 3 parts:
-
-1. **MCP Server (th-memory-mcp v1.1.0)** — stores/retrieves preferences, lessons, and usage history in SQLite, exposing 9 tools the AI can call
-2. **OpenCode Plugin (learning-capture)** — hooks events to auto-capture prompts/tool usage and injects the profile back into context on compaction
-3. **Global Instructions (memory-protocol.md)** — the Memory Protocol rules, attached to every agent/session via `"instructions"` in the global opencode.json
-
-### Important constraints
-LLM APIs are **not trained on our data** — the only real "learning" possible is **context-based learning**:
-- capture behavior → distill into preferences/lessons
-- recall into context at the start of a new session (AI calls `recall` / plugin injects)
-This is the same mechanism behind the memory features of leading AI products.
-
-## 2. Architecture
-
-```
-┌────────────────────────────────────────────┐
-│                  OpenCode                  │
-│                                            │
-│  ┌──────────────────┐   ┌───────────────┐  │
-│  │ learning-capture │   │   AI Agent    │  │
-│  │ Plugin (Bun)     │   │               │  │
-│  │ - message.updated│   │  calls MCP    │  │
-│  │ - tool.execute.* │   │  tools        │  │
-│  │ - compacting*    │   │               │  │
-│  └────────┬─────────┘   └──────┬────────┘  │
-└───────────┼─────────────────────┼──────────┘
-             │ write (bun:sqlite)  │ read/write (stdio JSON-RPC)
-             ▼                     ▼
-    ┌─────────────────────────────────────┐
-    │   th-memory-mcp v1.1.0 (Node+SDK)   │
-    │  better-sqlite3 (WAL) ◀── shared ── │
-    │  Tools (9): remember, recall,       │
-    │  get_profile, save_lesson,          │
-    │  search_history, forget,            │
-    │  memory_stats,                      │
-    │  get_recent_interactions,           │
-    │  export_memory                      │
-    └─────────────────────────────────────┘
-               │
-               ▼
-    D:/Coding_Project/mcp/data/memory.db
-```
-
-(*) compaction hook = `experimental.session.compacting` used in Phase 3
-
-### Learning loop
-1. **Capture** — plugin auto-writes prompts/tool usage to `interactions`; AI also saves preferences/lessons via tools
-2. **Distill** — summarize raw logs into profile: rule-based via `npm run distill` (Thai tokenization with Intl.Segmenter + prune older than 30 days) and AI-assisted via the Smart Distill workflow in the protocol
-3. **Recall** — new session: AI calls `get_profile` + `recall(topic)` per the Memory Protocol (global instructions)
-4. **Inject** — plugin auto-injects the profile on session compaction (`experimental.session.compacting`)
-
-## 3. Data Model (SQLite)
-
-DB file: `data/memory.db` (path overridable via `MEMORY_DB_PATH`)
-WAL mode + busy_timeout=5000 on every connection
-
-```sql
--- raw behavior (plugin writes)
-CREATE TABLE interactions (
-  id INTEGER PRIMARY KEY AUTOINCREMENT,
-  ts TEXT NOT NULL,                -- ISO datetime
-  session_id TEXT,
-  kind TEXT NOT NULL,              -- 'prompt' | 'tool_call' | 'error'
-  content TEXT NOT NULL,           -- text (truncated per rules)
-  meta TEXT                        -- JSON extra, e.g. tool name, project dir
-);
-
--- user preferences/requirements (AI/plugin writes)
-CREATE TABLE preferences (
-  id INTEGER PRIMARY KEY AUTOINCREMENT,
-  category TEXT NOT NULL,          -- work_style | coding_pref | language | domain | other
-  key TEXT NOT NULL,
-  value TEXT NOT NULL,
-  confidence REAL DEFAULT 0.5,     -- 0..1, +0.1 per repeated confirmation
-  source TEXT DEFAULT 'explicit',  -- explicit | corrected | inferred
-  updated_at TEXT NOT NULL,
-  UNIQUE(category, key)
-);
-
--- lessons from corrections
-CREATE TABLE lessons (
-  id INTEGER PRIMARY KEY AUTOINCREMENT,
-  situation TEXT NOT NULL,         -- original situation
-  mistake TEXT NOT NULL,           -- what was done wrong
-  correction TEXT NOT NULL,        -- correct approach
-  created_at TEXT NOT NULL
-);
-
--- distilled profile
-CREATE TABLE profile (
-  section TEXT PRIMARY KEY,        -- identity | goals | style | notes
-  content TEXT NOT NULL,
-  updated_at TEXT NOT NULL
-);
-
--- virtual table for search
-CREATE VIRTUAL TABLE search_index USING fts5(
-  ref_table, ref_id, title, body
-);
-```
-
-## 4. MCP Tools spec
-
-Server name: `th-memory-mcp`, version **1.1.0**, transport stdio
-Every tool returns `{ content: [{ type: "text", text }] }`; errors must be caught and returned as a message (never crash)
-
-| Tool | Args (zod) | Behavior |
-|------|-----------|----------|
-| `remember` | `category` enum, `key`: string, `value`: string | upsert preferences; same key → confidence += 0.1 (cap 1.0), update value+updated_at |
-| `recall` | `topic`: string, `limit`?: number (default 8) | FTS5 search search_index (preferences+lessons) + latest 20 matching interactions; grouped text, ≤ ~2000 chars |
-| `get_profile` | (none) | profile sections + top preferences (confidence desc, limit 15) + latest 5 lessons |
-| `save_lesson` | `situation`, `mistake`, `correction`: string | insert lessons + update search_index |
-| `search_history` | `query`: string, `limit`?: number (default 10) | FTS5 in interactions (kind='prompt'), 200-char snippets per row |
-| `forget` | `target_id`: number, `type`? enum("preference","lesson","interaction") | delete from table by id (+type prevents cross-table id clash) + sync search_index |
-| `memory_stats` | (none) | counts by kind + DB size + oldest/newest interaction + profile sections; ≤1500 chars |
-| `get_recent_interactions` | `limit`? (default 20, max 100), `kind`? enum("prompt","tool_call","error") | latest rows formatted `[id] ts [kind] content(300)`; ≤4000 chars |
-| `export_memory` | `includeInteractions`? bool (default false), `filename`? string | write JSON only under `data/exports/` (sanitize filename `[A-Za-z0-9._-]`, no `..`); return path+size+preview ≤500 chars |
-
-## 5. Plugin spec (learning-capture)
-
-File: `src/plugin/learning-capture.ts` → deploy to `~/.config/opencode/plugins/learning-capture.ts`
-Runtime: Bun (OpenCode plugins run on Bun) → uses `bun:sqlite` on the same DB (WAL supports multi-process)
-
-```ts
-// as-built: self-contained single file — logic inline, synced with src/lib/capture-core.ts
-// (declares minimal types itself; does not import @opencode-ai/plugin to avoid module resolution issues)
-import { Database } from "bun:sqlite"
-
-export const LearningCapture = async (ctx) => {
-  const db = new Database(process.env.MEMORY_DB_PATH ?? "D:/Coding_Project/mcp/data/memory.db")
-  db.exec("PRAGMA journal_mode=WAL; PRAGMA busy_timeout=5000;")
-  // CREATE TABLE IF NOT EXISTS interactions (...) in case DB was never created
-  const dedupe = createDedupe()
-  return {
-    event: async ({ event }) => {
-      // message.updated (role=user) → insert kind='prompt' (truncate 4000, dedupe by message id)
-      // session.error → insert kind='error'
-    },
-    "tool.execute.after": async (input, output) => {
-      // insert kind='tool_call' (dedupe by callID, truncate 500)
-    },
-    "experimental.session.compacting": async (input, output) => {
-      // buildProfileText(db): profile sections + top preferences (confidence desc, 15)
-      //   + latest 5 lessons → ≤3000 chars → output.context.push(txt)
-      // wrap everything in try/catch silently — failed injection does no harm
-    },
-  }
-}
-```
-
-Capture rules:
-- Dedupe by message id (prevent duplicate events) — keep a Set of recorded ids in process memory
-- Never store secrets: filter lines matching `/(api[_-]?key|secret|token|password)\s*[=:]/i` before saving
-- Every write must try/catch — the plugin must never crash OpenCode
-
-## 6. Making the AI use memory (Memory Protocol)
-
-Installed at 2 levels:
-
-1. **Global (in use)** — `~/.config/opencode/memory-protocol.md` attached via `"instructions"` in global opencode.json → covers **every agent, every session** without switching agents
-2. **Project-level (alternative)** — copy from `AGENTS.memory.example.md` into a project's AGENTS.md
-
-Protocol essentials:
-- call `get_profile` + `recall` before a new/complex task
-- `save_lesson` immediately when the user corrects you / `remember` immediately when the user states a preference / never guess — if recall finds nothing, ask
-- `search_history` when suspecting a prior conversation / `forget` after confirming with the user
-- call memory tools only when necessary (not every message) / never store secrets / if memory is offline, continue gracefully
-
-**Smart Distill**: when the user asks "summarize memory" → `get_recent_interactions(limit=50)` → analyze real patterns → save insights via `remember`/`save_lesson` → summarize to the user with the list of new items
-
-## 7. File structure
-
-```
-D:\Coding_Project\mcp\
-├── design.md                   # this document (rev.3 as-built)
-├── README.md                   # usage guide + scripts + tools
-├── package.json                # type: module, scripts: build/start/distill/test
-├── tsconfig.json               # NodeNext, ES2022, strict; exclude src/plugin + test
-├── .gitignore                  # node_modules, dist, data/
-├── data\                       # memory.db (+wal/shm) and exports\ (git ignored)
-├── src\
-│   ├── index.ts                # McpServer v1.1.0 + registerTool ×9 + StdioServerTransport
-│   ├── db.ts                   # schema init, WAL, helper query, FTS sync
-│   ├── lib\
-│   │   ├── capture-core.ts     # pure logic: filterSecrets/truncate/dedupe/buildRow/INSERT_SQL
-│   │   └── distill-core.ts     # pure logic: tokenize(Thai)/computeStats/formatProfileSections
-│   ├── distill.ts              # CLI: runDistill(db) + prune (RETENTION_DAYS default 30)
-│   ├── tools\
-│   │   ├── remember.ts recall.ts profile.ts lesson.ts history.ts forget.ts
-│   │   ├── memory_stats.ts recent_interactions.ts export_memory.ts
-│   └── plugin\
-│       └── learning-capture.ts # self-contained Bun plugin → deploy copy to ~/.config/opencode/plugins/
-├── test\
-│   ├── smoke.mjs               # 53 checks end-to-end JSON-RPC (spawns real server)
-│   ├── capture.test.mjs        # 8 checks (capture-core + SQL insert)
-│   └── distill.test.mjs        # 9 checks (tokenize/stats/runDistill/prune/idempotent)
-├── AGENTS.memory.example.md    # Memory Protocol + Smart Distill (project-level)
-└── opencode.example.json       # example mcp config
-```
-
-## 8. Technology
-
-| Part | Choice | Reason |
-|------|---------|--------|
-| MCP Server | Node.js ≥ 20 + TypeScript + `@modelcontextprotocol/sdk@1.30.0` + zod | official standard |
-| DB (server) | `better-sqlite3@12.x` + FTS5 | fast sync API, easy, prebuilt binary (no compile) |
-| DB (plugin) | `bun:sqlite` (built-in) | plugin runs on Bun, no native module install |
-| Thai tokenization | `Intl.Segmenter("th", { granularity: "word" })` + whitespace fallback | segment Thai (no spaces) built into Node |
-
-> as-built note: the plugin is **self-contained** (declares minimal types in-file), so `@opencode-ai/plugin` is not required
-
-## 9. Sub-tasks
-
-### Phase 1 — MVP: MCP Server ✅ 2026-08-25
-1. Init project: `"type": "module"`, deps: `@modelcontextprotocol/sdk`, `zod`, `better-sqlite3`; devDeps: `typescript`, `@types/node`, `@types/better-sqlite3`, `@opencode-ai/plugin`
-2. `src/db.ts`: schema per §3, WAL, busy_timeout, helper + FTS sync
-3. First 6 tools per §4 spec (separate files in `src/tools/` — later expanded to 9 in Phase 4)
-4. `src/index.ts`: McpServer("th-memory-mcp") + register + StdioServerTransport (**no console.log — stderr only**)
-5. Build + smoke test with MCP Inspector (`npx @modelcontextprotocol/inspector node dist/index.js`) — remember → recall → forget
-6. Create `opencode.example.json`:
-
-```json
-{
-  "$schema": "https://opencode.ai/config.json",
-  "mcp": {
-    "memory": {
-      "type": "local",
-      "command": ["node", "D:/Coding_Project/mcp/dist/index.js"],
-      "enabled": true,
-      "environment": {}
-    }
-  }
-}
-```
-
-7. Create `AGENTS.memory.example.md` per §6
-8. Guide user: merge config → restart OpenCode → test "remember I prefer pnpm" then ask back in a new session
-
-### Phase 2 — Plugin auto-capture ✅ 2026-08-26
-9. `src/plugin/learning-capture.ts` per §5 (dedupe + secret filter + try/catch everywhere)
-10. Copy to `~/.config/opencode/plugins/learning-capture.ts` → restart OpenCode → use a while → verify `interactions` has data (`search_history` finds old prompts)
-
-### Phase 3 — Inject + Distill ✅ 2026-08-26
-11. Add hook `"experimental.session.compacting"` to plugin: `output.context.push(profile text)` from get_profile logic
-12. Distill script: rule-based summarize interactions → profile sections (`npm run distill`, Thai tokenize via Intl.Segmenter) + prune older than RETENTION_DAYS
-
-### Phase 4 — Insight & Safety ✅ 2026-08-26
-13. 3 new tools: `memory_stats` / `get_recent_interactions` / `export_memory` (sanitize filename + write only under data/exports/) — server bump v1.1.0
-14. Smart Distill workflow added to memory-protocol.md (global) + AGENTS.memory.example.md + README.md
-
-> as-built note: global instructions (`memory-protocol.md` via `"instructions"` in opencode.json) replace a dedicated agent — covers every agent without switching; smoke test expanded to 53 checks including security cases (unsafe filename rejected)
-
-## 10. Risks and mitigation
-
-| Risk | Impact | Mitigation |
-|------|--------|-----------|
-| Context bloat from long recall | token waste | cap 2000 chars/tool call, default limit |
-| Wrong/stale memory | AI goes wrong | confidence + updated_at + tool forget + user review |
-| SQLite accessed by 2 processes (Bun+Node) | lock error | WAL mode + busy_timeout=5000 |
-| `message.updated` fires often | DB bloat/duplicate | dedupe by message id + truncate |
-| Secret leaks to DB | security | regex filter before every write |
-| stdout mixed with logs | protocol breaks | stderr only in server code |
-| Invalid config | OpenCode won't start | add `$schema` validated against https://opencode.ai/config.json |
-
-## 11. Dependencies
-
-- Node.js ≥ 20, npm
-- OpenCode supporting plugins + MCP (current version)
-- No external service/API — 100% local (privacy by design)
-
-## 12. Performance Budget (acceptance criteria)
-
-Building Agent must implement within this budget:
-
-| Item | Budget | Check |
-|------|--------|-------|
-| Query latency per tool call | < 100 ms (local SQLite) | time in smoke test |
-| Max output per tool | `recall` ≤ 2000 chars, `search_history` ≤ 200 chars/row, `get_profile` ≤ 3000 chars | assert in code (always truncate) |
-| Default limit | recall=8, search_history=10 rows | default in zod schema |
-| Plugin write per event | < 5 ms, fire-and-forget (no event-loop block) | code review |
-| Server startup | < 2 s to ready for initialize | time it |
-
-**Measured (2026-08-26):** latency per tool call **1–9 ms**, startup **792–997 ms**, every tool within budget, tests **70/70** (smoke 53 + capture 8 + distill 9)
-
-### Overhead prevention
-- Memory Protocol calls memory **only on new/complex tasks**, never every message
-- Graceful degradation: if DB/server errors, return a short error message and let the AI continue immediately; no tight retry until timeout
-- Never auto-inject profile every turn — inject only on compaction (Phase 3)
-
-### Long-term risks to monitor
-- Memory quality decay (self-contradiction) → use confidence + updated_at + forget + distill (Phase 3)
-- DB growth → FTS5 index supports it; plan periodic VACUUM/optimize
-
-## 13. Next phases (Optional / Future)
-
-- Semantic search with embeddings (local model or API) instead of FTS5
-- Usage statistics dashboard (small web app reading the DB)
-- Multi-project memory scoping (by directory/worktree)
-- Import memory from export file (export side done in Phase 4)
-- Automatic LLM-assisted distill via OpenCode SDK (instead of user-triggered command)
+# th-memory-mcp v2 — Implementation Plan (design.md)
+
+**Source of truth:** `ARCHITECTURE_v2.md` (on GitHub, baseline v1.2.2).
+This file is the working plan for the Building Agent — read it before continuing implementation.
+
+## Goal
+Evolve th-memory-mcp from a structured local memory MCP into a durable, temporal,
+conflict-aware, hybrid-retrieval memory engine. Local-first, offline, SQLite, no
+mandatory cloud/LLM. Keep v1 behavior working during the transition.
+
+## Current v1 state (summary)
+- 9 MCP tools: remember, recall, get_profile, save_lesson, search_history, forget, memory_stats, get_recent_interactions, export_memory.
+- Schema (inline `CREATE TABLE IF NOT EXISTS` in `src/db.ts`): `interactions`, `preferences`, `lessons`, `profile`, FTS5 `search_index`, `embeddings` (BLOB, 512-dim hashing-trick vectors).
+- No migration system; no schema version table.
+- Capture logic triplicated: `src/lib/capture-core.ts`, `src/plugin/learning-capture.ts` (Bun), `scripts/claude-capture.mjs`.
+- Semantic search = full in-memory linear scan over all embeddings every `recall`.
+- Version metadata inconsistent: `package.json` 1.2.2 vs `config.ts` VERSION 1.1.0 vs `smoke.mjs` assertion 1.1.0 (fix in Phase 10).
+
+## Decisions (Phase 1)
+- **Migrations are TS modules** (`src/db/migrations.ts`) exporting an ordered `MIGRATIONS` array + `runMigrations(db)`. Each `up(db)` is idempotent (`CREATE TABLE IF NOT EXISTS`) and tracked in `schema_meta`. This avoids `.sql` file-copy issues under `tsc` while keeping deterministic order (spec allows implementation differences).
+- **Reuse existing `search_index` + `embeddings`** for v2 `memories` (ref_table = `'memories'`). No new FTS table needed.
+- **Non-destructive:** v1 tables (`preferences`, `lessons`, `interactions`, `profile`) are preserved. v2 adds `memories`, `entities`, `relations`, `memory_links`, `schema_meta`.
+- **Backfill (M005):** map `preferences → memories(type=PREFERENCE)`, `lessons → memories(type=LESSON)`, sync FTS+embeddings. Guarded by `v1_backfilled` flag so it runs once. `recall` is unaffected because it filters by `ref_table IN ('preferences','lessons')`.
+- **No dual-write yet.** v1 tools keep writing only to v1 tables. v2 `memories` is seeded by backfill; new v2 tools (later phases) write to `memories`. Dedup/merge of backfilled vs new entries is Phase 4.
+- **Repository layer** (`src/db/repositories/memories.ts`) provides `createMemory`, `getMemoryById`, `setStatus`, `softDelete`, `syncMemoryIndex`, `searchMemories` (FTS + semantic blend, status/project filtering). Not yet wired to a public tool (that is Phase 7 `get_context`).
+
+## Phased roadmap
+See `ARCHITECTURE_v2.md` §35 for the canonical phase list. Status tracked in the session todo list.
+
+## This session (deliverables so far)
+
+### Phase 1 — Core abstraction (DONE)
+- [x] `src/memory/types.ts` — unified `MemoryType`, `SourceType`, `LifecycleState`, `Scope`, `LinkRelation`, `MemoryRecord`.
+- [x] `src/db/migrations.ts` — migration engine + 5 migrations (schema_meta, memories+indexes, entities/relations, memory_links, v1 backfill).
+- [x] `src/db/repositories/memories.ts` — core CRUD + index sync + `searchMemories` (hybrid FTS + semantic blend).
+- [x] `src/db/index.ts` — call `runMigrations(db)` after existing DDL (non-destructive).
+- [x] Build + full test suite green (capture/distill/smoke).
+
+### Phase 2 — Lifecycle engine (DONE)
+- [x] `src/memory/decay.ts` — `recencyFactor`, per-type `DECAY_LAMBDA_BY_TYPE` (policy classes, not constants).
+- [x] `src/memory/source-weights.ts` — `SOURCE_WEIGHTS` map (spec §8).
+- [x] `src/memory/scorer.ts` — `computeSalience` (weighted, configurable), `computeConfidence` (source weight + diminishing returns), `salienceForMemory`.
+- [x] `src/core/lifecycle-engine.ts` — `canTransition`, `transitionStatus`, `reinforce`, `touch`, `supersede` (sets old=superseded, new=active + `supersedes_id` + `memory_links`), `archive`, `softDelete`, `LifecycleError`.
+- [x] `test/lifecycle.test.mjs` — 17 checks (decay, scorer, transitions, supersession, archive). Added to `npm test`.
+
+### Phase 3 — Temporal model (DONE)
+- [x] `src/core/temporal-engine.ts` — `setValidity`, `memoriesValidAt` (point-in-time truth), `supersessionChain` (oldest→newest), `changesBetween` (change detection).
+- [x] `test/temporal.test.mjs` — 7 checks (validity intervals, historical retrieval, supersession chains, change detection). Added to `npm test`.
+
+### Phase 4 — Conflict & dedup (DONE)
+- [x] `src/memory/deduplicator.ts` — `normalizeText`, `findExactMatch`, `findSimilar`, `deduplicate` (spec §11).
+- [x] `src/memory/conflict-resolver.ts` — `isContradiction`, `classifyRelationship` (duplicate/update/contradiction/unrelated), `findRelated`, `resolveConflict` (merge duplicate / supersede update / link contradiction, preserving ambiguous evidence per §12).
+- [x] `test/conflict.test.mjs` — 14 checks. Added to `npm test`.
+- [x] **Bug fix (v1 too):** `src/lib/embed.ts` `serialize`/`deserialize` rewrote with `DataView` + explicit `byteOffset`. Old code used `Buffer.from(buf).buffer` which can carry a non-zero pool `byteOffset`, corrupting vectors (magnitude ~1e37). This silently broke v1 semantic search.
+
+### Phase 5 — Hybrid retrieval (DONE)
+- [x] `src/retrieval/fts.ts` — `ftsSearch` (FTS5 over `search_index`, status/project filters, `ORDER BY rank`).
+- [x] `src/retrieval/vector.ts` — `vectorSearch` (cosine over `embeddings`, floor 0.15, filters).
+- [x] `src/retrieval/fusion.ts` — `rrfFuse` (Reciprocal Rank Fusion, k=60).
+- [x] `src/retrieval/scorer.ts` — `finalScore` (RRF × confidence × importance × recency × scope) + `scopeFactorFor`.
+- [x] `src/core/retrieval-engine.ts` — `retrieve` (FTS + vector → RRF → scoring/rerank → filter → topK).
+- [x] `searchMemories` in repository now delegates to `retrieve` (hybrid). `buildFtsMatch` switched to OR for better recall.
+- [x] `test/retrieval.test.mjs` — 7 checks. Added to `npm test`.
+
+### Phase 6 — Graph engine (DONE)
+- [x] `src/core/graph-engine.ts` — `createEntity` (canonical dedup, aliases in metadata), `addRelation` (source_entity_id/relation/target_entity_id), `linkMemories`, `traverse` (bounded BFS over `memory_links`, maxDepth 1–5, relationFilter), `neighbors`.
+- [x] `test/graph.test.mjs` — 7 checks (linking, bounded traversal depth, relation filter, entity dedup, relation insert). Added to `npm test`.
+- [x] Note: `entities` columns are `(name, canonical_name, type, metadata)`; `relations` use `source_entity_id/relation/target_entity_id`; `memory_links` PK `(source_memory_id, relation, target_memory_id)`.
+
+### Phase 7 — Context engine (DONE)
+- [x] `src/core/context-engine.ts` — `getContext` (hybrid retrieve → optional graph expansion → temporal validity filter → token budgeting/truncation).
+- [x] `src/tools/context.ts` — `contextInput` (zod) + `contextHandler` (returns assembled context text).
+- [x] Wired `get_context` MCP tool into `index.ts` (now 10 tools total).
+- [x] `test/context.test.mjs` — 7 checks (assembly, graph expansion, token budget, temporal validity). Added to `npm test`.
+- [x] Updated `test/smoke.mjs` to expect 10 tools.
+
+### Phase 8 — Consolidation (DONE)
+- [x] `src/core/consolidation-engine.ts` — `clusterMemories` (embedding cosine + union-find), `createDerivedMemory` (type DERIVED, source consolidated, links `derived_from`), `getProvenance`.
+- [x] `src/tools/consolidate.ts` — `consolidateInput` + `consolidateHandler` (read-only cluster listing + optional `derive`).
+- [x] Wired `consolidate` MCP tool into `index.ts` (now 11 tools total).
+- [x] `test/consolidation.test.mjs` — 5 checks. Added to `npm test`.
+- [x] Added `DERIVED` to `MEMORY_TYPES`; added `DERIVED` lambda to `decay.ts`.
+
+### Phase 9 — Benchmark & security suite (DONE)
+- [x] `test/benchmark.test.mjs` — 2 checks (retrieve over 300 memories < 2000ms).
+- [x] `test/security.test.mjs` — 5 checks (FTS injection quoting, safe retrieve, malicious content stored verbatim, extreme budget, parameterized SQL).
+- [x] Both added to `npm test` (now 12 suites).
+
+### Phase 10 — v2 release (DOCS DONE; PUBLISH PENDING USER)
+- [x] `MIGRATION_v2.md` written (non-destructive upgrade guide, rollback notes).
+- [x] `README.md` updated to v2.0.0 (11 tools, v2 architecture, new test scripts).
+- [ ] Version bump to `2.0.0` + `npm publish --otp=CODE` (needs user OTP).
+- [ ] `git commit` + `git push` + GitHub Release v2.0.0 (needs user).
+- [ ] `.\mcp-publisher.exe publish` (Official MCP Registry; needs user GitHub OAuth + OTP).
+- [ ] Glama: claim ownership + sync from GitHub.
+
+## Next
+All v2 engine phases (0–9) complete and tested. Remaining: user-driven release steps above. After release, future work could include automatic entity extraction in consolidation and a periodic auto-consolidate scheduler.

package/dist/core/consolidation-engine.js

@@ -0,0 +1,87 @@
+import { db, getAllEmbeddings } from "../db/index.js";
+import { cosine, deserialize } from "../lib/embed.js";
+import { createMemory } from "../db/repositories/memories.js";
+import { linkMemories } from "./graph-engine.js";
+// Group similar active memories into clusters via embedding cosine + union-find (spec §16)
+export function clusterMemories(opts = {}) {
+    const threshold = opts.threshold ?? 0.7;
+    const minSize = opts.minClusterSize ?? 2;
+    const rows = getAllEmbeddings();
+    const valid = rows.filter((r) => {
+        const m = db
+            .prepare("SELECT status, project_id FROM memories WHERE id = ?")
+            .get(r.ref_id);
+        if (!m)
+            return false;
+        if (!opts.includeArchived &&
+            (m.status === "deleted" || m.status === "archived"))
+            return false;
+        if (opts.projectId &&
+            !(m.project_id === opts.projectId || m.project_id === null))
+            return false;
+        return true;
+    });
+    const vecs = new Map();
+    for (const r of valid)
+        vecs.set(r.ref_id, deserialize(r.vec));
+    const ids = [...vecs.keys()];
+    const parent = new Map();
+    ids.forEach((id) => parent.set(id, id));
+    function find(x) {
+        let root = parent.get(x);
+        while (root !== undefined && root !== x) {
+            const next = parent.get(root);
+            if (next === undefined)
+                break;
+            parent.set(x, next);
+            x = root;
+            root = next;
+        }
+        return root ?? x;
+    }
+    function union(a, b) {
+        const ra = find(a);
+        const rb = find(b);
+        if (ra !== rb)
+            parent.set(ra, rb);
+    }
+    for (let i = 0; i < ids.length; i++) {
+        const a = ids[i];
+        for (let j = i + 1; j < ids.length; j++) {
+            const b = ids[j];
+            if (cosine(vecs.get(a), vecs.get(b)) >= threshold) {
+                union(a, b);
+            }
+        }
+    }
+    const groups = new Map();
+    for (const id of ids) {
+        const root = find(id);
+        if (!groups.has(root))
+            groups.set(root, []);
+        groups.get(root).push(id);
+    }
+    return [...groups.values()].filter((g) => g.length >= minSize);
+}
+// Create a derived/consolidated memory and link its sources via `derived_from` (spec §16)
+export function createDerivedMemory(input) {
+    const id = createMemory({
+        type: "DERIVED",
+        content: input.content,
+        summary: input.summary ?? null,
+        source: "consolidated",
+        projectId: input.projectId ?? null,
+    });
+    for (const src of input.sourceIds) {
+        if (src !== id)
+            linkMemories(src, id, "derived_from");
+    }
+    return id;
+}
+export function getProvenance(memoryId) {
+    const rows = db
+        .prepare(`SELECT source_memory_id FROM memory_links
+       WHERE target_memory_id = ? AND relation = 'derived_from'`)
+        .all(memoryId);
+    return rows.map((r) => r.source_memory_id);
+}

package/dist/core/context-engine.js

@@ -0,0 +1,50 @@
+import { db } from "../db/index.js";
+import { retrieve } from "./retrieval-engine.js";
+import { traverse } from "./graph-engine.js";
+function isCurrentlyValid(m, now) {
+    if (m.valid_from && new Date(m.valid_from) > now)
+        return false;
+    if (m.valid_until && new Date(m.valid_until) < now)
+        return false;
+    return true;
+}
+// Assemble relevant memories for the current task (spec §15)
+export function getContext(opts = {}) {
+    const query = opts.query ?? "";
+    const limit = Math.min(Math.max(opts.limit ?? 10, 1), 50);
+    const maxTokens = opts.maxTokens ?? 2000;
+    const now = new Date();
+    const seeds = retrieve(query, {
+        limit,
+        projectId: opts.projectId,
+        includeArchived: opts.includeHistory,
+    });
+    const seedScores = new Map();
+    for (const s of seeds)
+        seedScores.set(s.id, s.final_score);
+    const ids = new Set(seeds.map((s) => s.id));
+    if (opts.includeGraph) {
+        for (const s of seeds) {
+            for (const n of traverse(s.id, { maxDepth: 1 }))
+                ids.add(n.memoryId);
+        }
+    }
+    const all = [...ids]
+        .map((id) => db.prepare("SELECT * FROM memories WHERE id = ?").get(id))
+        .filter((m) => !!m)
+        .filter((m) => opts.includeHistory || isCurrentlyValid(m, now));
+    all.sort((a, b) => (seedScores.get(b.id) ?? 0) - (seedScores.get(a.id) ?? 0));
+    let used = 0;
+    const out = [];
+    let truncated = false;
+    for (const m of all) {
+        const t = Math.ceil((m.content?.length ?? 0) / 4) + 8;
+        if (used + t > maxTokens && out.length > 0) {
+            truncated = true;
+            break;
+        }
+        used += t;
+        out.push({ ...m, final_score: seedScores.get(m.id) ?? 0, viaGraph: !seedScores.has(m.id) });
+    }
+    return { query, memories: out, tokenEstimate: used, truncated };
+}

package/dist/core/graph-engine.js

@@ -0,0 +1,67 @@
+import { db, nowISO } from "../db/index.js";
+export function createEntity(input) {
+    const canonical = input.name.toLowerCase().trim();
+    const existing = db
+        .prepare("SELECT id FROM entities WHERE canonical_name = ?")
+        .get(canonical);
+    if (existing)
+        return existing.id;
+    const info = db
+        .prepare(`INSERT INTO entities (name, canonical_name, type, metadata)
+       VALUES (?, ?, ?, ?)`)
+        .run(input.name, canonical, input.type ?? "concept", JSON.stringify({ aliases: input.aliases ?? [] }));
+    return Number(info.lastInsertRowid);
+}
+export function addRelation(input) {
+    const info = db
+        .prepare(`INSERT INTO relations
+       (source_entity_id, relation, target_entity_id, confidence, source_memory_id, metadata)
+       VALUES (?, ?, ?, ?, ?, ?)`)
+        .run(input.subjectId, input.predicate, input.objectId, input.confidence ?? 0.5, input.sourceMemoryId ?? null, JSON.stringify({}));
+    return Number(info.lastInsertRowid);
+}
+export function linkMemories(sourceId, targetId, relation) {
+    db.prepare(`INSERT OR IGNORE INTO memory_links
+     (source_memory_id, relation, target_memory_id, created_at)
+     VALUES (?, ?, ?, ?)`).run(sourceId, relation, targetId, nowISO());
+}
+// Bounded BFS over memory_links starting from a memory (spec §14)
+export function traverse(startMemoryId, opts = {}) {
+    const maxDepth = Math.min(Math.max(opts.maxDepth ?? 2, 1), 5);
+    const visited = new Set([startMemoryId]);
+    const queue = [
+        { id: startMemoryId, depth: 0, relation: null },
+    ];
+    const result = [];
+    while (queue.length) {
+        const cur = queue.shift();
+        if (cur.depth >= maxDepth)
+            continue;
+        const links = db
+            .prepare(`SELECT target_memory_id, relation FROM memory_links
+         WHERE source_memory_id = ?`)
+            .all(cur.id);
+        for (const l of links) {
+            if (opts.relationFilter && !opts.relationFilter.includes(l.relation)) {
+                continue;
+            }
+            if (!visited.has(l.target_memory_id)) {
+                visited.add(l.target_memory_id);
+                result.push({
+                    memoryId: l.target_memory_id,
+                    depth: cur.depth + 1,
+                    relation: l.relation,
+                });
+                queue.push({
+                    id: l.target_memory_id,
+                    depth: cur.depth + 1,
+                    relation: l.relation,
+                });
+            }
+        }
+    }
+    return result;
+}
+export function neighbors(memoryId) {
+    return traverse(memoryId, { maxDepth: 1 });
+}