mcp-rag-memory 2.3.0

package/.env.example

@@ -0,0 +1,35 @@
+# Copy to `.env` and adjust. Picked up automatically by `npx mcp-rag-memory`,
+# `npm run dev`, and `npm run start` (see src/env.ts). Host environment
+# variables always override these values.
+
+# Where the SQLite store lives (default: <project>/.rag-data)
+RAG_DB_DIR=.rag-data
+
+# Embedding backend: "local" (zero-dep, offline) | "transformers" (ONNX, higher quality)
+EMBEDDING_PROVIDER=local
+# EMBEDDING_MODEL=Xenova/all-MiniLM-L6-v2
+
+# Search mode: "hybrid" | "vector" | "keyword"
+SEARCH_MODE=hybrid
+
+# Recency decay half-life for memory recall scores (days)
+RAG_MEMORY_HALF_LIFE_DAYS=14
+
+# Ingestion guards
+RAG_MAX_FILE_MB=10
+# RAG_ALLOWED_DIRS=C:\repo\a;C:\repo\b
+
+# Session auto-sync / condensing (opencode plugin reads these from its own env)
+RAG_AUTOSYNC=1
+RAG_SESSION_CONDENSE=1
+RAG_SESSION_CONDENSE_MIN_CHARS=120
+RAG_SESSION_CONDENSE_RATIO=0.35
+
+# Optional pruning during consolidation
+RAG_PRUNE=0
+RAG_PRUNE_IMPORTANCE=0.2
+RAG_PRUNE_AGE_DAYS=90
+
+# opencode transcript sources (override autodetect)
+# OPENCODE_DB=C:\Users\you\.local\share\opencode\opencode.db
+# OPENCODE_DB_LOCAL=C:\Users\you\.local\share\opencode\opencode-local.db

package/README.md

@@ -0,0 +1,475 @@
+# RAG + Context Management MCP Server
+
+<img src="assets/logo.jpg" alt="mcp-rag-memory" />
+
+Persistent long-term memory & knowledge retrieval for AI assistants.
+Exposes RAG (vector storage + semantic search) and context management
+(memories) as MCP tools, so opencode can store and recall
+context **across sessions**.
+
+> 👤 Identity: **The Coder** — "I'm The Coder, Selamat datang dan Semoga perjalanan mu menyenangkan"
+>
+> 📖 Project documentation site: https://adyoi.github.io/mcp-rag-memory/ (see [`docs/`](./docs/))
+
+## What it does
+
+- **RAG** — ingest documents (text / files / whole directories), chunk + embed locally, store in SQLite, and semantically retrieve relevant context.
+- **Hybrid search** — FTS5 BM25 keyword hits fused with vector similarity (reciprocal-rank fusion). Tunable `SEARCH_MODE=hybrid|vector|keyword`.
+- **Context Management** — `remember` facts/decisions/preferences, `recall` them later (score-decayed by staleness), rate by importance, consolidate duplicates, filter by type/tag.
+- **Zero external APIs by default** — local hashing-embedder (1024-dim), built-in `node:sqlite`. Works fully offline. Optional `EMBEDDING_PROVIDER=transformers` for a higher-quality ONNX model.
+- **Safe ingestion** — content-hash deduplication, file size guard (`RAG_MAX_FILE_MB`) and an opt-in path allowlist (`RAG_ALLOWED_DIRS`).
+- **MCP server** — runs on stdio, 19 tools.
+
+## Quick start
+
+```bash
+npm install
+npm test                 # 103 checks: unit + MCP round-trip via SDK client
+npm run lint             # oxlint
+npm run build            # compile to dist/
+npm run cli -- stats     # CLI playground
+```
+
+### Run from npm (npx)
+
+Install or run directly from the npm package — no `tsx`, no source checkout:
+
+```bash
+npx mcp-rag-memory
+```
+
+Point your MCP client at `npx mcp-rag-memory` (the published `bin`), or
+at the local dev entry: `node --import tsx src/mcp/rag-server.ts`.
+
+## Works with any MCP client
+
+This is a **standard MCP server** (stdio transport, official
+`@modelcontextprotocol/sdk`). It speaks the MCP spec, so **any** agent
+that supports MCP can use it — not just opencode. Memory & knowledge
+become **shared across all your tools**: remember once, recall everywhere.
+
+### opencode
+
+```json
+{
+  "mcp": {
+    "rag-memory": {
+      "type": "local",
+      "command": ["node", "--import", "tsx", "D:/Project/mcp-server/src/mcp/rag-server.ts"],
+      "cwd": "D:/Project/mcp-server",
+      "enabled": true,
+      "environment": { "RAG_DB_DIR": "D:/Project/mcp-server/.rag-data" }
+    }
+  }
+}
+```
+
+### Claude Desktop
+
+`claude_desktop_config.json` (in `%APPDATA%\Claude`):
+
+```json
+{
+  "mcpServers": {
+    "rag-memory": {
+      "command": "node",
+      "args": ["--import", "tsx", "D:/Project/mcp-server/src/mcp/rag-server.ts"],
+      "env": { "RAG_DB_DIR": "D:/Project/mcp-server/.rag-data" }
+    }
+  }
+}
+```
+
+### Cursor / Windsurf
+
+Settings → MCP → add server, same `command`/`args` pattern as Claude
+Desktop. No `cwd` needed — use absolute paths for the entry file and DB.
+
+### VS Code (Copilot)
+
+`mcp.json` in `.vscode/`:
+
+```json
+{
+  "servers": {
+    "rag-memory": {
+      "type": "stdio",
+      "command": "node",
+      "args": ["--import", "tsx", "D:/Project/mcp-server/src/mcp/rag-server.ts"],
+      "env": { "RAG_DB_DIR": "D:/Project/mcp-server/.rag-data" }
+    }
+  }
+}
+```
+
+> Tip: once published (or installed), just point every client at
+> `npx mcp-rag-memory` — no `tsx`, no absolute paths, no local checkout.
+> Locally you can still use `node --import tsx <abs-path>/src/mcp/rag-server.ts`.
+
+## opencode configuration
+
+### Global config (`~/.config/opencode/opencode.json`)
+
+The MCP server is registered here so it works in **every project**, not just this folder:
+
+```json
+{
+  "$schema": "https://opencode.ai/config.json",
+  "mcp": {
+    "rag-memory": {
+      "type": "local",
+      "command": ["node", "--import", "tsx", "src/mcp/rag-server.ts"],
+      "cwd": "D:/Project/mcp-server",
+      "enabled": true,
+      "environment": {
+        "RAG_DB_DIR": "D:/Project/mcp-server/.rag-data"
+      }
+    }
+  }
+}
+```
+
+> `cwd` and `RAG_DB_DIR` are absolute paths so the server resolves `tsx`
+> from this project's `node_modules` and stores data in the same DB
+> regardless of which project opencode is opened from.
+
+### Project config (`opencode.json` in this repo)
+
+Keeps project-specific settings (model, LSP, permissions, instructions) — **no `mcp` block here**:
+
+```json
+{
+  "$schema": "https://opencode.ai/config.json",
+  "model": "anthropic/claude-sonnet-4-6",
+  "lsp": true,
+  "instructions": [".opencode/instructions.md"],
+  "permission": {
+    "edit": "allow",
+    "bash": { "git *": "allow", "*": "ask" }
+  }
+}
+```
+
+`.opencode/instructions.md` tells every new session to call
+`memory_context(topic="The Coder")` first, so the assistant always knows
+who you are without asking. The same instructions file is also wired into
+the global config to apply to every project.
+
+### Notifications & sound (`~/.config/opencode/tui.json`)
+
+```json
+{
+  "$schema": "https://opencode.ai/tui.json",
+  "attention": {
+    "enabled": true,
+    "sound": true,
+    "notifications": true,
+    "volume": 0.4
+  }
+}
+```
+
+Restart opencode after any config change.
+
+## MCP tools (19)
+
+| Group | Tool | Purpose |
+|-------|------|---------|
+| System | `system_stats` | DB dir, embedding backend, doc/memory counts |
+| RAG ingest | `rag_ingest_text` | Store text as a knowledge document (dedup-aware) |
+| | `rag_ingest_file` | Store a file (`content_type` auto-detected) |
+| | `rag_ingest_dir` | Recursively ingest source files |
+| RAG query | `rag_search` | Hybrid (BM25 + vector) search, ranked chunks + scores |
+| | `rag_retrieve` | Ready-to-inject context block with token count |
+| RAG docs | `rag_list_documents`, `rag_document_stats` | Inventory |
+| | `rag_delete_document` | Remove a document + chunks + FTS rows |
+| Session sync | `rag_sync_session` | Ingest an opencode session's user inputs (or the latest in a workspace) |
+| Memory | `memory_remember` | Save long-term memory (type/importance/tags) |
+| | `memory_recall` | Semantic memory search (decay + recall count) |
+| | `memory_context` | Compact context block from memories for prompts |
+| | `memory_list`, `memory_get`, `memory_update`, `memory_forget` | CRUD |
+| | `memory_consolidate` | Dedupe near-identical + promote hot memories |
+| | `memory_stats` | Counts, tokens, avg importance, by type |
+
+## Run in other AI assistants
+
+The server is a standard MCP stdio server — it works with any MCP client
+(Claude Desktop, Cursor, Continue, VS Code, ...), not just opencode.
+
+**Via `npx` (after publishing).**
+
+```bash
+npx mcp-rag-memory
+```
+
+**Claude Desktop** — `claude_desktop_config.json`:
+
+```json
+{
+  "mcpServers": {
+    "rag-memory": {
+      "command": "npx",
+      "args": ["-y", "mcp-rag-memory"],
+      "env": { "RAG_DB_DIR": "C:\\Users\\you\\rag-data" }
+    }
+  }
+}
+```
+
+**Cursor** (`.cursor/mcp.json`) — `continue` style JSON, same shape:
+
+```json
+{
+  "mcpServers": {
+    "rag-memory": {
+      "command": "node",
+      "args": ["D:/Project/mcp-server/dist/index.js"],
+      "env": { "RAG_DB_DIR": "D:/Project/mcp-server/.rag-data" }
+    }
+  }
+}
+```
+
+**Continue** — `~/.continue/config.json`:
+
+```json
+{
+  "mcpServers": {
+    "rag-memory": {
+      "type": "stdio",
+      "command": "node",
+      "args": ["D:/Project/mcp-server/dist/index.js"]
+    }
+  }
+}
+```
+
+**Configuration.** Every `RAG_*` variable can come from three places, with the
+first winning: the MCP client's `env` block → your shell environment → a
+`.env` file in the working directory. Copy [`.env.example`](./.env.example) to
+`.env` and edit, or point at any file via `RAG_ENV_FILE`. The store defaults to
+`.rag-data/` in the working directory.
+
+## Custom slash commands
+
+The 19 MCP tools are called by the AI automatically — but you can also
+trigger them directly with custom commands. Sources (same name, project
+wins):
+
+- Per-project: `.opencode/commands/`
+- Global: `~/.config/opencode/commands/`
+
+| Command | Backing tool | Use |
+|---------|--------------|-----|
+| `/remember <content> [--type][--importance]` | `memory_remember` | Save a memory |
+| `/recall <topic>` | `memory_recall` | Search memories semantically |
+| `/context <topic>` | `memory_context` | Context block for prompts |
+| `/consolidate` | `memory_consolidate` | Dedupe + promote hot memories |
+| `/search <query>` | `rag_search` | Semantic search documents |
+| `/retrieve <query>` | `rag_retrieve` | Raw context block + tokens |
+| `/ingest <text>` | `rag_ingest_text` | Store knowledge text |
+| `/docs` | `rag_list_documents` | List all documents |
+| `/stats` | `system_stats` + doc/memory stats | Full statistics |
+
+## CLI
+
+```bash
+npm run cli -- remember "deploys every Friday" --type task --importance 0.7
+npm run cli -- recall "deployment schedule"
+npm run cli -- ingest-dir ./src/rag
+npm run cli -- search "vector similarity"
+npm run cli -- mem-context "database"
+npm run cli -- sessions              # list recent opencode sessions
+npm run cli -- sync-session ses_123  # ingest one session's inputs
+npm run cli -- sync-latest           # ingest the most recent session
+npm run clean-db -- --force          # wipe ALL documents + memories (destructive)
+```
+
+Semua akses juga tersedia sebagai custom commands (`/remember`, `/recall`,
+`/search`, ...) dan sebagai MCP tools.
+
+## Auto-save session inputs
+
+Every user input you type in opencode becomes searchable in the RAG store —
+new, untouched, friendly to your existing long-term memories. Three layers:
+
+1. **Plugin (real-time, debounced).** `.opencode/plugin/session-logger.ts`
+   watches message events and, at most once a minute, runs
+   `sync-latest` so the session you are typing in lands in the store.
+   Disable with `RAG_AUTOSYNC=0`.
+2. **CLI on demand.**
+   ```bash
+   npm run sync-latest -- --dir "D:/Project/my-app"   # latest session in a workspace
+   npm run sync-session -- ses_abc123                  # a specific session
+   npm run sync-logs                                    # .session-logs/*.jsonl (custom logs)
+   ```
+3. **MCP tool.** `rag_sync_session` exposes the same logic over MCP:
+   `{ "session_id": "..." }` or `{ "directory": "..." }` for the latest.
+
+How it works: sessions are read from opencode's own DB
+(`~/.local/share/opencode/opencode.db` + `opencode-local.db`, override with
+`OPENCODE_DB` / `OPENCODE_DB_LOCAL`), each user message becomes one small
+document (title `<session>-<ts>`, metadata `source: "opencode-db"`). Long
+inputs are **condensed to key points** before storage (extractive, no LLM) so
+the store stays lean; short inputs are saved whole. Raw transcripts stay in
+opencode's DB; long-term **memories** are never mixed in. Re-runs are
+idempotent thanks to SHA-256 content-hash dedup (`metadata.condensed` marks
+reduced docs), so plugging this into any scheduler is safe.
+
+The plugin needs an opencode restart to take effect.
+
+## Architecture
+
+```
+src/
+├── db/database.ts          node:sqlite (documents, chunks, memories, chunks_fts, meta)
+├── rag/
+│   ├── embedder.ts         local hashing embedder (1024-d, FNV-1a, n-grams)
+│   ├── embeddings.ts       provider layer: local | transformers (optional dep)
+│   ├── chunker.ts          paragraph/code-aware chunking with overlap
+│   ├── vector-search.ts    vector cache + FTS5 BM25 + RRF hybrid scoring
+│   └── pipeline.ts         async ingest/search/retrieve, dedup, guards, doc mgmt
+├── session/transcript.ts   read opencode sessions (global+local DB), ingest inputs
+├── memory/memory.ts        remember / recall / consolidate + decay + prune
+├── mcp/rag-server.ts       MCP server (19 tools, stdio, npm bin)
+├── cli.ts                  CLI playground (incl. sync-session / sync-latest / sync-logs)
+└── test/test-all.ts        full test suite (unit + MCP round-trip)
+```
+
+Data lives in `.rag-data/rag.sqlite` (git-ignored).
+
+## Configuration (env vars)
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `RAG_DB_DIR` | `.rag-data` | Where the SQLite store lives |
+| `SEARCH_MODE` | `hybrid` | `hybrid` \| `vector` \| `keyword` |
+| `EMBEDDING_PROVIDER` | `local` | `local` (zero-dep) \| `transformers` (needs optional `@huggingface/transformers`) |
+| `EMBEDDING_MODEL` | `Xenova/all-MiniLM-L6-v2` | Transformers model name |
+| `RAG_MAX_FILE_MB` | `10` | Reject files larger than this |
+| `RAG_ALLOWED_DIRS` | *(unset = anywhere)* | Semicolon/pipe/comma-separated allowed ingest roots |
+| `RAG_MEMORY_HALF_LIFE_DAYS` | `14` | Recall score decay half-life |
+| `RAG_PRUNE` | `0` | Set `1` to allow `consolidate` to delete non-essential memories |
+| `RAG_PRUNE_IMPORTANCE` | `0.2` | Delete memories below this importance |
+| `RAG_PRUNE_AGE_DAYS` | `90` | ...and older than this (never-recalled only) |
+| `RAG_AUTOSYNC` | `1` | Set `0` to disable the session-logger plugin's auto-sync |
+| `OPENCODE_DB` / `OPENCODE_DB_LOCAL` | `~/.local/share/opencode/*.db` | Where session transcripts are read from |
+| `RAG_SESSION_CONDENSE` | `1` | Set `0` to store session inputs verbatim |
+| `RAG_SESSION_CONDENSE_MIN_CHARS` | `120` | Inputs at/below this length are saved whole |
+| `RAG_SESSION_CONDENSE_RATIO` | `0.35` | Fraction of long-input length to keep as key points |
+
+## Testing
+
+`npm test` spins up the real MCP server over stdio using the SDK client
+and exercises every tool end-to-end against a scratch DB (`.test-data`).
+`npm run lint` checks the codebase with oxlint (run in CI too).
+
+## GitHub Pages site
+
+The [`docs/`](./docs/) directory is a self-contained static site of this
+project (single `index.html`, no build step). To publish it on GitHub
+Pages:
+
+1. Push this repo to GitHub.
+2. Repo **Settings → Pages → Build and deployment → Source: Deploy from a
+   branch**.
+3. Choose branch `main` and folder `/docs`.
+4. Your site is live at `https://adyoi.github.io/mcp-rag-memory/`.
+
+## Troubleshooting
+
+### `ConfigInvalidError: missing key "command"` for typescript / javascript / json
+
+The `lsp` block in `opencode.json` has the wrong shape.
+Each language key must have a `command` array:
+
+```json
+"lsp": {
+  "typescript": {
+    "command": ["typescript-language-server", "--stdio"]
+  }
+}
+```
+
+Fields like `language_id` or `extensions` alone are not allowed — the
+schema enforces `additionalProperties: false` and requires `command`.
+
+**Fix:** either write the `command` array, or (if you just want built-in
+LSP) replace the whole block with `"lsp": true`.
+
+---
+
+### MCP tools not showing up in opencode
+
+1. **Wrong config filename** — opencode reads only `opencode.json`,
+   `opencode.jsonc`, or `.opencode/opencode.json`. A file named
+   `opencode.jsonx` is **silently ignored**; no error, no tools.
+2. **File not in the right place** — global MCP config lives at
+   `~/.config/opencode/opencode.json`, not in the project folder.
+3. **Missing `cwd` for global use** — when `command` uses a relative
+   path (`src/mcp/rag-server.ts`), opencode resolves it against the
+   *workspace* directory, not the project. Add `"cwd"` to point at the
+   project that contains `node_modules/tsx`:
+
+```json
+"rag-memory": {
+  "type": "local",
+  "command": ["node", "--import", "tsx", "src/mcp/rag-server.ts"],
+  "cwd": "D:/Project/mcp-server"
+}
+```
+
+4. **Missing env var** — `RAG_DB_DIR` must be set (absolute path for
+   global config) or the DB defaults to `.rag-data` relative to cwd.
+
+---
+
+### Sound / notification not playing when response finishes
+
+The `attention` feature is **off by default**. Create
+`~/.config/opencode/tui.json`:
+
+```json
+{
+  "$schema": "https://opencode.ai/tui.json",
+  "attention": {
+    "enabled": true,
+    "sound": true,
+    "notifications": true,
+    "volume": 0.4
+  }
+}
+```
+
+**Known issue** ([#40445](https://github.com/anomalyco/opencode/issues/40445)):
+sound silently fails when opencode runs under the **Node runtime** instead
+of Bun — the audio library depends on Bun FFI. Symptoms: no sound despite
+correct config. Not a configuration error.
+
+---
+
+### Server slow to start on first load (~3 s)
+
+`tsx` compiles TypeScript on first invocation. This is normal and only
+happens on cold start; subsequent tool calls within the same session are
+instant.
+
+---
+
+### Opencode feels sluggish / UI lag after adding plugins
+
+Plugins that run synchronous I/O or write to stderr on every tool call
+can slow down the TUI. This project no longer ships any plugins — keep
+`opencode.json` plugin-free for clean operation.
+
+---
+
+### `ajv-cli` fails with `strict mode: unknown keyword: allowComments`
+
+The opencode schema uses custom JSON-Schema extensions (`allowComments`,
+`allowTrailingCommas`). `ajv-cli` rejects these by default. Validate
+config manually instead:
+
+```bash
+node -e "JSON.parse(require('fs').readFileSync('opencode.json','utf8')); console.log('OK')"
+```

package/dist/db/database.d.ts

@@ -0,0 +1,22 @@
+import { DatabaseSync } from "node:sqlite";
+export declare const STORAGE_DIR: string;
+export declare const DB_PATH: string;
+export declare function getDB(): DatabaseSync;
+/** SHA-256 hex of raw content — used for document dedup. */
+export declare function contentHash(content: string): string;
+export declare function newId(): string;
+/** Serialize a Float64Array embedding to a SQLite BLOB. */
+export declare function packVector(v: Float64Array): Uint8Array;
+/** Deserialize a BLOB back into a Float64Array. */
+export declare function unpackVector(blob: Uint8Array | null): Float64Array | null;
+export declare function nowMs(): number;
+export declare function closeDB(): void;
+export declare function getEmbedDim(): number;
+export declare function setEmbedMeta(dim: number, model: string): void;
+/**
+ * Lock the store to one embedding dimension. Call after producing a vector.
+ * Prevents silently mixing vectors of different models/providers.
+ */
+export declare function ensureDim(vec: Float64Array, model: string): void;
+export declare function getEmbedModel(): string;
+//# sourceMappingURL=database.d.ts.map

package/dist/db/database.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"database.d.ts","sourceRoot":"","sources":["../../src/db/database.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAK3C,eAAO,MAAM,WAAW,QAEoB,CAAC;AAE7C,eAAO,MAAM,OAAO,QAAuC,CAAC;AAU5D,wBAAgB,KAAK,IAAI,YAAY,CAUpC;AAqED,4DAA4D;AAC5D,wBAAgB,WAAW,CAAC,OAAO,EAAE,MAAM,GAAG,MAAM,CAEnD;AAED,wBAAgB,KAAK,IAAI,MAAM,CAE9B;AAED,2DAA2D;AAC3D,wBAAgB,UAAU,CAAC,CAAC,EAAE,YAAY,GAAG,UAAU,CAEtD;AAED,mDAAmD;AACnD,wBAAgB,YAAY,CAAC,IAAI,EAAE,UAAU,GAAG,IAAI,GAAG,YAAY,GAAG,IAAI,CAGzE;AAED,wBAAgB,KAAK,IAAI,MAAM,CAE9B;AAED,wBAAgB,OAAO,SAGtB;AAMD,wBAAgB,WAAW,IAAI,MAAM,CAKpC;AAED,wBAAgB,YAAY,CAAC,GAAG,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,QAQtD;AAED;;;GAGG;AACH,wBAAgB,SAAS,CAAC,GAAG,EAAE,YAAY,EAAE,KAAK,EAAE,MAAM,GAAG,IAAI,CAahE;AAED,wBAAgB,aAAa,IAAI,MAAM,CAKtC"}

package/dist/db/database.js

@@ -0,0 +1,144 @@
+import { DatabaseSync } from "node:sqlite";
+import * as path from "path";
+import * as fs from "fs";
+import { createHash, randomUUID } from "node:crypto";
+export const STORAGE_DIR = process.env.RAG_DB_DIR
+    ? path.resolve(process.env.RAG_DB_DIR)
+    : path.resolve(process.cwd(), ".rag-data");
+export const DB_PATH = path.join(STORAGE_DIR, "rag.sqlite");
+let _db = null;
+function ensureDir() {
+    if (!fs.existsSync(STORAGE_DIR)) {
+        fs.mkdirSync(STORAGE_DIR, { recursive: true });
+    }
+}
+export function getDB() {
+    if (_db)
+        return _db;
+    ensureDir();
+    const db = new DatabaseSync(DB_PATH);
+    db.exec("PRAGMA journal_mode = WAL;");
+    db.exec("PRAGMA synchronous = NORMAL;");
+    db.exec("PRAGMA foreign_keys = ON;");
+    _db = db;
+    migrate(db);
+    return db;
+}
+function columnExists(db, table, column) {
+    const cols = db.prepare(`PRAGMA table_info(${table})`).all();
+    return cols.some((c) => c.name === column);
+}
+function migrate(db) {
+    db.exec(`
+    CREATE TABLE IF NOT EXISTS documents (
+      id           TEXT PRIMARY KEY,
+      title        TEXT NOT NULL,
+      source       TEXT,
+      content_type TEXT DEFAULT 'text',
+      metadata     TEXT DEFAULT '{}',
+      chunk_count  INTEGER DEFAULT 0,
+      created_at   INTEGER NOT NULL,
+      updated_at   INTEGER NOT NULL
+    );
+
+    CREATE TABLE IF NOT EXISTS chunks (
+      id        TEXT PRIMARY KEY,
+      doc_id    TEXT NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
+      idx       INTEGER NOT NULL,
+      content   TEXT NOT NULL,
+      embedding BLOB,
+      token_count INTEGER DEFAULT 0,
+      created_at INTEGER NOT NULL
+    );
+    CREATE INDEX IF NOT EXISTS idx_chunks_doc_id ON chunks(doc_id);
+
+    CREATE TABLE IF NOT EXISTS memories (
+      id           TEXT PRIMARY KEY,
+      type         TEXT DEFAULT 'fact',
+      content      TEXT NOT NULL,
+      importance   REAL DEFAULT 0.5,
+      embedding    BLOB,
+      tags         TEXT DEFAULT '[]',
+      recall_count INTEGER DEFAULT 0,
+      last_recalled INTEGER,
+      created_at   INTEGER NOT NULL,
+      updated_at   INTEGER NOT NULL
+    );
+    CREATE INDEX IF NOT EXISTS idx_memories_type ON memories(type);
+    CREATE INDEX IF NOT EXISTS idx_memories_prune ON memories(importance, recall_count, created_at);
+
+    CREATE TABLE IF NOT EXISTS meta (
+      key   TEXT PRIMARY KEY,
+      value TEXT
+    );
+  `);
+    // Content-hash dedup for documents (unique partial index — NULLs allowed).
+    if (!columnExists(db, "documents", "content_hash")) {
+        db.exec("ALTER TABLE documents ADD COLUMN content_hash TEXT;");
+    }
+    db.exec("CREATE UNIQUE INDEX IF NOT EXISTS idx_documents_content_hash ON documents(content_hash) WHERE content_hash IS NOT NULL;");
+    // FTS5 full-text index over chunks for hybrid (BM25 + vector) search.
+    const hasFts = db
+        .prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'chunks_fts'")
+        .get();
+    if (!hasFts) {
+        db.exec("CREATE VIRTUAL TABLE chunks_fts USING fts5(content, tokenize = 'unicode61');");
+    }
+}
+/** SHA-256 hex of raw content — used for document dedup. */
+export function contentHash(content) {
+    return createHash("sha256").update(content, "utf8").digest("hex");
+}
+export function newId() {
+    return randomUUID();
+}
+/** Serialize a Float64Array embedding to a SQLite BLOB. */
+export function packVector(v) {
+    return new Uint8Array(v.buffer.slice(0));
+}
+/** Deserialize a BLOB back into a Float64Array. */
+export function unpackVector(blob) {
+    if (!blob || blob.byteLength === 0)
+        return null;
+    return new Float64Array(blob.buffer.slice(blob.byteOffset, blob.byteOffset + blob.byteLength));
+}
+export function nowMs() {
+    return Date.now();
+}
+export function closeDB() {
+    _db?.close();
+    _db = null;
+}
+/* ------------------------------------------------------------------ */
+/* Embedding dimension metadata                                        */
+/* ------------------------------------------------------------------ */
+export function getEmbedDim() {
+    const r = getDB().prepare("SELECT value FROM meta WHERE key = 'embed_dim'").get();
+    return r ? Number(r.value) : 0;
+}
+export function setEmbedMeta(dim, model) {
+    const db = getDB();
+    db.prepare("INSERT INTO meta(key, value) VALUES ('embed_dim', ?) ON CONFLICT(key) DO UPDATE SET value = excluded.value").run(String(dim));
+    db.prepare("INSERT INTO meta(key, value) VALUES ('embed_model', ?) ON CONFLICT(key) DO UPDATE SET value = excluded.value").run(model);
+}
+/**
+ * Lock the store to one embedding dimension. Call after producing a vector.
+ * Prevents silently mixing vectors of different models/providers.
+ */
+export function ensureDim(vec, model) {
+    const current = getEmbedDim();
+    if (current === 0) {
+        setEmbedMeta(vec.length, model);
+        return;
+    }
+    if (current !== vec.length) {
+        throw new Error(`Embedding dimension mismatch: DB stores ${current} (model: ${getEmbedModel()}), ` +
+            `new vectors are ${vec.length} (model: ${model}). ` +
+            `Use a consistent EMBEDDING_PROVIDER or wipe ${STORAGE_DIR}.`);
+    }
+}
+export function getEmbedModel() {
+    const r = getDB().prepare("SELECT value FROM meta WHERE key = 'embed_model'").get();
+    return r ? r.value : "";
+}
+//# sourceMappingURL=database.js.map

package/dist/db/database.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"database.js","sourceRoot":"","sources":["../../src/db/database.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAC3C,OAAO,KAAK,IAAI,MAAM,MAAM,CAAC;AAC7B,OAAO,KAAK,EAAE,MAAM,IAAI,CAAC;AACzB,OAAO,EAAE,UAAU,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAErD,MAAM,CAAC,MAAM,WAAW,GAAG,OAAO,CAAC,GAAG,CAAC,UAAU;IAC/C,CAAC,CAAC,IAAI,CAAC,OAAO,CAAC,OAAO,CAAC,GAAG,CAAC,UAAU,CAAC;IACtC,CAAC,CAAC,IAAI,CAAC,OAAO,CAAC,OAAO,CAAC,GAAG,EAAE,EAAE,WAAW,CAAC,CAAC;AAE7C,MAAM,CAAC,MAAM,OAAO,GAAG,IAAI,CAAC,IAAI,CAAC,WAAW,EAAE,YAAY,CAAC,CAAC;AAE5D,IAAI,GAAG,GAAwB,IAAI,CAAC;AAEpC,SAAS,SAAS;IAChB,IAAI,CAAC,EAAE,CAAC,UAAU,CAAC,WAAW,CAAC,EAAE,CAAC;QAChC,EAAE,CAAC,SAAS,CAAC,WAAW,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;IACjD,CAAC;AACH,CAAC;AAED,MAAM,UAAU,KAAK;IACnB,IAAI,GAAG;QAAE,OAAO,GAAG,CAAC;IACpB,SAAS,EAAE,CAAC;IACZ,MAAM,EAAE,GAAG,IAAI,YAAY,CAAC,OAAO,CAAC,CAAC;IACrC,EAAE,CAAC,IAAI,CAAC,4BAA4B,CAAC,CAAC;IACtC,EAAE,CAAC,IAAI,CAAC,8BAA8B,CAAC,CAAC;IACxC,EAAE,CAAC,IAAI,CAAC,2BAA2B,CAAC,CAAC;IACrC,GAAG,GAAG,EAAE,CAAC;IACT,OAAO,CAAC,EAAE,CAAC,CAAC;IACZ,OAAO,EAAE,CAAC;AACZ,CAAC;AAED,SAAS,YAAY,CAAC,EAAgB,EAAE,KAAa,EAAE,MAAc;IACnE,MAAM,IAAI,GAAG,EAAE,CAAC,OAAO,CAAC,qBAAqB,KAAK,GAAG,CAAC,CAAC,GAAG,EAA6B,CAAC;IACxF,OAAO,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,MAAM,CAAC,CAAC;AAC7C,CAAC;AAED,SAAS,OAAO,CAAC,EAAgB;IAC/B,EAAE,CAAC,IAAI,CAAC;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA0CP,CAAC,CAAC;IAEH,2EAA2E;IAC3E,IAAI,CAAC,YAAY,CAAC,EAAE,EAAE,WAAW,EAAE,cAAc,CAAC,EAAE,CAAC;QACnD,EAAE,CAAC,IAAI,CAAC,qDAAqD,CAAC,CAAC;IACjE,CAAC;IACD,EAAE,CAAC,IAAI,CACL,yHAAyH,CAC1H,CAAC;IAEF,sEAAsE;IACtE,MAAM,MAAM,GAAG,EAAE;SACd,OAAO,CAAC,6EAA6E,CAAC;SACtF,GAAG,EAAkC,CAAC;IACzC,IAAI,CAAC,MAAM,EAAE,CAAC;QACZ,EAAE,CAAC,IAAI,CAAC,8EAA8E,CAAC,CAAC;IAC1F,CAAC;AACH,CAAC;AAED,4DAA4D;AAC5D,MAAM,UAAU,WAAW,CAAC,OAAe;IACzC,OAAO,UAAU,CAAC,QAAQ,CAAC,CAAC,MAAM,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;AACpE,CAAC;AAED,MAAM,UAAU,KAAK;IACnB,OAAO,UAAU,EAAE,CAAC;AACtB,CAAC;AAED,2DAA2D;AAC3D,MAAM,UAAU,UAAU,CAAC,CAAe;IACxC,OAAO,IAAI,UAAU,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;AAC3C,CAAC;AAED,mDAAmD;AACnD,MAAM,UAAU,YAAY,CAAC,IAAuB;IAClD,IAAI,CAAC,IAAI,IAAI,IAAI,CAAC,UAAU,KAAK,CAAC;QAAE,OAAO,IAAI,CAAC;IAChD,OAAO,IAAI,YAAY,CAAC,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,UAAU,EAAE,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC;AACjG,CAAC;AAED,MAAM,UAAU,KAAK;IACnB,OAAO,IAAI,CAAC,GAAG,EAAE,CAAC;AACpB,CAAC;AAED,MAAM,UAAU,OAAO;IACrB,GAAG,EAAE,KAAK,EAAE,CAAC;IACb,GAAG,GAAG,IAAI,CAAC;AACb,CAAC;AAED,wEAAwE;AACxE,yEAAyE;AACzE,wEAAwE;AAExE,MAAM,UAAU,WAAW;IACzB,MAAM,CAAC,GAAG,KAAK,EAAE,CAAC,OAAO,CAAC,gDAAgD,CAAC,CAAC,GAAG,EAElE,CAAC;IACd,OAAO,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;AACjC,CAAC;AAED,MAAM,UAAU,YAAY,CAAC,GAAW,EAAE,KAAa;IACrD,MAAM,EAAE,GAAG,KAAK,EAAE,CAAC;IACnB,EAAE,CAAC,OAAO,CACR,4GAA4G,CAC7G,CAAC,GAAG,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC;IACnB,EAAE,CAAC,OAAO,CACR,8GAA8G,CAC/G,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC;AACf,CAAC;AAED;;;GAGG;AACH,MAAM,UAAU,SAAS,CAAC,GAAiB,EAAE,KAAa;IACxD,MAAM,OAAO,GAAG,WAAW,EAAE,CAAC;IAC9B,IAAI,OAAO,KAAK,CAAC,EAAE,CAAC;QAClB,YAAY,CAAC,GAAG,CAAC,MAAM,EAAE,KAAK,CAAC,CAAC;QAChC,OAAO;IACT,CAAC;IACD,IAAI,OAAO,KAAK,GAAG,CAAC,MAAM,EAAE,CAAC;QAC3B,MAAM,IAAI,KAAK,CACb,2CAA2C,OAAO,YAAY,aAAa,EAAE,KAAK;YAChF,mBAAmB,GAAG,CAAC,MAAM,YAAY,KAAK,KAAK;YACnD,+CAA+C,WAAW,GAAG,CAChE,CAAC;IACJ,CAAC;AACH,CAAC;AAED,MAAM,UAAU,aAAa;IAC3B,MAAM,CAAC,GAAG,KAAK,EAAE,CAAC,OAAO,CAAC,kDAAkD,CAAC,CAAC,GAAG,EAEpE,CAAC;IACd,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,CAAC;AAC1B,CAAC"}