rag-memory-epf-mcp 6.1.0 → 6.3.1

package/README.md

@@ -16,7 +16,7 @@ A **project-local RAG memory** MCP server — knowledge graph + multilingual vec
 - **38 MCP tools** — knowledge graph CRUD, observation lifecycle (correct / retract / history), document pipeline, hybrid search, multi-hop traversal, graph analytics (centrality / community detection / structure), export/import, temporal queries
 - **Observations that hold their history** — corrections supersede instead of overwrite, search returns only current facts, and every revision keeps its provenance
 - **Structure-anchored chunking (c1, v5)** — boundaries anchor to markdown structure (fence-aware, H1–H4 first, block-greedy, exact-token fallback), so editing the top of a file no longer re-embeds the whole document: unchanged text reuses its stored vectors at sync time. Chunk offsets are Unicode codepoints, language-neutral across SQL `substr`, Python slicing, and JS `[...str]` iteration; a publish-time invariant gate locks the gap-free partition. `overlap` is retired (omit or 0).
-- **SQLite optimized** — WAL mode, 32MB cache, 256MB mmap, FTS5 triggers, 7 indexes
+- **SQLite optimized** — WAL mode, 32MB cache, FTS5 triggers, 7 indexes; mmap is opt-in (`RAG_MEMORY_MMAP_SIZE`, default 0 since 6.3.0 — see Environment Variables)
 - **MCP SDK 1.27.1** — Tool Annotations (readOnly/destructive/idempotent), latest protocol 2025-11-25
 
 ## Quick Start
@@ -145,7 +145,8 @@ storeDocument(id, content, metadata)
 
 | Variable | Default | Description |
 |----------|---------|-------------|
-| `DB_FILE_PATH` | `rag-memory.db` (server dir) | Path to project-local SQLite database |
+| `DB_FILE_PATH` | `rag-memory.db` (server dir) | Path to project-local SQLite database. A **relative** path resolves against the server's working directory (since 6.3.0; before that it resolved against the package install directory, which under `npx` is the npm cache) |
+| `RAG_MEMORY_MMAP_SIZE` | `0` | SQLite `mmap_size` in bytes. **Off by default since 6.3.0** — with mmap on, one writer + two readers on a Google Drive folder (Windows) produced `database disk image is malformed` reads; set e.g. `268435456` to restore the previous behaviour. The applied value is printed in the boot banner (`| mmap <n>`) |
 | `EMBEDDING_MODEL` | `Xenova/bge-m3` | HuggingFace model ID for embeddings |
 | `RAG_MEMORY_EMBEDDINGS` | `lazy` | Boot mode: `lazy` (connect instantly, model loads in background), `eager` (wait for model + reconciliation, pre-3.6 behavior), `off` (never load the model — FTS5-only, zero download) |
 | `RAG_MEMORY_MODEL_CACHE_DIR` | OS user cache | Version-independent model cache location (see `docs/UPDATING.md`) |
@@ -153,6 +154,60 @@ storeDocument(id, content, metadata)
 
 ## Changelog
 
+### v6.3.1
+
+- **Changed — the engine empties the WAL while it is idle.** It runs `wal_checkpoint(TRUNCATE)` itself:
+  on a 1 s tick whenever the `-wal` file has bytes in it, ~200 ms after a tool call, first thing in the
+  signal handler, and before closing. SQLite folds the WAL only when the *last* connection closes and
+  only if that close gets to run; with several engines on one file, or a host that kills the engine
+  (measured: codex-cli 0.155.1 sends SIGTERM and SIGKILLs ~185 ms later; Windows terminates children
+  outright), frames stayed in the WAL — on 6.3.0 a fresh database sat at 4 KB main + 663 KB WAL for as
+  long as the engine ran. In a cloud-synced folder such a WAL can be replayed onto a main file written
+  by another machine. A write made in the last ~1 s before a hard kill can still be in the WAL; a 0-byte
+  `-wal` and a `-shm` left behind by a hard kill are harmless. Running engines on two machines against
+  the same synced file at the same time is still unsafe.
+- **Changed — SIGHUP and SIGBREAK shut the engine down cleanly.** Closing the terminal or multiplexer
+  pane used to kill the process by default signal action, before the database was closed.
+- No tool, argument, return shape or schema changed.
+
+### v6.3.0 (2026-09-16)
+
+- **Changed — a relative `DB_FILE_PATH` now resolves against the server's working directory.** It used
+  to resolve against the package install directory, which under `npx` is the npm cache: a relative
+  value opened a database in a place nobody looks, with no error. Absolute paths and the unset default
+  are unchanged. (Framework spec 2026-09-14 §11-1.) If you relied on the old install-directory
+  resolution of a relative path, set an absolute path.
+- **Changed — SQLite `mmap_size` is opt-in, default 0.** Measured 2026-09-15 on a Google Drive folder
+  (Windows, WAL): with mmap 256 MB, one writer + two readers produced 373,684 `database disk image is
+  malformed` reads in 20 s; with mmap 0, none. `RAG_MEMORY_MMAP_SIZE=<bytes>` restores it; the boot
+  banner reports the applied value. (Spec §11-2.)
+
+### v6.1.0
+
+**Versioning note.** This content was tagged `v6.0.2` locally before release, but npm never received
+a 6.0.2 — it published as **6.1.0**. Coming from any 6.0.x, upgrading to 6.1.0 brings exactly this
+fix.
+
+- **Fixed — a failure mid-deletion could leave partial state.** Deleting an entity ran its cleanup
+  steps (embeddings → chunk associations → relationships → the entity row) with no wrapping
+  transaction, so a failure in the middle committed the earlier steps: embeddings and links purged
+  while the entity itself survived. Each entity's deletion is now one transaction — any step fails,
+  that entity's whole sequence rolls back. Batch semantics are unchanged: other entities still
+  proceed when one fails.
+- **Fixed — deleting an entity left its knowledge-graph chunks behind.** Two gaps compounded: the
+  delete path never invoked the entity-chunk sweeper, and KG relationship chunks are keyed by
+  `relationship_id`, so once the relationship rows were deleted their ids could no longer be found
+  and those chunks dangled forever — still vector-searchable after both of their endpoints were
+  gone. Deletion now captures relationship ids *before* removing the rows and sweeps both: stale
+  entity chunks via the existing `deleteStaleKgChunks`, captured relationship chunks via the new
+  `deleteKgRelationshipChunks` projection helper. Practical exposure today is bounded — the chunk
+  generation path is dormant (not tool-exposed) — but once seeded these chunks stay retrievable
+  unless swept here.
+- Regression: `test/delete-entities-kg-hygiene.test.mjs` (registered in `verify:engine`) covers both,
+  verified RED before the fix; a mutation check confirms that removing the transaction reproduces the
+  partial state (`entities_alive=1` with `relationships_left=0`). Full suite green (`npm test`,
+  46-call chain).
+
 ### v6.0.1
 
 **Versioning note.** Both changes below stop links from being created that should never have been

package/dist/index.d.ts

@@ -4,6 +4,8 @@ import { EmbeddingGate } from './src/embeddingGate.js';
 import type { EmbedPriority } from './src/embeddingGate.js';
 import { BackfillCoordinator } from './src/backfillCoordinator.js';
 export declare function compileFtsLiteralQuery(q: string): string | null;
+export declare function resolveDbFilePath(envValue: string | undefined, cwd: string, serverDir: string): string;
+export declare function parseMmapSize(raw: string | undefined): number;
 interface Entity {
     name: string;
     entityType: string;
@@ -71,6 +73,7 @@ export declare class RAGKnowledgeGraphManager {
     grandfatherAllowed: boolean;
     coordinator: BackfillCoordinator | null;
     readonly calendarTimeZone: string;
+    mmapApplied: number;
     private embeddingCache;
     private readonly EMBEDDING_CACHE_MAX;
     private dictionaryCache;
@@ -98,6 +101,11 @@ export declare class RAGKnowledgeGraphManager {
             description: string;
         }>;
     }>;
+    checkpointWal(): boolean;
+    private walTimer;
+    private walSoon;
+    startWalKeeper(tickMs?: number): void;
+    noteActivity(delayMs?: number): void;
     cleanup(): void;
     private _timestampObservation;
     createEntities(entities: Array<Entity & {

package/dist/index.js

@@ -59,7 +59,7 @@ function sanitizeErrorMessage(msg) {
 }
 // v3.6: startup self-report banner (version reliability — spec §8).
 function printBanner(opts) {
-    console.error(`🚀 rag-memory-epf-mcp v${PKG_VERSION} | node v${process.versions.node} | model ${opts.model}@${opts.revision} (${opts.dtype}) | cache ${opts.cachePath} | db ${opts.dbPath}`);
+    console.error(`🚀 rag-memory-epf-mcp v${PKG_VERSION} | node v${process.versions.node} | model ${opts.model}@${opts.revision} (${opts.dtype}) | cache ${opts.cachePath} | db ${opts.dbPath} | mmap ${opts.mmap}`);
 }
 // v3.6 (spec §5): ONE FTS5 literal-query compiler shared by chunk and entity
 // search — raw user input can never produce MATCH syntax errors or trigger
@@ -79,13 +79,36 @@ export function compileFtsLiteralQuery(q) {
 if (env.backends?.onnx?.wasm) {
     env.backends.onnx.wasm.wasmPaths = './node_modules/@huggingface/transformers/dist/';
 }
-// Define database file path using environment variable with fallback
-const defaultDbPath = path.join(path.dirname(fileURLToPath(import.meta.url)), 'rag-memory.db');
-const DB_FILE_PATH = process.env.DB_FILE_PATH
-    ? path.isAbsolute(process.env.DB_FILE_PATH)
-        ? process.env.DB_FILE_PATH
-        : path.join(path.dirname(fileURLToPath(import.meta.url)), process.env.DB_FILE_PATH)
-    : defaultDbPath;
+// Define database file path using environment variable with fallback.
+// Next release (major candidate; the version is set at release time — framework spec
+// 2026-09-14-universal-mcp-config-design §11-1): a RELATIVE DB_FILE_PATH
+// resolves against the server's working directory (process.cwd()), the way every other CLI tool
+// reads a relative path. Before this it resolved against the package's own install directory —
+// under npx that is the npm cache, so a relative value silently opened a database nobody could
+// find (measured 2026-09-14 on 6.1.0 dist/index.js:83-88) and every framework config had to carry
+// a machine-specific absolute path. The UNSET default is deliberately unchanged (README contract:
+// `rag-memory.db` next to the server); the framework always sets DB_FILE_PATH explicitly instead.
+// Exported so the contract is unit-testable without writing a database into dist/.
+export function resolveDbFilePath(envValue, cwd, serverDir) {
+    if (!envValue)
+        return path.join(serverDir, 'rag-memory.db');
+    return path.isAbsolute(envValue) ? envValue : path.resolve(cwd, envValue);
+}
+const DB_FILE_PATH = resolveDbFilePath(process.env.DB_FILE_PATH, process.cwd(), path.dirname(fileURLToPath(import.meta.url)));
+// Next release (spec §11-2): SQLite mmap is opt-in, default 0 (off). Measured 2026-09-15 (Windows, Google
+// Drive G:, WAL, mmap 256 MB): one writer + two readers -> 373,684 `database disk image is
+// malformed` reads in 20 s; mmap 0 -> 0 errors; local disk -> 0 either way. On macOS Drive the same
+// probe read clean but the WAL grew to ~4 GB in 20 s with mmap on (n=1). The framework opens one
+// DB from several CLIs on synced folders, so correctness wins over an unmeasured read-speed gain.
+// RAG_MEMORY_MMAP_SIZE=<bytes> turns it back on; garbage or negative values count as 0. The value
+// SQLite actually applied is read back and printed in the boot banner (`| mmap <n>`).
+export function parseMmapSize(raw) {
+    if (raw === undefined || raw.trim() === '')
+        return 0;
+    const n = Number(raw);
+    return Number.isFinite(n) && n >= 0 ? Math.floor(n) : 0;
+}
+const MMAP_SIZE = parseMmapSize(process.env.RAG_MEMORY_MMAP_SIZE);
 const EMBEDDING_MODEL = process.env.EMBEDDING_MODEL || 'Xenova/bge-m3';
 // v3.5 default model config — grandfathering legacy vectors is only automatic
 // when the current config matches this (spec §6b custom-model guard). An
@@ -183,6 +206,8 @@ export class RAGKnowledgeGraphManager {
     // The calendar that date-only human labels are written in. Resolved once, here, so an invalid
     // zone fails at construction instead of quietly writing wrong days for weeks.
     calendarTimeZone = resolveCalendarTimeZone(process.env.RAG_MEMORY_CALENDAR_TZ);
+    // The mmap_size SQLite actually applied at initialize() (-1 until then). Banner-only.
+    mmapApplied = -1;
     embeddingCache = new Map();
     EMBEDDING_CACHE_MAX = 500;
     dictionaryCache = null;
@@ -201,7 +226,10 @@ export class RAGKnowledgeGraphManager {
         this.db.pragma('busy_timeout = 5000');
         this.db.pragma('cache_size = -32000');
         this.db.pragma('temp_store = MEMORY');
-        this.db.pragma('mmap_size = 268435456');
+        // Opt-in mmap (see MMAP_SIZE above). Read the applied value back so the banner reports
+        // what SQLite did, not what we asked for (a compile-time cap can lower it silently).
+        this.db.pragma(`mmap_size = ${MMAP_SIZE}`);
+        this.mmapApplied = Number(this.db.pragma('mmap_size', { simple: true }));
         this.db.pragma('foreign_keys = ON');
         // spec §5.2: 관찰 lifecycle 의 무결성은 전부 FK CASCADE 를 전제한다 — root 를 지우면
         // revision 이, revision 을 지우면 source 가 따라가야 history 가 고아로 남지 않는다.
@@ -565,13 +593,78 @@ export class RAGKnowledgeGraphManager {
             }))
         };
     }
+    // Fold the WAL into the main file and empty it. Returns true when the WAL holds no
+    // frames afterwards.
+    //
+    // Why the engine does this itself instead of leaving it to SQLite: SQLite folds the WAL
+    // only when the LAST connection closes, and only if that close gets to run. Neither holds
+    // here — several engines keep the same file open (one per CLI), and some hosts end the
+    // engine without letting it finish (codex-cli 0.155.1, measured: SIGTERM, then SIGKILL
+    // ~185 ms later; on Windows a child is simply terminated). A WAL with frames that outlives
+    // its process is replayed at the next open against whatever main file is there by then, and
+    // in a cloud-synced folder that can be a main file written by another machine. SQLite does
+    // not check that a WAL belongs to the main file next to it. That corrupted a live database
+    // twice. So the invariant is kept continuously: while idle, the WAL is empty.
+    //
+    // TRUNCATE, not PASSIVE: PASSIVE copies the frames into the main file but leaves them in
+    // the WAL, still valid, still replayable onto a foreign main file.
+    // busy_timeout is dropped to 0 around the call: TRUNCATE runs the busy handler, and a 5 s
+    // stall inside a signal handler or a timer tick is worse than retrying on the next tick.
+    checkpointWal() {
+        if (!this.db)
+            return true;
+        try {
+            const st = fsSync.statSync(DB_FILE_PATH + '-wal', { throwIfNoEntry: false });
+            if (!st || st.size === 0)
+                return true;
+            this.db.pragma('busy_timeout = 0');
+            try {
+                const r = this.db.pragma('wal_checkpoint(TRUNCATE)');
+                return r?.[0]?.busy === 0;
+            }
+            finally {
+                this.db.pragma('busy_timeout = 5000');
+            }
+        }
+        catch {
+            return false; // inside a transaction, or the file is locked — the next tick retries
+        }
+    }
+    walTimer = null;
+    walSoon = null;
+    // Every write path ends up here without having to be listed: the periodic tick looks at
+    // the size of the -wal file, not at which tool ran, so background writers (reconciliation,
+    // backfill) are covered as well. noteActivity() only shortens the wait after a tool call.
+    startWalKeeper(tickMs = 1000) {
+        if (this.walTimer)
+            return;
+        this.walTimer = setInterval(() => { this.checkpointWal(); }, tickMs);
+        this.walTimer.unref();
+    }
+    noteActivity(delayMs = 200) {
+        if (this.walSoon)
+            return;
+        this.walSoon = setTimeout(() => { this.walSoon = null; this.checkpointWal(); }, delayMs);
+        this.walSoon.unref();
+    }
     cleanup() {
+        if (this.walTimer) {
+            clearInterval(this.walTimer);
+            this.walTimer = null;
+        }
+        if (this.walSoon) {
+            clearTimeout(this.walSoon);
+            this.walSoon = null;
+        }
         if (this.encoding) {
             this.encoding.free();
             this.encoding = null;
         }
         this.embeddingCache.clear();
         if (this.db) {
+            // close() folds the WAL only for the last connection; with other engines attached it
+            // would leave every frame behind.
+            this.checkpointWal();
             this.db.close();
             this.db = null;
         }
@@ -4033,6 +4126,7 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
     try {
         // Validate arguments using our structured schema
         const validatedArgs = validateToolArgs(name, args);
+        ragKgManager.noteActivity(); // fold the WAL shortly after this call, whatever it wrote
         switch (name) {
             // Original MCP tools
             case "createEntities":
@@ -4175,6 +4269,7 @@ async function main() {
             model: EMBEDDING_MODEL, revision: MODEL_REVISION, dtype: MODEL_DTYPE,
             cachePath: resolveModelCacheDir(process.env, process.platform, os.homedir()),
             dbPath: DB_FILE_PATH,
+            mmap: ragKgManager.mmapApplied,
         });
         if (ragKgManager.embeddingsMode === 'eager') {
             // eager = wait for BOTH the first model load attempt and reconciliation to
@@ -4206,6 +4301,9 @@ async function main() {
             if (shuttingDown)
                 return;
             shuttingDown = true;
+            // Synchronously, before anything is awaited: a host may SIGKILL before the settle
+            // sequence below completes (codex: ~185 ms after SIGTERM).
+            ragKgManager.checkpointWal();
             void (async () => {
                 try {
                     await server.close();
@@ -4221,6 +4319,13 @@ async function main() {
         };
         process.on('SIGINT', shutdown);
         process.on('SIGTERM', shutdown);
+        // SIGHUP = the terminal or multiplexer pane was closed. Without a handler the default
+        // action kills the process before 'exit' listeners run, so the database is never closed.
+        // On Windows, Node raises SIGHUP when the console window is closed and SIGBREAK on
+        // Ctrl+Break; registering them is harmless where they never fire.
+        process.on('SIGHUP', shutdown);
+        process.on('SIGBREAK', shutdown);
+        ragKgManager.startWalKeeper();
         process.on('exit', () => { try {
             ragKgManager.cleanup();
         }

package/package.json

@@ -1,10 +1,10 @@
 {
  "name": "rag-memory-epf-mcp",
- "version": "6.1.0",
+ "version": "6.3.1",
  "engines": {
   "node": ">=24"
  },
- "description": "Project-local RAG memory MCP server \u2014 knowledge graph + multilingual vector + FTS5 in a single SQLite file. Per-project isolation, 38 MCP tools, codepoint-safe chunking (Korean/CJK/emoji).",
+ "description": "Project-local RAG memory MCP server — knowledge graph + multilingual vector + FTS5 in a single SQLite file. Per-project isolation, 38 MCP tools, codepoint-safe chunking (Korean/CJK/emoji).",
  "keywords": [
   "mcp",
   "model-context-protocol",
@@ -45,7 +45,7 @@
   "prepare": "npm run build",
   "watch": "tsc --watch",
   "verify:invariants": "node test/chunk-invariants.test.mjs",
-  "verify:engine": "node test/engine-smoke.test.mjs && node test/launch-smoke.test.mjs && node test/sync-atomicity.test.mjs && node test/dedup.test.mjs && node test/search-degradation.test.mjs && node test/entity-embed-cap.test.mjs && node test/migration12.test.mjs && node test/model-cache.test.mjs && node test/embedding-gate.test.mjs && node test/lazy-boot.test.mjs && node test/reconciliation.test.mjs && node test/backfill.test.mjs && node test/fts-query.test.mjs && node test/search-contracts.test.mjs && node test/tool-contracts.test.mjs && node test/bounded-exit.test.mjs && node test/observation-schema.test.mjs && node test/search-graph-default.test.mjs && node test/observation-migration.test.mjs && node test/observation-lifecycle.test.mjs && node test/observation-contracts.test.mjs && node test/observation-search.test.mjs && node test/observation-cascade.test.mjs && node test/observation-realdata.test.mjs && node test/chunker-c.test.mjs && node test/migration14.test.mjs && node test/chunk-params-validation.test.mjs && node test/vector-reuse.test.mjs && node test/entity-range-linking.test.mjs && node test/alias-link-gate.test.mjs && node test/entity-name-boundary.test.mjs && node test/stats-chunking.test.mjs && node test/migration14-realdata.test.mjs && node test/migration14-realdata-sync.test.mjs && node test/search-summaries-off.test.mjs && node test/document-return-contracts.test.mjs && node test/observation-date-prefix.test.mjs && node test/delete-entities-cascade.test.mjs && node test/delete-entities-kg-hygiene.test.mjs && node test/backup-publish-portable.test.mjs && node test/graph-context-explain.test.mjs && node test/eval-graph-role-libs.test.mjs && node test/eval-graph-role-t5b.test.mjs && node --test test/eval-graph-role-t8-fix.test.mjs && node --test test/eval-graph-role-t7-upstream.test.mjs && node --test test/eval-graph-role-t11-decision.test.mjs && node --test test/eval-graph-role-prereq-fix.test.mjs",
+  "verify:engine": "node test/engine-smoke.test.mjs && node test/launch-smoke.test.mjs && node test/sync-atomicity.test.mjs && node test/dedup.test.mjs && node test/search-degradation.test.mjs && node test/entity-embed-cap.test.mjs && node test/migration12.test.mjs && node test/model-cache.test.mjs && node test/embedding-gate.test.mjs && node test/lazy-boot.test.mjs && node test/reconciliation.test.mjs && node test/backfill.test.mjs && node test/fts-query.test.mjs && node test/search-contracts.test.mjs && node test/tool-contracts.test.mjs && node test/bounded-exit.test.mjs && node test/observation-schema.test.mjs && node test/search-graph-default.test.mjs && node test/observation-migration.test.mjs && node test/observation-lifecycle.test.mjs && node test/observation-contracts.test.mjs && node test/observation-search.test.mjs && node test/observation-cascade.test.mjs && node test/observation-realdata.test.mjs && node test/chunker-c.test.mjs && node test/migration14.test.mjs && node test/chunk-params-validation.test.mjs && node test/vector-reuse.test.mjs && node test/entity-range-linking.test.mjs && node test/alias-link-gate.test.mjs && node test/entity-name-boundary.test.mjs && node test/stats-chunking.test.mjs && node test/migration14-realdata.test.mjs && node test/migration14-realdata-sync.test.mjs && node test/search-summaries-off.test.mjs && node test/document-return-contracts.test.mjs && node test/observation-date-prefix.test.mjs && node test/delete-entities-cascade.test.mjs && node test/delete-entities-kg-hygiene.test.mjs && node test/backup-publish-portable.test.mjs && node test/graph-context-explain.test.mjs && node test/eval-graph-role-libs.test.mjs && node test/eval-graph-role-t5b.test.mjs && node --test test/eval-graph-role-t8-fix.test.mjs && node --test test/eval-graph-role-t7-upstream.test.mjs && node --test test/eval-graph-role-t11-decision.test.mjs && node --test test/eval-graph-role-prereq-fix.test.mjs && node test/db-path.test.mjs && node test/mmap-option.test.mjs && node test/wal-exit.test.mjs",
   "test": "npm run build && npm run verify:invariants && npm run verify:engine",
   "prepublishOnly": "npm run build && npm run verify:invariants && npm run verify:engine"
  },
@@ -68,4 +68,4 @@
   "shx": "^0.3.4",
   "typescript": "^5.6.2"
  }
-}
+}