rag-memory-epf-mcp 5.2.0 → 5.3.0

package/README.md

@@ -10,9 +10,9 @@ A **project-local RAG memory** MCP server — knowledge graph + multilingual vec
 ## Key Features
 
 - **Project-local isolation** — each project gets its own `.memory/rag-memory.db`. Multiple projects run simultaneously without interference.
-- **3-signal hybrid search** — vector similarity (bge-m3, 1024-dim) + FTS5 BM25 keyword matching + knowledge graph re-ranking, combined via Reciprocal Rank Fusion
+- **Hybrid search** — vector similarity (bge-m3, 1024-dim) + FTS5 BM25 keyword matching (RRF-fused). Knowledge-graph re-ranking is an **opt-in legacy/experimental** signal since v5.3.0 (measured to hurt known-item retrieval)
 - **100+ languages** — Korean, Chinese, Japanese, Arabic, and more. Cross-lingual search works out of the box.
-- **Graph-aware scoring** — per-entity geometric decay (0.5^i) with hard cap prevents any single document from dominating results
+- **Graph re-ranker (opt-in)** — per-entity geometric decay (0.5^i) with a hard cap 0.4; the cap bounds the boost but was measured (2026-08-17) to let heavily-linked chunks saturate it and outrank the exact chunk — hence off by default
 - **38 MCP tools** — knowledge graph CRUD, observation lifecycle (correct / retract / history), document pipeline, hybrid search, multi-hop traversal, graph analytics (centrality / community detection / structure), export/import, temporal queries
 - **Observations that hold their history** — corrections supersede instead of overwrite, search returns only current facts, and every revision keeps its provenance
 - **Structure-anchored chunking (c1, v5)** — boundaries anchor to markdown structure (fence-aware, H1–H4 first, block-greedy, exact-token fallback), so editing the top of a file no longer re-embeds the whole document: unchanged text reuses its stored vectors at sync time. Chunk offsets are Unicode codepoints, language-neutral across SQL `substr`, Python slicing, and JS `[...str]` iteration; a publish-time invariant gate locks the gap-free partition. `overlap` is retired (omit or 0).
@@ -77,7 +77,7 @@ Place this `.mcp.json` in each project folder with its own `DB_FILE_PATH`. Each
 ### Search & Retrieval (9)
 | Tool | Description | Annotation |
 |------|------------|------------|
-| `hybridSearch` | Vector + FTS5 BM25 + graph traversal (3-signal). Degrades to FTS5-only (`search_mode`) when the embedding model is down | readOnly |
+| `hybridSearch` | Vector + FTS5 BM25, plus an **opt-in** graph re-ranker (`useGraph: true`; default off since v5.3.0). Degrades to FTS5-only (`search_mode`) when the embedding model is down | readOnly |
 | `searchNodes` | Semantic entity search with `since`/`until` temporal filtering | readOnly |
 | `openNodes` | Retrieve specific entities by name | readOnly |
 | `readGraph` | Get complete knowledge graph | readOnly |
@@ -153,6 +153,11 @@ storeDocument(id, content, metadata)
 
 ## Changelog
 
+### v5.3.0
+- (Published first as `5.3.0-rc.1` on the `next` dist-tag; promoted to `latest` after a canary run of the published artifact against a real project database: default call carries no `graph_boost` and equals explicit `useGraph:false`, the known-item probe from the 2026-08-17 measurement returns the correct gotcha at rank 1, opt-in `true` still exposes `graph_boost`, schema/MCP defaults read `false`.)
+- **Behavior change — `hybridSearch` graph re-ranking is now opt-in** (`useGraph` default `true` → `false`; tool schema, MCP exposure and the manager signature agree). Omitting the argument now means "no graph re-ranking" — a behavior change for callers that relied on the old default, hence a release-candidate first (`next` dist-tag, fleet canary) before stable. Measured 2026-08-17 on three real corpora (self-retrieval, usable samples 120/117/120, summaries off): with the additive graph boost on, the known-item chunk got worse in 46/49/52 samples and better in 3/2/0 (sign test p < 7e-11 per corpus), 106 targets left the top-10 entirely; reproduced on the summaries-on product path (HAL, 20 paired samples: hit@1 10→7, hit@5 18→13). Mechanism: only query-matched/connected entities score, but the per-entity boost saturates the cap quickly, so heavily-linked chunks can outrank the exact chunk even at `vector_similarity` 0. This is a harm-reduced default, not a validated graph improvement: the boost path is unchanged for `useGraph: true` (legacy/experimental re-ranker for back-compat and evaluation; the graph does not generate candidates — for relationship exploration use `openNodes` → `getNeighbors`). Regression lock: `test/search-graph-default.test.mjs`.
+- (v5.0.0–v5.2.0 notes live in the git tags / `docs/UPDATING.md`.)
+
 ### v4.0.0
 
 **Observation lifecycle (schema v13).** Observations used to be a JSON array of strings on the

package/dist/index.js

@@ -2923,7 +2923,19 @@ export class RAGKnowledgeGraphManager {
         this.coordinator?.kick();
         return { imported, skipped, observation_order_remap: remapReport };
     }
-    async hybridSearch(query, limit = 5, useGraph = true) {
+    // v5.3.0: the graph re-ranker is OPT-IN (harm-reduced default, not a validated improvement).
+    // Measured 2026-08-17 on three real corpora (self-retrieval, usable samples 120/117/120,
+    // summaries off): with the additive graph boost on, the known-item chunk got WORSE in
+    // 46/49/52 samples and BETTER in 3/2/0 (sign test p < 7e-11 per corpus); 106 targets left the
+    // top-10 entirely. Reproduced on the summaries-on product path (HAL, 20 paired samples:
+    // hit@1 10 -> 7, hit@5 18 -> 13, worse 8 / better 3). Mechanism: only query-matched or
+    // connected entities score, but the per-entity boost (0.5^i decay, cap 0.4) saturates fast, so
+    // heavily-linked chunks get more chances to match and reach the cap — they can outrank the
+    // exact chunk even at vector_similarity 0. Note the graph does not generate candidates: it only
+    // re-orders the vector/FTS candidate pool, so useGraph:true is a legacy/experimental re-ranker
+    // (backward compatibility, controlled evaluation), not a relationship-exploration path — that
+    // contract is openNodes -> getNeighbors. The boost path itself is unchanged.
+    async hybridSearch(query, limit = 5, useGraph = false) {
         if (!this.db)
             throw new Error('Database not initialized');
         if (!this.encoding)
@@ -3870,7 +3882,8 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
                 return { content: [{ type: "text", text: JSON.stringify(await ragKgManager.linkEntitiesToDocument(validatedArgs.documentId, validatedArgs.entityNames), null, 2) }] };
             case "hybridSearch":
                 const limit = typeof validatedArgs.limit === 'number' ? validatedArgs.limit : 5;
-                const useGraph = validatedArgs.useGraph !== false;
+                // v5.3.0: graph is opt-in — only an explicit true enables the re-ranker (schema default false).
+                const useGraph = validatedArgs.useGraph === true;
                 return { content: [{ type: "text", text: JSON.stringify(await ragKgManager.hybridSearch(validatedArgs.query, limit, useGraph), null, 2) }] };
             case "getDetailedContext":
                 return { content: [{ type: "text", text: JSON.stringify(await ragKgManager.getDetailedContext(validatedArgs.chunkId, validatedArgs.includeSurrounding !== false), null, 2) }] };

package/dist/src/migrations/migrations.js

@@ -9,7 +9,16 @@ export const migrations = [
         version: 1,
         description: 'Complete RAG Knowledge Graph schema - all tables and features',
         up: (db) => {
-            // Disable foreign key enforcement to make deletions easier
+            // ⚠ 이 줄은 **아무 일도 하지 않는다** (2026-08-11 실측). migration-manager 가 각
+            // 마이그레이션을 `db.transaction(...)` 으로 감싸는데, SQLite 에서 `PRAGMA foreign_keys`
+            // 는 **트랜잭션 안에서 no-op** 이기 때문이다. 실측 = 신규 DB 에 13개 마이그레이션을
+            // 전부 적용한 뒤에도 `foreign_keys = 1`.
+            //
+            // **고치지 말 것.** 이 줄이 실제로 동작하게 만들면(트랜잭션 밖으로 빼는 등) FK 는
+            // 마이그레이션 이후 **그 프로세스 수명 내내 꺼진 채로 남는다** — 부팅 게이트(index.ts)
+            // 는 `runMigrations()` **앞**에서 판정하므로 그걸 못 잡는다. 그 상태에서 entity 를
+            // 지우면 observation 계열이 CASCADE 되지 않아 고아·FK 위반이 쌓인다.
+            // 회귀 잠금 = test/observation-cascade.test.mjs T25(신규 DB 부팅 후 FK==1).
             db.pragma('foreign_keys = OFF');
             // Original entities table (enhanced)
             db.exec(`

package/dist/src/tools/knowledge-graph-tools.js

@@ -292,22 +292,24 @@ const hybridSearchCapability = {
             },
             useGraph: {
                 type: 'boolean',
-                description: 'Whether to enhance results with knowledge graph connections',
-                default: true
+                description: 'Opt-in legacy/experimental graph re-ranker (default false since v5.3.0 — measured to hurt known-item retrieval). It only re-orders the vector/FTS candidate pool; for relationship exploration use openNodes -> getNeighbors instead',
+                default: false
             }
         },
         required: ['query'],
     },
 };
 const hybridSearchDescription = () => `<description>
-Perform sophisticated hybrid search that combines vector similarity with knowledge graph traversal for superior results.
-**The most powerful search tool in the system** - leverages both semantic similarity and structural relationships.
-Perfect for complex queries that benefit from both content matching and conceptual connections.
+Hybrid document search: vector similarity + FTS5 BM25 over document chunks, with an **opt-in** knowledge-graph re-ranker.
+Default (useGraph false) is the harm-reduced default for finding a fact you know exists (known-item retrieval).
+useGraph: true is a legacy/experimental re-ranker kept for backward compatibility and controlled evaluation — it does not
+add candidates, it only re-orders the vector/FTS pool, and measured on three real corpora (2026-08-17) it pushed the exact
+chunk out of the top-5 in ~40% of known-item queries. For relationship exploration use openNodes -> getNeighbors.
 </description>
 
 <importantNotes>
-- (!important!) **Hybrid approach is more powerful** than pure vector or graph search alone
-- (!important!) Graph enhancement finds related concepts even if not directly mentioned
+- (!important!) **Graph re-ranking is opt-in (default false since v5.3.0)** — it re-orders candidates by matched/connected entity links, which is measured to hurt known-item retrieval; enable it only for backward compatibility or controlled evaluation
+- (!important!) Graph re-ranking does not generate candidates — for "what is connected to X" use openNodes -> getNeighbors
 - (!important!) Results include similarity scores, graph boost, and hybrid rankings
 - (!important!) **Best results when knowledge graph is well-populated** with entities and relationships
 </importantNotes>
@@ -332,7 +334,7 @@ Perfect for complex queries that benefit from both content matching and conceptu
 
 <bestPractices>
 - Use natural language queries rather than keywords
-- Enable graph enhancement for better conceptual coverage
+- Keep graph re-ranking off for "find the fact I know exists"; for "what is connected to this" use openNodes -> getNeighbors, not useGraph
 - Start with broader queries, then narrow down based on results
 - Review entity associations to understand why results were selected
 - Use appropriate limits based on your analysis needs
@@ -342,19 +344,18 @@ Perfect for complex queries that benefit from both content matching and conceptu
 <parameters>
 - query: Natural language search query (string, required)
 - limit: Maximum results to return, default 5 (number, optional)
-- useGraph: Enable knowledge graph enhancement, default true (boolean, optional)
+- useGraph: Enable knowledge graph re-ranking, default false / opt-in (boolean, optional)
 </parameters>
 
 <examples>
-- Conceptual search: {"query": "machine learning applications in healthcare", "limit": 10}
-- Technical research: {"query": "React performance optimization techniques", "useGraph": true}
-- Discovery mode: {"query": "Einstein's contributions to modern physics", "limit": 15}
-- Quick lookup: {"query": "quantum computing advantages", "limit": 3, "useGraph": false}
+- Known-item lookup (default, graph off): {"query": "machine learning applications in healthcare", "limit": 10}
+- Legacy re-ranker (opt-in, evaluation/back-compat only): {"query": "React performance optimization techniques", "useGraph": true}
+- Discovery / relationship exploration: use openNodes then getNeighbors on the entity, not this tool's useGraph
 </examples>`;
 const hybridSearchSchema = {
     query: z.string().describe('The search query to find relevant information'),
     limit: z.number().optional().default(5).describe('Maximum number of results to return'),
-    useGraph: z.boolean().optional().default(true).describe('Whether to enhance results with knowledge graph connections'),
+    useGraph: z.boolean().optional().default(false).describe('Opt-in legacy/experimental graph re-ranker (default false since v5.3.0; measured to hurt known-item retrieval; for exploration use openNodes -> getNeighbors)'),
 };
 export const hybridSearchTool = {
     capability: hybridSearchCapability,

package/docs/UPDATING.md

@@ -108,6 +108,30 @@ path and holder pid (e.g. `.download-<key>.lock`). Verify the holder process
 is genuinely gone or hung (`ps -p <pid>`), then remove the lock file manually;
 the next start becomes a clean download owner.
 
+## v5.3.0 (schema v14, unchanged): `hybridSearch` graph re-ranking is opt-in
+
+**What changed.** `useGraph` defaults to `false` (was `true`) — in the manager signature, the tool JSON
+schema, the zod schema (`validateToolArgs` fills `false`) and the MCP dispatch (`=== true`). Nothing else
+in the scoring path moved: `useGraph: true` runs exactly the pre-5.3 graph boost.
+
+**Why.** Measured 2026-08-17 on three real corpora (self-retrieval, usable samples 120/117/120, summaries
+off): with the additive graph boost on, the known-item chunk got WORSE in 46/49/52 samples and BETTER in
+3/2/0 (paired sign test p < 7e-11 per corpus); 106 targets fell out of the top-10 entirely. Reproduced on
+the summaries-on product path (HAL, 20 paired samples: hit@1 10→7, hit@5 18→13). Only query-matched or
+connected entities score, but the per-entity boost (0.5^i decay, cap 0.4) saturates fast, so heavily-linked
+chunks can outrank the exact chunk even at `vector_similarity` 0. This is a reversible harm mitigation, not
+a validated graph improvement — the graph's role (candidate generation vs. re-ranking, explicit mode) is
+still open and will be decided on a graph-required query suite. Note that `useGraph: true` does not add
+candidates (it re-orders the vector/FTS pool): for relationship exploration use `openNodes` → `getNeighbors`.
+
+**Fleet rollout.** No migration, no schema change. Published first as `5.3.0-rc.1` on the `next`
+dist-tag (omitting an argument changes behavior, so it got a canary before `latest`): the published rc was
+installed fresh and run against a real project database — default call had no `graph_boost` and matched
+explicit `useGraph:false`, the known-item probe returned the right chunk at rank 1, opt-in `true` kept
+`graph_boost`, schema/MCP defaults read `false` — then `5.3.0` was published to `latest`. A caller that
+relied on graph re-ranking by default must now pass `useGraph: true`. Callers that already choose per query
+see no change. Regression lock: `test/search-graph-default.test.mjs`.
+
 ## v5.1.0 (schema v14, unchanged): destructive-replace reporting + `excludePattern`
 
 **What changes on upgrade**: nothing you have to do. No migration, no schema

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "rag-memory-epf-mcp",
-  "version": "5.2.0",
+  "version": "5.3.0",
   "engines": {
     "node": ">=24"
   },
@@ -45,7 +45,7 @@
     "prepare": "npm run build",
     "watch": "tsc --watch",
     "verify:invariants": "node test/chunk-invariants.test.mjs",
-    "verify:engine": "node test/engine-smoke.test.mjs && node test/launch-smoke.test.mjs && node test/sync-atomicity.test.mjs && node test/dedup.test.mjs && node test/search-degradation.test.mjs && node test/entity-embed-cap.test.mjs && node test/migration12.test.mjs && node test/model-cache.test.mjs && node test/embedding-gate.test.mjs && node test/lazy-boot.test.mjs && node test/reconciliation.test.mjs && node test/backfill.test.mjs && node test/fts-query.test.mjs && node test/search-contracts.test.mjs && node test/tool-contracts.test.mjs && node test/bounded-exit.test.mjs && node test/observation-schema.test.mjs && node test/observation-migration.test.mjs && node test/observation-lifecycle.test.mjs && node test/observation-contracts.test.mjs && node test/observation-search.test.mjs && node test/observation-cascade.test.mjs && node test/observation-realdata.test.mjs && node test/chunker-c.test.mjs && node test/migration14.test.mjs && node test/chunk-params-validation.test.mjs && node test/vector-reuse.test.mjs && node test/entity-range-linking.test.mjs && node test/stats-chunking.test.mjs && node test/migration14-realdata.test.mjs && node test/migration14-realdata-sync.test.mjs && node test/search-summaries-off.test.mjs && node test/document-return-contracts.test.mjs && node test/observation-date-prefix.test.mjs",
+    "verify:engine": "node test/engine-smoke.test.mjs && node test/launch-smoke.test.mjs && node test/sync-atomicity.test.mjs && node test/dedup.test.mjs && node test/search-degradation.test.mjs && node test/entity-embed-cap.test.mjs && node test/migration12.test.mjs && node test/model-cache.test.mjs && node test/embedding-gate.test.mjs && node test/lazy-boot.test.mjs && node test/reconciliation.test.mjs && node test/backfill.test.mjs && node test/fts-query.test.mjs && node test/search-contracts.test.mjs && node test/tool-contracts.test.mjs && node test/bounded-exit.test.mjs && node test/observation-schema.test.mjs && node test/search-graph-default.test.mjs && node test/observation-migration.test.mjs && node test/observation-lifecycle.test.mjs && node test/observation-contracts.test.mjs && node test/observation-search.test.mjs && node test/observation-cascade.test.mjs && node test/observation-realdata.test.mjs && node test/chunker-c.test.mjs && node test/migration14.test.mjs && node test/chunk-params-validation.test.mjs && node test/vector-reuse.test.mjs && node test/entity-range-linking.test.mjs && node test/stats-chunking.test.mjs && node test/migration14-realdata.test.mjs && node test/migration14-realdata-sync.test.mjs && node test/search-summaries-off.test.mjs && node test/document-return-contracts.test.mjs && node test/observation-date-prefix.test.mjs && node test/delete-entities-cascade.test.mjs",
     "test": "npm run build && npm run verify:invariants && npm run verify:engine",
     "prepublishOnly": "npm run build && npm run verify:invariants && npm run verify:engine"
   },

package/dist/src/chunkText.d.ts

@@ -1,10 +0,0 @@
-import type { Tiktoken } from 'tiktoken';
-export interface ChunkSegment {
-    text: string;
-    start_pos: number | null;
-    end_pos: number | null;
-    start_token: number;
-    end_token: number;
-}
-export declare function trimIncompleteUtf8(bytes: Uint8Array, trimHead: boolean, trimTail: boolean): Uint8Array;
-export declare function chunkText(text: string, encoding: Tiktoken, maxTokens?: number, overlap?: number): ChunkSegment[];

package/dist/src/chunkText.js

@@ -1,104 +0,0 @@
-// Tokenize and chunk text using a BPE encoder while reporting both token-space
-// and char-space (Unicode codepoint) offsets back into the original string.
-//
-// BPE tokenizers (cl100k_base) split multi-byte UTF-8 sequences across tokens.
-// Slicing token arrays at arbitrary boundaries can leave incomplete UTF-8
-// prefix/suffix bytes, which TextDecoder replaces with U+FFFD (�). We trim the
-// incomplete sequences at chunk boundaries; overlap covers the removed bytes.
-//
-// Each chunk records both token-space offsets (start_token/end_token from the
-// BPE encoder loop) and char-space offsets (start_pos/end_pos into the original
-// text). Char offsets are Unicode codepoint counts — language-neutral, so SQL
-// substr, Python str slicing, and JS [...str] iteration all line up. JS's
-// native UTF-16 indexing differs for supplementary characters (emoji, rare CJK),
-// so the function maintains parallel UTF-16 and codepoint cursors and reports
-// codepoint offsets. On a coincidental indexOf miss the char offsets are NULL.
-//
-// Extracted to a standalone module so publish-time invariant tests can exercise
-// the algorithm directly without booting the full RAG-Memory stack.
-// trimIncompleteUtf8: strip incomplete UTF-8 sequences from the head/tail of a
-// byte buffer produced by decoding an arbitrary token slice. A multi-byte
-// codepoint that begins or ends on the cut edge belongs to an adjacent chunk
-// and must be removed so TextDecoder does not emit U+FFFD. Pass
-// trimHead/trimTail=false to preserve head/tail bytes (first/last chunks).
-export function trimIncompleteUtf8(bytes, trimHead, trimTail) {
-    let start = 0;
-    let end = bytes.length;
-    if (trimHead) {
-        while (start < end && (bytes[start] & 0xC0) === 0x80)
-            start++;
-    }
-    if (trimTail) {
-        let i = end - 1;
-        while (i >= start && (bytes[i] & 0xC0) === 0x80)
-            i--;
-        if (i >= start) {
-            const lead = bytes[i];
-            let needed = 1;
-            if ((lead & 0x80) === 0)
-                needed = 1;
-            else if ((lead & 0xE0) === 0xC0)
-                needed = 2;
-            else if ((lead & 0xF0) === 0xE0)
-                needed = 3;
-            else if ((lead & 0xF8) === 0xF0)
-                needed = 4;
-            if (end - i < needed)
-                end = i;
-        }
-    }
-    return bytes.subarray(start, end);
-}
-export function chunkText(text, encoding, maxTokens = 800, overlap = 160) {
-    const tokens = encoding.encode(text);
-    const segments = [];
-    let utf16Cursor = 0;
-    let cpCursor = 0;
-    for (let i = 0; i < tokens.length; i += maxTokens - overlap) {
-        const chunkTokens = tokens.slice(i, i + maxTokens);
-        const decodedBytes = encoding.decode(chunkTokens);
-        const isFirst = i === 0;
-        const isLast = i + chunkTokens.length >= tokens.length;
-        const safeBytes = trimIncompleteUtf8(decodedBytes, !isFirst, !isLast);
-        const chunkTextStr = new TextDecoder('utf-8').decode(safeBytes);
-        let startPos;
-        let endPos;
-        if (isFirst) {
-            startPos = 0;
-            endPos = [...chunkTextStr].length;
-            utf16Cursor = 0;
-            cpCursor = 0;
-        }
-        else if (chunkTextStr.length === 0) {
-            startPos = null;
-            endPos = null;
-        }
-        else {
-            const utfIdx = text.indexOf(chunkTextStr, utf16Cursor);
-            if (utfIdx >= 0) {
-                // Advance cpCursor by codepoints between the previous cursor and the
-                // new chunk's start (handles overlap by anchoring at the previous
-                // chunk's start, not its end).
-                if (utfIdx > utf16Cursor) {
-                    cpCursor += [...text.slice(utf16Cursor, utfIdx)].length;
-                    utf16Cursor = utfIdx;
-                }
-                const cpLen = [...chunkTextStr].length;
-                startPos = cpCursor;
-                endPos = cpCursor + cpLen;
-            }
-            else {
-                startPos = null;
-                endPos = null;
-            }
-        }
-        segments.push({
-            text: chunkTextStr,
-            start_pos: startPos,
-            end_pos: endPos,
-            start_token: i,
-            end_token: i + chunkTokens.length
-        });
-    }
-    return segments;
-}