rag-memory-epf-mcp 3.3.4 → 3.3.6

package/README.md

@@ -13,7 +13,8 @@ A **project-local RAG memory** MCP server — knowledge graph + multilingual vec
 - **3-signal hybrid search** — vector similarity (bge-m3, 1024-dim) + FTS5 BM25 keyword matching + knowledge graph re-ranking, combined via Reciprocal Rank Fusion
 - **100+ languages** — Korean, Chinese, Japanese, Arabic, and more. Cross-lingual search works out of the box.
 - **Graph-aware scoring** — per-entity geometric decay (0.5^i) with hard cap prevents any single document from dominating results
-- **27 MCP tools** — entity/relation CRUD, document pipeline, multi-hop graph traversal, export/import, temporal queries
+- **30 MCP tools** — knowledge graph CRUD, document pipeline, hybrid search, multi-hop traversal, graph analytics (centrality / community detection / structure), export/import, temporal queries
+- **Codepoint-safe chunking** — chunk offsets are Unicode codepoints, language-neutral across SQL `substr`, Python slicing, and JS `[...str]` iteration. Korean/CJK/emoji documents stay aligned. Verified by a publish-time invariant test.
 - **SQLite optimized** — WAL mode, 32MB cache, 256MB mmap, FTS5 triggers, 7 indexes
 - **MCP SDK 1.27.1** — Tool Annotations (readOnly/destructive/idempotent), latest protocol 2025-11-25
 
@@ -35,7 +36,7 @@ A **project-local RAG memory** MCP server — knowledge graph + multilingual vec
 
 Place this `.mcp.json` in each project folder with its own `DB_FILE_PATH`. Each project maintains completely isolated memory.
 
-## Tools (27)
+## Tools (30)
 
 ### Knowledge Graph (7)
 | Tool | Description | Annotation |
@@ -80,6 +81,13 @@ Place this `.mcp.json` in each project folder with its own `DB_FILE_PATH`. Each
 | `runMigrations` | Apply pending migrations | idempotent |
 | `rollbackMigration` | Revert to a previous schema version | destructive |
 
+### Graph Analytics (3)
+| Tool | Description | Annotation |
+|------|------------|------------|
+| `getGraphMetrics` | Per-entity centrality (degree, betweenness, closeness, pagerank) | readOnly |
+| `detectCommunities` | Louvain community detection + modularity score | readOnly |
+| `analyzeGraphStructure` | Density, connected components, clustering coefficient | readOnly |
+
 ## Document Processing Pipeline
 
 ```
@@ -112,7 +120,7 @@ storeDocument(id, content, metadata)
 │  │  ├── chunks (sqlite-vec, 1024-dim)     │  │
 │  │  ├── entity_embeddings (sqlite-vec)    │  │
 │  │  ├── entities_fts + chunks_fts (FTS5)  │  │
-│  │  └── 7 migrations (auto-applied)       │  │
+│  │  └── 11 migrations (auto-applied)      │  │
 │  └────────────────────────────────────────┘  │
 │                                              │
 │  bge-m3 (ONNX, 100+ langs)                   │
@@ -128,6 +136,24 @@ storeDocument(id, content, metadata)
 
 ## Changelog
 
+### v3.3.6
+- **Publish-time invariant test** — `npm run verify:invariants` (wired as `prepublishOnly`) catches `chunkText` offset regressions before they ship. Tests ASCII / Korean / emoji-heavy / mixed CJK + supplementary plane / pure supplementary inputs against the codepoint-slice contract.
+- **`chunkText` extracted to `src/chunkText.ts`** — algorithm now testable in isolation. The class method is a thin wrapper. No user-facing API change.
+- **README accuracy** — tool count corrected to 30, migration count to 11, Graph Analytics tools surfaced.
+
+### v3.3.5
+- **Fix: chunk offsets stored as JS UTF-16 units instead of Unicode codepoints** — Korean/CJK/emoji documents had `start_pos`/`end_pos` that disagreed with SQL `substr` and Python slicing for any chunk crossing a supplementary character. `chunkText` now maintains parallel UTF-16 + codepoint cursors and reports codepoint offsets.
+- **Migration v11** — converts existing `chunk_metadata.start_pos`/`end_pos` from UTF-16 units to codepoints by re-locating each chunk via `indexOf` and counting codepoints.
+
+### v3.3.4
+- **Migration version 9 → 10 jump + `ALTER TABLE` idempotency guards** — some databases from early v3.x experiments (Ollama dimension swap) had recorded a migration at version 9, causing the new v9 migration to silently no-op. Bumped to version 10 and added `PRAGMA table_info` guards so the column-add is safe to re-run.
+
+### v3.3.3
+- **Separate token-space and char-space offsets in `chunk_metadata`** — added `start_token`/`end_token` columns. Existing `start_pos`/`end_pos` are reinterpreted as character offsets into `documents.content`. Backfill migration recomputes char offsets via `indexOf` with a per-document cursor; misses leave NULL so callers can re-chunk to repair.
+
+### v3.3.0
+- **Graph analytics (graphology)** — three new MCP tools: `getGraphMetrics` (degree / betweenness / closeness / pagerank), `detectCommunities` (Louvain + modularity), `analyzeGraphStructure` (density / components / clustering). 27 → 30 tools.
+
 ### v3.2.1
 - **Fix: `autoLinkEntities` silent failure** — was JOINing a non-existent `observations` table (observations are stored as JSON array column in `entities`). Changed to direct column select + `JSON.parse()`.
 
@@ -148,11 +174,11 @@ storeDocument(id, content, metadata)
 - **Multi-hop graph traversal** — `getNeighbors` tool with `WITH RECURSIVE` CTE, depth 1-5, cycle detection, bidirectional
 - **Embedding LRU cache** — 500-entry in-memory cache, skips redundant re-computation
 - **Configurable model** — `EMBEDDING_MODEL` env var to use alternative embedding models
-- 27 tools total
+- 27 tools total at this version (30 as of v3.3.0+)
 
 ### v1.8.0
 - **MCP SDK 1.27.1** — protocol 2025-11-25, security fix GHSA-345p-7cg4-v4c7 (CVSS 7.1)
-- **Tool Annotations** — all 27 tools annotated (readOnlyHint, destructiveHint, idempotentHint)
+- **Tool Annotations** — all tools annotated (readOnlyHint, destructiveHint, idempotentHint)
 - **SIGTERM graceful shutdown** — clean exit without ONNX mutex crash
 
 ### v1.7.0
@@ -188,8 +214,12 @@ git clone https://github.com/bripin123/rag-memory-epf-mcp.git
 cd rag-memory-epf-mcp
 npm install
 npm run build
+npm test                  # build + invariant verification
+npm run verify:invariants # standalone invariant test (assumes dist/ built)
 ```
 
+`npm publish` automatically runs `prepublishOnly` (`build` + `verify:invariants`); a chunk-offset regression blocks the publish at the source.
+
 ## License
 
 MIT License. See [LICENSE](LICENSE).

package/dist/index.js

@@ -21,6 +21,8 @@ import modularity from 'graphology-metrics/graph/modularity.js';
 import { getAllMCPTools, validateToolArgs, getSystemInfo } from './src/tools/tool-registry.js';
 // Import migration system
 import { MigrationManager } from './src/migrations/migration-manager.js';
+// Import chunk text algorithm (extracted for publish-time invariant testing)
+import { chunkText as splitTextIntoChunks } from './src/chunkText.js';
 import { migrations } from './src/migrations/migrations.js';
 import { createHash } from 'crypto';
 import { createRequire } from 'module';
@@ -45,34 +47,7 @@ const EMBEDDING_MODEL = process.env.EMBEDDING_MODEL || 'Xenova/bge-m3';
 // When a chunk is not at the document head/tail, any partial sequence at that
 // edge belongs to an adjacent chunk and must be removed so TextDecoder does
 // not emit U+FFFD. Pass trimHead/trimTail=false to preserve head/tail bytes.
-function trimIncompleteUtf8(bytes, trimHead, trimTail) {
-    let start = 0;
-    let end = bytes.length;
-    if (trimHead) {
-        while (start < end && (bytes[start] & 0xC0) === 0x80)
-            start++;
-    }
-    if (trimTail) {
-        let i = end - 1;
-        while (i >= start && (bytes[i] & 0xC0) === 0x80)
-            i--;
-        if (i >= start) {
-            const lead = bytes[i];
-            let needed = 1;
-            if ((lead & 0x80) === 0)
-                needed = 1;
-            else if ((lead & 0xE0) === 0xC0)
-                needed = 2;
-            else if ((lead & 0xF0) === 0xE0)
-                needed = 3;
-            else if ((lead & 0xF8) === 0xF0)
-                needed = 4;
-            if (end - i < needed)
-                end = i;
-        }
-    }
-    return bytes.subarray(start, end);
-}
+// (Implementation moved to src/chunkText.ts for testability.)
 function safeRowid(value) {
     const n = Number(value);
     if (!Number.isInteger(n) || n < 0) {
@@ -1139,59 +1114,26 @@ class RAGKnowledgeGraphManager {
     //
     // Each chunk records both token-space offsets (start_token/end_token from the
     // BPE encoder loop) and char-space offsets (start_pos/end_pos into the original
-    // text). Char offsets are recovered with indexOf using a running cursor at the
-    // previous chunk's start, so substr(text, start_pos, end_pos - start_pos)
-    // equals chunk.text. On a coincidental indexOf miss the char offsets are NULL.
+    // text). Char offsets are Unicode codepoint counts — language-neutral, so SQL
+    // substr, Python str slicing, and JS [...str] iteration all line up. JS's
+    // native UTF-16 indexing differs for supplementary characters (emoji, rare
+    // CJK), so the function maintains parallel UTF-16 and codepoint cursors and
+    // reports codepoint offsets. On a coincidental indexOf miss the char offsets
+    // are NULL.
     chunkText(text, maxTokens = 800, overlap = 160) {
         if (!this.encoding)
             throw new Error('Tokenizer not initialized');
-        const tokens = this.encoding.encode(text);
-        const chunks = [];
-        let charCursor = 0;
-        for (let i = 0; i < tokens.length; i += maxTokens - overlap) {
-            const chunkTokens = tokens.slice(i, i + maxTokens);
-            const decodedBytes = this.encoding.decode(chunkTokens);
-            const isFirst = i === 0;
-            const isLast = i + chunkTokens.length >= tokens.length;
-            const safeBytes = trimIncompleteUtf8(decodedBytes, !isFirst, !isLast);
-            const chunkText = new TextDecoder('utf-8').decode(safeBytes);
-            // Recover char-space offsets. First chunk is anchored at 0 because
-            // trimIncompleteUtf8 leaves the leading bytes intact when isFirst is true.
-            let startPos;
-            let endPos;
-            if (isFirst) {
-                startPos = 0;
-                endPos = chunkText.length;
-                charCursor = 0;
-            }
-            else if (chunkText.length === 0) {
-                startPos = null;
-                endPos = null;
-            }
-            else {
-                const idx = text.indexOf(chunkText, charCursor);
-                if (idx >= 0) {
-                    startPos = idx;
-                    endPos = idx + chunkText.length;
-                    charCursor = idx;
-                }
-                else {
-                    startPos = null;
-                    endPos = null;
-                }
-            }
-            chunks.push({
-                id: '',
-                document_id: '',
-                chunk_index: chunks.length,
-                text: chunkText,
-                start_pos: startPos,
-                end_pos: endPos,
-                start_token: i,
-                end_token: i + chunkTokens.length
-            });
-        }
-        return chunks;
+        const segments = splitTextIntoChunks(text, this.encoding, maxTokens, overlap);
+        return segments.map((seg, idx) => ({
+            id: '',
+            document_id: '',
+            chunk_index: idx,
+            text: seg.text,
+            start_pos: seg.start_pos,
+            end_pos: seg.end_pos,
+            start_token: seg.start_token,
+            end_token: seg.end_token
+        }));
     }
     // Generate embeddings using sentence transformers
     // isQuery: true for search queries (adds instruction prefix), false for documents/entities

package/dist/src/chunkText.d.ts

@@ -0,0 +1,10 @@
+import type { Tiktoken } from 'tiktoken';
+export interface ChunkSegment {
+    text: string;
+    start_pos: number | null;
+    end_pos: number | null;
+    start_token: number;
+    end_token: number;
+}
+export declare function trimIncompleteUtf8(bytes: Uint8Array, trimHead: boolean, trimTail: boolean): Uint8Array;
+export declare function chunkText(text: string, encoding: Tiktoken, maxTokens?: number, overlap?: number): ChunkSegment[];

package/dist/src/chunkText.js

@@ -0,0 +1,104 @@
+// Tokenize and chunk text using a BPE encoder while reporting both token-space
+// and char-space (Unicode codepoint) offsets back into the original string.
+//
+// BPE tokenizers (cl100k_base) split multi-byte UTF-8 sequences across tokens.
+// Slicing token arrays at arbitrary boundaries can leave incomplete UTF-8
+// prefix/suffix bytes, which TextDecoder replaces with U+FFFD (�). We trim the
+// incomplete sequences at chunk boundaries; overlap covers the removed bytes.
+//
+// Each chunk records both token-space offsets (start_token/end_token from the
+// BPE encoder loop) and char-space offsets (start_pos/end_pos into the original
+// text). Char offsets are Unicode codepoint counts — language-neutral, so SQL
+// substr, Python str slicing, and JS [...str] iteration all line up. JS's
+// native UTF-16 indexing differs for supplementary characters (emoji, rare CJK),
+// so the function maintains parallel UTF-16 and codepoint cursors and reports
+// codepoint offsets. On a coincidental indexOf miss the char offsets are NULL.
+//
+// Extracted to a standalone module so publish-time invariant tests can exercise
+// the algorithm directly without booting the full RAG-Memory stack.
+// trimIncompleteUtf8: strip incomplete UTF-8 sequences from the head/tail of a
+// byte buffer produced by decoding an arbitrary token slice. A multi-byte
+// codepoint that begins or ends on the cut edge belongs to an adjacent chunk
+// and must be removed so TextDecoder does not emit U+FFFD. Pass
+// trimHead/trimTail=false to preserve head/tail bytes (first/last chunks).
+export function trimIncompleteUtf8(bytes, trimHead, trimTail) {
+    let start = 0;
+    let end = bytes.length;
+    if (trimHead) {
+        while (start < end && (bytes[start] & 0xC0) === 0x80)
+            start++;
+    }
+    if (trimTail) {
+        let i = end - 1;
+        while (i >= start && (bytes[i] & 0xC0) === 0x80)
+            i--;
+        if (i >= start) {
+            const lead = bytes[i];
+            let needed = 1;
+            if ((lead & 0x80) === 0)
+                needed = 1;
+            else if ((lead & 0xE0) === 0xC0)
+                needed = 2;
+            else if ((lead & 0xF0) === 0xE0)
+                needed = 3;
+            else if ((lead & 0xF8) === 0xF0)
+                needed = 4;
+            if (end - i < needed)
+                end = i;
+        }
+    }
+    return bytes.subarray(start, end);
+}
+export function chunkText(text, encoding, maxTokens = 800, overlap = 160) {
+    const tokens = encoding.encode(text);
+    const segments = [];
+    let utf16Cursor = 0;
+    let cpCursor = 0;
+    for (let i = 0; i < tokens.length; i += maxTokens - overlap) {
+        const chunkTokens = tokens.slice(i, i + maxTokens);
+        const decodedBytes = encoding.decode(chunkTokens);
+        const isFirst = i === 0;
+        const isLast = i + chunkTokens.length >= tokens.length;
+        const safeBytes = trimIncompleteUtf8(decodedBytes, !isFirst, !isLast);
+        const chunkTextStr = new TextDecoder('utf-8').decode(safeBytes);
+        let startPos;
+        let endPos;
+        if (isFirst) {
+            startPos = 0;
+            endPos = [...chunkTextStr].length;
+            utf16Cursor = 0;
+            cpCursor = 0;
+        }
+        else if (chunkTextStr.length === 0) {
+            startPos = null;
+            endPos = null;
+        }
+        else {
+            const utfIdx = text.indexOf(chunkTextStr, utf16Cursor);
+            if (utfIdx >= 0) {
+                // Advance cpCursor by codepoints between the previous cursor and the
+                // new chunk's start (handles overlap by anchoring at the previous
+                // chunk's start, not its end).
+                if (utfIdx > utf16Cursor) {
+                    cpCursor += [...text.slice(utf16Cursor, utfIdx)].length;
+                    utf16Cursor = utfIdx;
+                }
+                const cpLen = [...chunkTextStr].length;
+                startPos = cpCursor;
+                endPos = cpCursor + cpLen;
+            }
+            else {
+                startPos = null;
+                endPos = null;
+            }
+        }
+        segments.push({
+            text: chunkTextStr,
+            start_pos: startPos,
+            end_pos: endPos,
+            start_token: i,
+            end_token: i + chunkTokens.length
+        });
+    }
+    return segments;
+}

package/dist/src/migrations/migrations.js

@@ -520,5 +520,55 @@ export const migrations = [
             // start_token/end_token columns remain (no DROP COLUMN); they will be
             // ignored by older code.
         }
+    },
+    // Migration 11: Convert chunk_metadata.start_pos/end_pos from JS UTF-16 code
+    // unit indices to Unicode codepoint indices. v3.3.4 stored offsets in JS's
+    // native UTF-16 unit space, which mismatches SQL substr/length and Python
+    // string indexing for any document containing supplementary characters
+    // (emoji, rare CJK). Codepoints are language-neutral. Walks each document's
+    // chunks in chunk_index order, locating each chunk in the source via UTF-16
+    // indexOf and counting codepoints between cursor positions to derive the
+    // codepoint offsets. Idempotent — chunks where indexOf misses keep their
+    // existing values.
+    {
+        version: 11,
+        description: 'Convert chunk_metadata.start_pos/end_pos to Unicode codepoint indices',
+        up: (db) => {
+            const docRows = db.prepare(`SELECT DISTINCT document_id FROM chunk_metadata
+        WHERE chunk_type='document' AND document_id IS NOT NULL
+          AND start_pos IS NOT NULL AND end_pos IS NOT NULL`).all();
+            const getContent = db.prepare(`SELECT content FROM documents WHERE id = ?`);
+            const getChunks = db.prepare(`SELECT rowid, text FROM chunk_metadata
+        WHERE document_id=? AND chunk_type='document'
+          AND start_pos IS NOT NULL AND end_pos IS NOT NULL
+        ORDER BY chunk_index ASC`);
+            const upd = db.prepare(`UPDATE chunk_metadata SET start_pos=?, end_pos=? WHERE rowid=?`);
+            for (const { document_id } of docRows) {
+                const doc = getContent.get(document_id);
+                if (!doc)
+                    continue;
+                const content = doc.content;
+                const chunks = getChunks.all(document_id);
+                let utf16Cursor = 0;
+                let cpCursor = 0;
+                for (const c of chunks) {
+                    if (!c.text)
+                        continue;
+                    const utfIdx = content.indexOf(c.text, utf16Cursor);
+                    if (utfIdx < 0)
+                        continue; // miss: leave existing
+                    if (utfIdx > utf16Cursor) {
+                        cpCursor += [...content.slice(utf16Cursor, utfIdx)].length;
+                        utf16Cursor = utfIdx;
+                    }
+                    const cpLen = [...c.text].length;
+                    upd.run(cpCursor, cpCursor + cpLen, c.rowid);
+                }
+            }
+        },
+        down: (_db) => {
+            // No clean reversal: would need the original document content to recompute
+            // UTF-16 indices, and we never recorded which chunks were touched. No-op.
+        }
     }
 ];

package/package.json

@@ -1,7 +1,25 @@
 {
   "name": "rag-memory-epf-mcp",
-  "version": "3.3.4",
-  "description": "MCP server for project-local RAG memory with knowledge graph and multilingual vector search",
+  "version": "3.3.6",
+  "description": "Project-local RAG memory MCP server — knowledge graph + multilingual vector + FTS5 in a single SQLite file. Per-project isolation, 30 MCP tools, codepoint-safe chunking (Korean/CJK/emoji).",
+  "keywords": [
+    "mcp",
+    "model-context-protocol",
+    "rag",
+    "knowledge-graph",
+    "vector-search",
+    "fts5",
+    "sqlite",
+    "embeddings",
+    "bge-m3",
+    "multilingual",
+    "korean",
+    "claude-code",
+    "gemini-cli",
+    "codex-cli",
+    "memory",
+    "agent-memory"
+  ],
   "license": "MIT",
   "author": "bripin123",
   "homepage": "https://github.com/bripin123/rag-memory-epf-mcp",
@@ -21,7 +39,10 @@
   "scripts": {
     "build": "tsc && shx chmod +x dist/*.js",
     "prepare": "npm run build",
-    "watch": "tsc --watch"
+    "watch": "tsc --watch",
+    "verify:invariants": "node test/chunk-invariants.test.mjs",
+    "test": "npm run build && npm run verify:invariants",
+    "prepublishOnly": "npm run build && npm run verify:invariants"
   },
   "dependencies": {
     "@huggingface/transformers": "^3.5.1",