rag-memory-epf-mcp 3.4.0 → 3.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +13 -4
- package/dist/index.d.ts +293 -0
- package/dist/index.js +121 -36
- package/package.json +4 -3
- package/dist/rag-memory.db +0 -0
- package/dist/rag-memory.db-shm +0 -0
- package/dist/rag-memory.db-wal +0 -0
package/README.md
CHANGED
|
@@ -13,7 +13,7 @@ A **project-local RAG memory** MCP server — knowledge graph + multilingual vec
|
|
|
13
13
|
- **3-signal hybrid search** — vector similarity (bge-m3, 1024-dim) + FTS5 BM25 keyword matching + knowledge graph re-ranking, combined via Reciprocal Rank Fusion
|
|
14
14
|
- **100+ languages** — Korean, Chinese, Japanese, Arabic, and more. Cross-lingual search works out of the box.
|
|
15
15
|
- **Graph-aware scoring** — per-entity geometric decay (0.5^i) with hard cap prevents any single document from dominating results
|
|
16
|
-
- **
|
|
16
|
+
- **31 MCP tools** — knowledge graph CRUD, document pipeline, hybrid search, multi-hop traversal, graph analytics (centrality / community detection / structure), export/import, temporal queries
|
|
17
17
|
- **Codepoint-safe chunking** — chunk offsets are Unicode codepoints, language-neutral across SQL `substr`, Python slicing, and JS `[...str]` iteration. Korean/CJK/emoji documents stay aligned. Verified by a publish-time invariant test.
|
|
18
18
|
- **SQLite optimized** — WAL mode, 32MB cache, 256MB mmap, FTS5 triggers, 7 indexes
|
|
19
19
|
- **MCP SDK 1.27.1** — Tool Annotations (readOnly/destructive/idempotent), latest protocol 2025-11-25
|
|
@@ -36,7 +36,7 @@ A **project-local RAG memory** MCP server — knowledge graph + multilingual vec
|
|
|
36
36
|
|
|
37
37
|
Place this `.mcp.json` in each project folder with its own `DB_FILE_PATH`. Each project maintains completely isolated memory.
|
|
38
38
|
|
|
39
|
-
## Tools (
|
|
39
|
+
## Tools (31)
|
|
40
40
|
|
|
41
41
|
### Knowledge Graph (7)
|
|
42
42
|
| Tool | Description | Annotation |
|
|
@@ -49,7 +49,7 @@ Place this `.mcp.json` in each project folder with its own `DB_FILE_PATH`. Each
|
|
|
49
49
|
| `deleteRelations` | Remove specific relationships | destructive |
|
|
50
50
|
| `deleteObservations` | Remove specific observations | destructive |
|
|
51
51
|
|
|
52
|
-
### Document Pipeline (
|
|
52
|
+
### Document Pipeline (9)
|
|
53
53
|
| Tool | Description | Annotation |
|
|
54
54
|
|------|------------|------------|
|
|
55
55
|
| `storeDocument` | Store documents with metadata | idempotent |
|
|
@@ -60,11 +60,12 @@ Place this `.mcp.json` in each project folder with its own `DB_FILE_PATH`. Each
|
|
|
60
60
|
| `linkEntitiesToDocument` | Link entities to chunks where they actually appear (text-matched) | idempotent |
|
|
61
61
|
| `deleteDocuments` | Remove documents and associated data | destructive |
|
|
62
62
|
| `listDocuments` | View all stored documents | readOnly |
|
|
63
|
+
| `syncDocumentFromFile` | One-call server-side sync: reads file + delete/store/chunk/embed/link, content stays off model context. Atomic (embed-first transaction swap) + `content_hash` dedup (skips unchanged files) | idempotent |
|
|
63
64
|
|
|
64
65
|
### Search & Retrieval (9)
|
|
65
66
|
| Tool | Description | Annotation |
|
|
66
67
|
|------|------------|------------|
|
|
67
|
-
| `hybridSearch` | Vector + FTS5 BM25 + graph traversal (3-signal) | readOnly |
|
|
68
|
+
| `hybridSearch` | Vector + FTS5 BM25 + graph traversal (3-signal). Degrades to FTS5-only (`search_mode`) when the embedding model is down | readOnly |
|
|
68
69
|
| `searchNodes` | Semantic entity search with `since`/`until` temporal filtering | readOnly |
|
|
69
70
|
| `openNodes` | Retrieve specific entities by name | readOnly |
|
|
70
71
|
| `readGraph` | Get complete knowledge graph | readOnly |
|
|
@@ -136,6 +137,14 @@ storeDocument(id, content, metadata)
|
|
|
136
137
|
|
|
137
138
|
## Changelog
|
|
138
139
|
|
|
140
|
+
### v3.5.0
|
|
141
|
+
- **Atomic `syncDocumentFromFile`**: embeddings are computed before any DB write, then applied in a single synchronous transaction, so a failed embedding (e.g. model still loading) leaves the existing document fully intact instead of a half-deleted or partially-embedded state.
|
|
142
|
+
- **`content_hash` dedup**: unchanged files short-circuit the delete/chunk/embed pipeline (`skipped: true`), with a completeness gate that still re-processes a partially-embedded document.
|
|
143
|
+
- **FTS5-only graceful degradation**: when the embedding model is unavailable, `hybridSearch` returns BM25 (full-text) results tagged `search_mode: 'fts-only'` instead of failing the whole query.
|
|
144
|
+
|
|
145
|
+
### v3.4.0
|
|
146
|
+
- **`syncDocumentFromFile`: one-call server-side document sync** - reads a file on the server and runs the full pipeline (`deleteDocuments` → `storeDocument` → `chunkDocument` → `embedChunks` → `linkEntitiesToDocument`) in a single call, returning only a terse summary `{ documentId, bytes, chunks, embeddedChunks, linkedEntities, warning? }`. File content is read server-side and never routed through the model context, collapsing the usual 5 tool calls per document into one and keeping conversation context flat (the dominant cost of large sync runs). 30 → 31 tools.
|
|
147
|
+
|
|
139
148
|
### v3.3.6
|
|
140
149
|
- **Publish-time invariant test** — `npm run verify:invariants` (wired as `prepublishOnly`) catches `chunkText` offset regressions before they ship. Tests ASCII / Korean / emoji-heavy / mixed CJK + supplementary plane / pure supplementary inputs against the codepoint-slice contract.
|
|
141
150
|
- **`chunkText` extracted to `src/chunkText.ts`** — algorithm now testable in isolation. The class method is a thin wrapper. No user-facing API change.
|
package/dist/index.d.ts
CHANGED
|
@@ -1,2 +1,295 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
|
+
interface Entity {
|
|
3
|
+
name: string;
|
|
4
|
+
entityType: string;
|
|
5
|
+
observations: string[];
|
|
6
|
+
}
|
|
7
|
+
interface Relation {
|
|
8
|
+
from: string;
|
|
9
|
+
to: string;
|
|
10
|
+
relationType: string;
|
|
11
|
+
}
|
|
12
|
+
interface KnowledgeGraph {
|
|
13
|
+
entities: Entity[];
|
|
14
|
+
relations: Relation[];
|
|
15
|
+
}
|
|
16
|
+
interface EnhancedSearchResult {
|
|
17
|
+
relevance_score: number;
|
|
18
|
+
key_highlight: string;
|
|
19
|
+
content_summary: string;
|
|
20
|
+
chunk_id: string;
|
|
21
|
+
document_title: string;
|
|
22
|
+
entities: string[];
|
|
23
|
+
vector_similarity: number;
|
|
24
|
+
graph_boost?: number;
|
|
25
|
+
fts_boost?: number;
|
|
26
|
+
full_context_available: boolean;
|
|
27
|
+
chunk_type: 'document' | 'entity' | 'relationship';
|
|
28
|
+
source_id?: string;
|
|
29
|
+
search_mode?: 'hybrid' | 'fts-only';
|
|
30
|
+
}
|
|
31
|
+
interface DetailedContext {
|
|
32
|
+
chunk_id: string;
|
|
33
|
+
document_id: string;
|
|
34
|
+
full_text: string;
|
|
35
|
+
document_title: string;
|
|
36
|
+
surrounding_chunks?: Array<{
|
|
37
|
+
chunk_id: string;
|
|
38
|
+
text: string;
|
|
39
|
+
position: 'before' | 'after';
|
|
40
|
+
}>;
|
|
41
|
+
entities: string[];
|
|
42
|
+
metadata: Record<string, any>;
|
|
43
|
+
}
|
|
44
|
+
export declare class RAGKnowledgeGraphManager {
|
|
45
|
+
private db;
|
|
46
|
+
private encoding;
|
|
47
|
+
private embeddingModel;
|
|
48
|
+
private modelInitialized;
|
|
49
|
+
private embeddingCache;
|
|
50
|
+
private readonly EMBEDDING_CACHE_MAX;
|
|
51
|
+
private dictionaryCache;
|
|
52
|
+
initialize(opts?: {
|
|
53
|
+
skipModel?: boolean;
|
|
54
|
+
}): Promise<void>;
|
|
55
|
+
private initializeEmbeddingModel;
|
|
56
|
+
runMigrations(): Promise<{
|
|
57
|
+
applied: number;
|
|
58
|
+
currentVersion: number;
|
|
59
|
+
appliedMigrations: Array<{
|
|
60
|
+
version: number;
|
|
61
|
+
description: string;
|
|
62
|
+
}>;
|
|
63
|
+
}>;
|
|
64
|
+
cleanup(): void;
|
|
65
|
+
private _timestampObservation;
|
|
66
|
+
createEntities(entities: Entity[]): Promise<Entity[]>;
|
|
67
|
+
createRelations(relations: Relation[]): Promise<Relation[]>;
|
|
68
|
+
addObservations(observations: {
|
|
69
|
+
entityName: string;
|
|
70
|
+
contents: string[];
|
|
71
|
+
}[]): Promise<{
|
|
72
|
+
entityName: string;
|
|
73
|
+
addedObservations: string[];
|
|
74
|
+
}[]>;
|
|
75
|
+
deleteEntities(entityNames: string[]): Promise<void>;
|
|
76
|
+
deleteObservations(deletions: {
|
|
77
|
+
entityName: string;
|
|
78
|
+
observations: string[];
|
|
79
|
+
}[]): Promise<void>;
|
|
80
|
+
deleteRelations(relations: Relation[]): Promise<void>;
|
|
81
|
+
updateRelations(updates: {
|
|
82
|
+
from: string;
|
|
83
|
+
to: string;
|
|
84
|
+
relationType: string;
|
|
85
|
+
confidence?: number;
|
|
86
|
+
metadata?: Record<string, any>;
|
|
87
|
+
}[]): Promise<{
|
|
88
|
+
updated: number;
|
|
89
|
+
notFound: number;
|
|
90
|
+
}>;
|
|
91
|
+
readGraph(): Promise<KnowledgeGraph>;
|
|
92
|
+
getNeighbors(entityNames: string[], depth?: number, relationType?: string): Promise<{
|
|
93
|
+
entities: Array<{
|
|
94
|
+
name: string;
|
|
95
|
+
entityType: string;
|
|
96
|
+
observations: string[];
|
|
97
|
+
depth: number;
|
|
98
|
+
}>;
|
|
99
|
+
relations: Array<{
|
|
100
|
+
from: string;
|
|
101
|
+
to: string;
|
|
102
|
+
relationType: string;
|
|
103
|
+
depth: number;
|
|
104
|
+
}>;
|
|
105
|
+
paths: Array<{
|
|
106
|
+
from: string;
|
|
107
|
+
to: string;
|
|
108
|
+
path: string[];
|
|
109
|
+
}>;
|
|
110
|
+
}>;
|
|
111
|
+
searchNodes(query: string, limit?: number, since?: string, until?: string): Promise<KnowledgeGraph>;
|
|
112
|
+
openNodes(names: string[]): Promise<KnowledgeGraph>;
|
|
113
|
+
private generateEntityEmbeddingText;
|
|
114
|
+
private splitIntoSentences;
|
|
115
|
+
private calculateSentenceSimilarities;
|
|
116
|
+
private cosineSimilarity;
|
|
117
|
+
private enhanceSimilarityWithContext;
|
|
118
|
+
private generateContentSummary;
|
|
119
|
+
private embedEntity;
|
|
120
|
+
embedAllEntities(): Promise<{
|
|
121
|
+
totalEntities: number;
|
|
122
|
+
embeddedEntities: number;
|
|
123
|
+
}>;
|
|
124
|
+
generateKnowledgeGraphChunks(): Promise<{
|
|
125
|
+
entityChunks: number;
|
|
126
|
+
relationshipChunks: number;
|
|
127
|
+
}>;
|
|
128
|
+
embedKnowledgeGraphChunks(): Promise<{
|
|
129
|
+
embeddedChunks: number;
|
|
130
|
+
totalChunks: number;
|
|
131
|
+
errors?: string[];
|
|
132
|
+
}>;
|
|
133
|
+
private generateEntityChunkText;
|
|
134
|
+
private generateRelationshipChunkText;
|
|
135
|
+
private cleanupKnowledgeGraphChunks;
|
|
136
|
+
private loadDictionary;
|
|
137
|
+
private hasKorean;
|
|
138
|
+
private isLikelyEnglish;
|
|
139
|
+
private normalizeQueryText;
|
|
140
|
+
private buildCrossLingualDictionary;
|
|
141
|
+
private translateQueryWithMap;
|
|
142
|
+
private buildCrossLingualVariants;
|
|
143
|
+
private extractTermsFromText;
|
|
144
|
+
private chunkText;
|
|
145
|
+
private generateEmbedding;
|
|
146
|
+
syncDocumentFromFile(filePath: string, documentId: string, options?: {
|
|
147
|
+
metadata?: Record<string, any>;
|
|
148
|
+
content?: string;
|
|
149
|
+
entityNames?: string[];
|
|
150
|
+
chunkParams?: {
|
|
151
|
+
maxTokens?: number;
|
|
152
|
+
overlap?: number;
|
|
153
|
+
};
|
|
154
|
+
}): Promise<{
|
|
155
|
+
documentId: string;
|
|
156
|
+
bytes: number;
|
|
157
|
+
chunks: number;
|
|
158
|
+
embeddedChunks: number;
|
|
159
|
+
linkedEntities: number;
|
|
160
|
+
explicitlyLinked?: number;
|
|
161
|
+
warning?: string;
|
|
162
|
+
skipped?: boolean;
|
|
163
|
+
reason?: string;
|
|
164
|
+
}>;
|
|
165
|
+
storeDocument(id: string, content: string, metadata?: Record<string, any>): Promise<{
|
|
166
|
+
id: string;
|
|
167
|
+
stored: boolean;
|
|
168
|
+
}>;
|
|
169
|
+
chunkDocument(documentId: string, options?: {
|
|
170
|
+
maxTokens?: number;
|
|
171
|
+
overlap?: number;
|
|
172
|
+
}): Promise<{
|
|
173
|
+
documentId: string;
|
|
174
|
+
chunks: Array<{
|
|
175
|
+
id: string;
|
|
176
|
+
text: string;
|
|
177
|
+
startPos: number | null;
|
|
178
|
+
endPos: number | null;
|
|
179
|
+
startToken: number | null;
|
|
180
|
+
endToken: number | null;
|
|
181
|
+
}>;
|
|
182
|
+
}>;
|
|
183
|
+
embedChunks(documentId: string): Promise<{
|
|
184
|
+
documentId: string;
|
|
185
|
+
embeddedChunks: number;
|
|
186
|
+
totalChunks: number;
|
|
187
|
+
linkedEntities?: number;
|
|
188
|
+
errors?: string[];
|
|
189
|
+
}>;
|
|
190
|
+
private hasCJK;
|
|
191
|
+
private buildEntityMatcher;
|
|
192
|
+
private autoLinkEntities;
|
|
193
|
+
extractTerms(documentId: string, options?: {
|
|
194
|
+
minLength?: number;
|
|
195
|
+
includeCapitalized?: boolean;
|
|
196
|
+
customPatterns?: string[];
|
|
197
|
+
}): Promise<{
|
|
198
|
+
documentId: string;
|
|
199
|
+
terms: string[];
|
|
200
|
+
}>;
|
|
201
|
+
linkEntitiesToDocument(documentId: string, entityNames: string[]): Promise<{
|
|
202
|
+
documentId: string;
|
|
203
|
+
linkedEntities: number;
|
|
204
|
+
}>;
|
|
205
|
+
private cleanupDocument;
|
|
206
|
+
deleteDocument(documentId: string): Promise<{
|
|
207
|
+
documentId: string;
|
|
208
|
+
deleted: boolean;
|
|
209
|
+
}>;
|
|
210
|
+
deleteMultipleDocuments(documentIds: string[]): Promise<{
|
|
211
|
+
results: Array<{
|
|
212
|
+
documentId: string;
|
|
213
|
+
deleted: boolean;
|
|
214
|
+
}>;
|
|
215
|
+
summary: {
|
|
216
|
+
deleted: number;
|
|
217
|
+
failed: number;
|
|
218
|
+
total: number;
|
|
219
|
+
};
|
|
220
|
+
}>;
|
|
221
|
+
deleteDocuments(documentIds: string | string[]): Promise<{
|
|
222
|
+
results: Array<{
|
|
223
|
+
documentId: string;
|
|
224
|
+
deleted: boolean;
|
|
225
|
+
}>;
|
|
226
|
+
summary: {
|
|
227
|
+
deleted: number;
|
|
228
|
+
failed: number;
|
|
229
|
+
total: number;
|
|
230
|
+
};
|
|
231
|
+
}>;
|
|
232
|
+
listDocuments(includeMetadata?: boolean): Promise<{
|
|
233
|
+
documents: Array<{
|
|
234
|
+
id: string;
|
|
235
|
+
metadata?: any;
|
|
236
|
+
created_at: string;
|
|
237
|
+
}>;
|
|
238
|
+
}>;
|
|
239
|
+
exportGraph(): Promise<{
|
|
240
|
+
entities: any[];
|
|
241
|
+
relations: any[];
|
|
242
|
+
documents: any[];
|
|
243
|
+
metadata: {
|
|
244
|
+
exportedAt: string;
|
|
245
|
+
version: string;
|
|
246
|
+
entityCount: number;
|
|
247
|
+
relationCount: number;
|
|
248
|
+
documentCount: number;
|
|
249
|
+
};
|
|
250
|
+
}>;
|
|
251
|
+
importGraph(data: {
|
|
252
|
+
entities?: any[];
|
|
253
|
+
relations?: any[];
|
|
254
|
+
documents?: any[];
|
|
255
|
+
}, options?: {
|
|
256
|
+
merge?: boolean;
|
|
257
|
+
}): Promise<{
|
|
258
|
+
imported: {
|
|
259
|
+
entities: number;
|
|
260
|
+
relations: number;
|
|
261
|
+
documents: number;
|
|
262
|
+
};
|
|
263
|
+
skipped: {
|
|
264
|
+
entities: number;
|
|
265
|
+
relations: number;
|
|
266
|
+
documents: number;
|
|
267
|
+
};
|
|
268
|
+
}>;
|
|
269
|
+
hybridSearch(query: string, limit?: number, useGraph?: boolean): Promise<EnhancedSearchResult[]>;
|
|
270
|
+
getDetailedContext(chunkId: string, includeSurrounding?: boolean): Promise<DetailedContext>;
|
|
271
|
+
getKnowledgeGraphStats(): Promise<any>;
|
|
272
|
+
private _buildGraphologyGraph;
|
|
273
|
+
getGraphMetrics(entityNames?: string[], metrics?: string[], limit?: number): Promise<any>;
|
|
274
|
+
detectCommunities(resolution?: number): Promise<any>;
|
|
275
|
+
analyzeGraphStructure(): Promise<any>;
|
|
276
|
+
getMigrationStatus(): Promise<{
|
|
277
|
+
currentVersion: number;
|
|
278
|
+
migrations: Array<{
|
|
279
|
+
version: number;
|
|
280
|
+
description: string;
|
|
281
|
+
applied: boolean;
|
|
282
|
+
applied_at?: string;
|
|
283
|
+
}>;
|
|
284
|
+
pendingCount: number;
|
|
285
|
+
}>;
|
|
286
|
+
rollbackMigration(targetVersion: number): Promise<{
|
|
287
|
+
rolledBack: number;
|
|
288
|
+
currentVersion: number;
|
|
289
|
+
rolledBackMigrations: Array<{
|
|
290
|
+
version: number;
|
|
291
|
+
description: string;
|
|
292
|
+
}>;
|
|
293
|
+
}>;
|
|
294
|
+
}
|
|
2
295
|
export {};
|
package/dist/index.js
CHANGED
|
@@ -56,7 +56,7 @@ function safeRowid(value) {
|
|
|
56
56
|
return n;
|
|
57
57
|
}
|
|
58
58
|
// Enhanced RAG-enabled Knowledge Graph Manager
|
|
59
|
-
class RAGKnowledgeGraphManager {
|
|
59
|
+
export class RAGKnowledgeGraphManager {
|
|
60
60
|
db = null;
|
|
61
61
|
encoding = null;
|
|
62
62
|
embeddingModel = null;
|
|
@@ -64,7 +64,7 @@ class RAGKnowledgeGraphManager {
|
|
|
64
64
|
embeddingCache = new Map();
|
|
65
65
|
EMBEDDING_CACHE_MAX = 500;
|
|
66
66
|
dictionaryCache = null;
|
|
67
|
-
async initialize() {
|
|
67
|
+
async initialize(opts = {}) {
|
|
68
68
|
console.error('🚀 Initializing RAG Knowledge Graph MCP Server...');
|
|
69
69
|
// Initialize database
|
|
70
70
|
this.db = new Database(DB_FILE_PATH);
|
|
@@ -80,8 +80,13 @@ class RAGKnowledgeGraphManager {
|
|
|
80
80
|
this.db.pragma('foreign_keys = ON');
|
|
81
81
|
// Initialize tiktoken
|
|
82
82
|
this.encoding = get_encoding("cl100k_base");
|
|
83
|
-
// Initialize embedding model
|
|
84
|
-
|
|
83
|
+
// Initialize embedding model (skippable for tests / FTS-only environments)
|
|
84
|
+
if (!opts.skipModel) {
|
|
85
|
+
await this.initializeEmbeddingModel();
|
|
86
|
+
}
|
|
87
|
+
else {
|
|
88
|
+
console.error('⏭️ Skipping embedding model load (skipModel=true)');
|
|
89
|
+
}
|
|
85
90
|
// Run database migrations
|
|
86
91
|
await this.runMigrations();
|
|
87
92
|
console.error('✅ RAG-enabled knowledge graph initialized');
|
|
@@ -1179,28 +1184,87 @@ class RAGKnowledgeGraphManager {
|
|
|
1179
1184
|
? options.content
|
|
1180
1185
|
: fsSync.readFileSync(filePath, 'utf-8');
|
|
1181
1186
|
const bytes = Buffer.byteLength(content, 'utf-8');
|
|
1182
|
-
// 2. Metadata: default source=path, updated=today
|
|
1187
|
+
// 2. Metadata: default source=path, updated=today, content_hash; caller can override.
|
|
1183
1188
|
const today = new Date().toISOString().slice(0, 10);
|
|
1184
|
-
const
|
|
1189
|
+
const contentHash = createHash('sha256').update(content).digest('hex');
|
|
1190
|
+
const metadata = { source: filePath, updated: today, content_hash: contentHash, ...(options.metadata || {}) };
|
|
1191
|
+
// 2b. Dedup gate: skip the full delete/store/chunk/embed pipeline when the
|
|
1192
|
+
// file is unchanged AND the existing document is fully embedded. The
|
|
1193
|
+
// completeness check avoids wrongly skipping a partial/failed prior sync.
|
|
1194
|
+
const existingDoc = this.db.prepare(`SELECT metadata FROM documents WHERE id = ?`).get(documentId);
|
|
1195
|
+
if (existingDoc) {
|
|
1196
|
+
let existingHash;
|
|
1197
|
+
try {
|
|
1198
|
+
existingHash = JSON.parse(existingDoc.metadata)?.content_hash;
|
|
1199
|
+
}
|
|
1200
|
+
catch { /* ignore */ }
|
|
1201
|
+
if (existingHash === contentHash) {
|
|
1202
|
+
const cmCount = this.db.prepare(`SELECT count(*) AS n FROM chunk_metadata WHERE document_id = ?`).get(documentId).n;
|
|
1203
|
+
const embCount = this.db.prepare(`
|
|
1204
|
+
SELECT count(*) AS n FROM chunks c JOIN chunk_metadata m ON c.rowid = m.rowid WHERE m.document_id = ?
|
|
1205
|
+
`).get(documentId).n;
|
|
1206
|
+
if (cmCount > 0 && cmCount === embCount) {
|
|
1207
|
+
const linked = this.db.prepare(`
|
|
1208
|
+
SELECT count(DISTINCT ce.entity_id) AS n FROM chunk_entities ce
|
|
1209
|
+
JOIN chunk_metadata m ON ce.chunk_rowid = m.rowid WHERE m.document_id = ?
|
|
1210
|
+
`).get(documentId).n;
|
|
1211
|
+
console.error(`⏭️ syncDocumentFromFile: ${documentId} unchanged (hash match, ${cmCount} chunks embedded) — skipped`);
|
|
1212
|
+
return { documentId, bytes, chunks: cmCount, embeddedChunks: embCount, linkedEntities: linked, skipped: true, reason: 'unchanged' };
|
|
1213
|
+
}
|
|
1214
|
+
}
|
|
1215
|
+
}
|
|
1185
1216
|
console.error(`🔄 syncDocumentFromFile: ${documentId} <- ${filePath} (${bytes} bytes)`);
|
|
1186
|
-
// 3.
|
|
1187
|
-
|
|
1188
|
-
|
|
1189
|
-
const
|
|
1190
|
-
const
|
|
1191
|
-
|
|
1217
|
+
// 3. Pre-compute chunks + embeddings BEFORE any DB mutation. If embedding
|
|
1218
|
+
// throws (model down), the existing document is left completely intact.
|
|
1219
|
+
const { maxTokens = 800, overlap = 160 } = options.chunkParams || {};
|
|
1220
|
+
const segments = this.chunkText(content, maxTokens, overlap);
|
|
1221
|
+
const embedded = [];
|
|
1222
|
+
for (const seg of segments) {
|
|
1223
|
+
const embedding = await this.generateEmbedding(seg.text);
|
|
1224
|
+
embedded.push({ seg, embedding });
|
|
1225
|
+
}
|
|
1226
|
+
// 4. Atomic swap: delete old -> insert doc -> insert chunks + embeddings,
|
|
1227
|
+
// all in a single synchronous better-sqlite3 transaction (all-or-nothing).
|
|
1228
|
+
const applyTx = this.db.transaction(() => {
|
|
1229
|
+
const db = this.db;
|
|
1230
|
+
// 4a. cleanup old doc (inlined sync version of cleanupDocument).
|
|
1231
|
+
const existing = db.prepare(`SELECT rowid FROM chunk_metadata WHERE document_id = ?`).all(documentId);
|
|
1232
|
+
for (const ch of existing) {
|
|
1233
|
+
db.prepare(`DELETE FROM chunk_entities WHERE chunk_rowid = ?`).run(ch.rowid);
|
|
1234
|
+
db.exec(`DELETE FROM chunks WHERE rowid = ${safeRowid(ch.rowid)}`);
|
|
1235
|
+
}
|
|
1236
|
+
db.prepare(`DELETE FROM chunk_metadata WHERE document_id = ?`).run(documentId);
|
|
1237
|
+
db.prepare(`DELETE FROM documents WHERE id = ?`).run(documentId);
|
|
1238
|
+
// 4b. insert document.
|
|
1239
|
+
db.prepare(`INSERT INTO documents (id, content, metadata) VALUES (?, ?, ?)`)
|
|
1240
|
+
.run(documentId, content, JSON.stringify(metadata));
|
|
1241
|
+
// 4c. insert chunk_metadata (FTS5 chunks_fts auto-filled by trigger) + embeddings.
|
|
1242
|
+
for (const { seg, embedding } of embedded) {
|
|
1243
|
+
const chunkId = `${documentId}_chunk_${seg.chunk_index}`;
|
|
1244
|
+
const info = db.prepare(`
|
|
1245
|
+
INSERT INTO chunk_metadata (chunk_id, document_id, chunk_index, text, start_pos, end_pos, start_token, end_token)
|
|
1246
|
+
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
|
1247
|
+
`).run(chunkId, documentId, seg.chunk_index, seg.text, seg.start_pos, seg.end_pos, seg.start_token, seg.end_token);
|
|
1248
|
+
const rowid = Number(info.lastInsertRowid);
|
|
1249
|
+
db.prepare(`INSERT INTO chunks (rowid, embedding) VALUES (${rowid}, ?)`).run(Buffer.from(embedding.buffer));
|
|
1250
|
+
}
|
|
1251
|
+
});
|
|
1252
|
+
applyTx();
|
|
1253
|
+
const embeddedChunks = embedded.length;
|
|
1254
|
+
// 5. Entity linking AFTER commit. Non-destructive + idempotent (INSERT OR
|
|
1255
|
+
// IGNORE), so a linking failure cannot corrupt the doc/embeddings.
|
|
1256
|
+
const linkedEntities = await this.autoLinkEntities(documentId);
|
|
1192
1257
|
let explicitlyLinked;
|
|
1193
1258
|
if (options.entityNames && options.entityNames.length > 0) {
|
|
1194
1259
|
const linkResult = await this.linkEntitiesToDocument(documentId, options.entityNames);
|
|
1195
1260
|
explicitlyLinked = linkResult.linkedEntities;
|
|
1196
1261
|
}
|
|
1197
|
-
|
|
1198
|
-
// 5. Terse summary only (no chunk text / content echo) to keep caller context flat.
|
|
1262
|
+
// 6. Terse summary only (no chunk text / content echo) to keep caller context flat.
|
|
1199
1263
|
const result = {
|
|
1200
1264
|
documentId,
|
|
1201
1265
|
bytes,
|
|
1202
|
-
chunks:
|
|
1203
|
-
embeddedChunks
|
|
1266
|
+
chunks: segments.length,
|
|
1267
|
+
embeddedChunks,
|
|
1204
1268
|
linkedEntities,
|
|
1205
1269
|
...(explicitlyLinked !== undefined ? { explicitlyLinked } : {}),
|
|
1206
1270
|
};
|
|
@@ -1716,7 +1780,8 @@ class RAGKnowledgeGraphManager {
|
|
|
1716
1780
|
if (queryVariants.length > 1) {
|
|
1717
1781
|
console.error(`🌐 Cross-lingual variants: ${queryVariants.slice(1).join(' | ')}`);
|
|
1718
1782
|
}
|
|
1719
|
-
|
|
1783
|
+
let vectorDegraded = false;
|
|
1784
|
+
let primaryQueryEmbedding = null;
|
|
1720
1785
|
// Vector search helper
|
|
1721
1786
|
const searchChunks = (embedding, k) => {
|
|
1722
1787
|
return this.db.prepare(`
|
|
@@ -1746,16 +1811,23 @@ class RAGKnowledgeGraphManager {
|
|
|
1746
1811
|
};
|
|
1747
1812
|
// Search original query plus cross-lingual expansions and keep best match per chunk.
|
|
1748
1813
|
const resultMap = new Map();
|
|
1749
|
-
|
|
1750
|
-
|
|
1751
|
-
|
|
1752
|
-
|
|
1753
|
-
const
|
|
1754
|
-
|
|
1755
|
-
resultMap.
|
|
1814
|
+
try {
|
|
1815
|
+
primaryQueryEmbedding = await this.generateEmbedding(queryVariants[0], 1024, true);
|
|
1816
|
+
for (const variant of queryVariants) {
|
|
1817
|
+
const embedding = await this.generateEmbedding(variant, 1024, true);
|
|
1818
|
+
const variantResults = searchChunks(embedding, limit * 3);
|
|
1819
|
+
for (const r of variantResults) {
|
|
1820
|
+
const existing = resultMap.get(r.chunk_id);
|
|
1821
|
+
if (!existing || r.distance < existing.distance) {
|
|
1822
|
+
resultMap.set(r.chunk_id, r);
|
|
1823
|
+
}
|
|
1756
1824
|
}
|
|
1757
1825
|
}
|
|
1758
1826
|
}
|
|
1827
|
+
catch (embErr) {
|
|
1828
|
+
vectorDegraded = true;
|
|
1829
|
+
console.error(`⚠️ Vector search unavailable (embedding model down) — degrading to FTS5-only:`, embErr instanceof Error ? embErr.message : embErr);
|
|
1830
|
+
}
|
|
1759
1831
|
const vectorResults = Array.from(resultMap.values()).sort((a, b) => a.distance - b.distance);
|
|
1760
1832
|
// FTS5 full-text search as additional signal (Reciprocal Rank Fusion)
|
|
1761
1833
|
const ftsBoostMap = new Map();
|
|
@@ -1843,7 +1915,7 @@ class RAGKnowledgeGraphManager {
|
|
|
1843
1915
|
// Get entity information for graph enhancement via vector similarity
|
|
1844
1916
|
let connectedEntities = new Set();
|
|
1845
1917
|
let queryMatchedEntities = new Set();
|
|
1846
|
-
if (useGraph) {
|
|
1918
|
+
if (useGraph && !vectorDegraded) {
|
|
1847
1919
|
// Vector search: find entities semantically similar to the query (dual search)
|
|
1848
1920
|
try {
|
|
1849
1921
|
const searchEntities = (embedding) => {
|
|
@@ -1949,7 +2021,7 @@ class RAGKnowledgeGraphManager {
|
|
|
1949
2021
|
}
|
|
1950
2022
|
// Enhanced graph boost calculation with decay + cap
|
|
1951
2023
|
let graphBoost = 0;
|
|
1952
|
-
if (useGraph) {
|
|
2024
|
+
if (useGraph && !vectorDegraded) {
|
|
1953
2025
|
const queryEntities = this.extractTermsFromText(query);
|
|
1954
2026
|
// Base boost for knowledge graph chunks
|
|
1955
2027
|
if (result.chunk_type === 'entity') {
|
|
@@ -1996,9 +2068,17 @@ class RAGKnowledgeGraphManager {
|
|
|
1996
2068
|
// Hard cap to prevent graph domination
|
|
1997
2069
|
graphBoost += Math.min(entityBoost, 0.4);
|
|
1998
2070
|
}
|
|
1999
|
-
// Generate semantic summary
|
|
2000
|
-
|
|
2001
|
-
)
|
|
2071
|
+
// Generate semantic summary (skip when degraded — no embeddings available).
|
|
2072
|
+
let summary, keyHighlight, relevanceScore;
|
|
2073
|
+
if (vectorDegraded || !primaryQueryEmbedding) {
|
|
2074
|
+
keyHighlight = result.text.slice(0, 150);
|
|
2075
|
+
summary = result.text.slice(0, 300);
|
|
2076
|
+
relevanceScore = 0;
|
|
2077
|
+
}
|
|
2078
|
+
else {
|
|
2079
|
+
({ summary, keyHighlight, relevanceScore } = await this.generateContentSummary(result.text, primaryQueryEmbedding, chunkEntities, result.chunk_type === 'relationship' ? 1 : 2 // Shorter summary for relationships
|
|
2080
|
+
));
|
|
2081
|
+
}
|
|
2002
2082
|
const vectorSimilarity = Math.max(0, 1 - result.distance / 2);
|
|
2003
2083
|
const ftsBoost = ftsBoostMap.get(result.chunk_id) || 0;
|
|
2004
2084
|
const finalScore = Math.max(vectorSimilarity, relevanceScore) + graphBoost + ftsBoost;
|
|
@@ -2030,11 +2110,12 @@ class RAGKnowledgeGraphManager {
|
|
|
2030
2110
|
document_title: documentTitle,
|
|
2031
2111
|
entities: chunkEntities,
|
|
2032
2112
|
vector_similarity: vectorSimilarity,
|
|
2033
|
-
graph_boost: useGraph ? graphBoost : undefined,
|
|
2113
|
+
graph_boost: (useGraph && !vectorDegraded) ? graphBoost : undefined,
|
|
2034
2114
|
fts_boost: ftsBoost > 0 ? ftsBoost : undefined,
|
|
2035
2115
|
full_context_available: true,
|
|
2036
2116
|
chunk_type: result.chunk_type,
|
|
2037
|
-
source_id: sourceId
|
|
2117
|
+
source_id: sourceId,
|
|
2118
|
+
search_mode: vectorDegraded ? 'fts-only' : 'hybrid'
|
|
2038
2119
|
});
|
|
2039
2120
|
}
|
|
2040
2121
|
// Sort by relevance and return top results
|
|
@@ -2579,8 +2660,12 @@ async function main() {
|
|
|
2579
2660
|
process.exit(1);
|
|
2580
2661
|
}
|
|
2581
2662
|
}
|
|
2582
|
-
|
|
2583
|
-
|
|
2584
|
-
|
|
2585
|
-
|
|
2586
|
-
|
|
2663
|
+
// Only boot the server when run as the entry point — not when imported (tests).
|
|
2664
|
+
const isDirectRun = process.argv[1] === fileURLToPath(import.meta.url);
|
|
2665
|
+
if (isDirectRun) {
|
|
2666
|
+
main().catch((error) => {
|
|
2667
|
+
console.error("Fatal error in main():", error);
|
|
2668
|
+
ragKgManager.cleanup();
|
|
2669
|
+
process.exit(1);
|
|
2670
|
+
});
|
|
2671
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "rag-memory-epf-mcp",
|
|
3
|
-
"version": "3.
|
|
3
|
+
"version": "3.5.0",
|
|
4
4
|
"description": "Project-local RAG memory MCP server — knowledge graph + multilingual vector + FTS5 in a single SQLite file. Per-project isolation, 30 MCP tools, codepoint-safe chunking (Korean/CJK/emoji).",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"mcp",
|
|
@@ -41,8 +41,9 @@
|
|
|
41
41
|
"prepare": "npm run build",
|
|
42
42
|
"watch": "tsc --watch",
|
|
43
43
|
"verify:invariants": "node test/chunk-invariants.test.mjs",
|
|
44
|
-
"
|
|
45
|
-
"
|
|
44
|
+
"verify:engine": "node test/engine-smoke.test.mjs && node test/sync-atomicity.test.mjs && node test/dedup.test.mjs && node test/search-degradation.test.mjs",
|
|
45
|
+
"test": "npm run build && npm run verify:invariants && npm run verify:engine",
|
|
46
|
+
"prepublishOnly": "npm run build && npm run verify:invariants && npm run verify:engine"
|
|
46
47
|
},
|
|
47
48
|
"dependencies": {
|
|
48
49
|
"@huggingface/transformers": "^3.5.1",
|
package/dist/rag-memory.db
DELETED
|
Binary file
|
package/dist/rag-memory.db-shm
DELETED
|
Binary file
|
package/dist/rag-memory.db-wal
DELETED
|
Binary file
|