rag-memory-epf-mcp 5.1.0 → 5.3.0-rc.1

package/README.md

@@ -10,9 +10,9 @@ A **project-local RAG memory** MCP server — knowledge graph + multilingual vec
 ## Key Features
 
 - **Project-local isolation** — each project gets its own `.memory/rag-memory.db`. Multiple projects run simultaneously without interference.
-- **3-signal hybrid search** — vector similarity (bge-m3, 1024-dim) + FTS5 BM25 keyword matching + knowledge graph re-ranking, combined via Reciprocal Rank Fusion
+- **Hybrid search** — vector similarity (bge-m3, 1024-dim) + FTS5 BM25 keyword matching (RRF-fused). Knowledge-graph re-ranking is an **opt-in legacy/experimental** signal since v5.3.0 (measured to hurt known-item retrieval)
 - **100+ languages** — Korean, Chinese, Japanese, Arabic, and more. Cross-lingual search works out of the box.
-- **Graph-aware scoring** — per-entity geometric decay (0.5^i) with hard cap prevents any single document from dominating results
+- **Graph re-ranker (opt-in)** — per-entity geometric decay (0.5^i) with a hard cap 0.4; the cap bounds the boost but was measured (2026-08-17) to let heavily-linked chunks saturate it and outrank the exact chunk — hence off by default
 - **38 MCP tools** — knowledge graph CRUD, observation lifecycle (correct / retract / history), document pipeline, hybrid search, multi-hop traversal, graph analytics (centrality / community detection / structure), export/import, temporal queries
 - **Observations that hold their history** — corrections supersede instead of overwrite, search returns only current facts, and every revision keeps its provenance
 - **Structure-anchored chunking (c1, v5)** — boundaries anchor to markdown structure (fence-aware, H1–H4 first, block-greedy, exact-token fallback), so editing the top of a file no longer re-embeds the whole document: unchanged text reuses its stored vectors at sync time. Chunk offsets are Unicode codepoints, language-neutral across SQL `substr`, Python slicing, and JS `[...str]` iteration; a publish-time invariant gate locks the gap-free partition. `overlap` is retired (omit or 0).
@@ -77,7 +77,7 @@ Place this `.mcp.json` in each project folder with its own `DB_FILE_PATH`. Each
 ### Search & Retrieval (9)
 | Tool | Description | Annotation |
 |------|------------|------------|
-| `hybridSearch` | Vector + FTS5 BM25 + graph traversal (3-signal). Degrades to FTS5-only (`search_mode`) when the embedding model is down | readOnly |
+| `hybridSearch` | Vector + FTS5 BM25, plus an **opt-in** graph re-ranker (`useGraph: true`; default off since v5.3.0). Degrades to FTS5-only (`search_mode`) when the embedding model is down | readOnly |
 | `searchNodes` | Semantic entity search with `since`/`until` temporal filtering | readOnly |
 | `openNodes` | Retrieve specific entities by name | readOnly |
 | `readGraph` | Get complete knowledge graph | readOnly |
@@ -153,6 +153,10 @@ storeDocument(id, content, metadata)
 
 ## Changelog
 
+### v5.3.0-rc.1
+- **Behavior change — `hybridSearch` graph re-ranking is now opt-in** (`useGraph` default `true` → `false`; tool schema, MCP exposure and the manager signature agree). Omitting the argument now means "no graph re-ranking" — a behavior change for callers that relied on the old default, hence a release-candidate first (`next` dist-tag, fleet canary) before stable. Measured 2026-08-17 on three real corpora (self-retrieval, usable samples 120/117/120, summaries off): with the additive graph boost on, the known-item chunk got worse in 46/49/52 samples and better in 3/2/0 (sign test p < 7e-11 per corpus), 106 targets left the top-10 entirely; reproduced on the summaries-on product path (HAL, 20 paired samples: hit@1 10→7, hit@5 18→13). Mechanism: only query-matched/connected entities score, but the per-entity boost saturates the cap quickly, so heavily-linked chunks can outrank the exact chunk even at `vector_similarity` 0. This is a harm-reduced default, not a validated graph improvement: the boost path is unchanged for `useGraph: true` (legacy/experimental re-ranker for back-compat and evaluation; the graph does not generate candidates — for relationship exploration use `openNodes` → `getNeighbors`). Regression lock: `test/search-graph-default.test.mjs`.
+- (v5.0.0–v5.2.0 notes live in the git tags / `docs/UPDATING.md`.)
+
 ### v4.0.0
 
 **Observation lifecycle (schema v13).** Observations used to be a JSON array of strings on the

package/dist/index.d.ts

@@ -70,6 +70,7 @@ export declare class RAGKnowledgeGraphManager {
     currentProfileId: number;
     grandfatherAllowed: boolean;
     coordinator: BackfillCoordinator | null;
+    readonly calendarTimeZone: string;
     private embeddingCache;
     private readonly EMBEDDING_CACHE_MAX;
     private dictionaryCache;

package/dist/index.js

@@ -32,6 +32,7 @@ import { migrations } from './src/migrations/migrations.js';
 import { EmbeddingGate, GateNotReadyError, GateDisabledError, TerminalConfigError } from './src/embeddingGate.js';
 import { resolveModelCacheDir, preflightCacheDir, artifactKey, ModelDownloadLock, handleLoaderFailure } from './src/modelCache.js';
 import { BackfillCoordinator } from './src/backfillCoordinator.js';
+import { calendarDate, resolveCalendarTimeZone, stampDatePrefix, stripDatePrefix } from './src/observations/date-prefix.js';
 import os from 'node:os';
 import { createHash } from 'crypto';
 import { createRequire } from 'module';
@@ -179,6 +180,9 @@ export class RAGKnowledgeGraphManager {
     // default model config, or with the explicit trust opt-in (spec §6b guard).
     grandfatherAllowed = IS_DEFAULT_MODEL_CONFIG || process.env.RAG_MEMORY_TRUST_LEGACY_VECTORS === '1';
     coordinator = null;
+    // The calendar that date-only human labels are written in. Resolved once, here, so an invalid
+    // zone fails at construction instead of quietly writing wrong days for weeks.
+    calendarTimeZone = resolveCalendarTimeZone(process.env.RAG_MEMORY_CALENDAR_TZ);
     embeddingCache = new Map();
     EMBEDDING_CACHE_MAX = 500;
     dictionaryCache = null;
@@ -574,11 +578,12 @@ export class RAGKnowledgeGraphManager {
     }
     // === ORIGINAL MCP FUNCTIONALITY ===
     _timestampObservation(obs) {
-        // If observation already has a date prefix like [2026-03-21], skip
-        if (/^\[\d{4}-\d{2}-\d{2}\]/.test(obs))
-            return obs;
-        const today = new Date().toISOString().slice(0, 10);
-        return `[${today}] ${obs}`;
+        // Stamp and dedup share one parser (src/observations/date-prefix.ts). They used to carry
+        // separate regexes and disagreed about "[2026-08-11 session16] ...": stamping treated it as
+        // undated and prepended a second date, while dedup could not strip it — so the same sentence
+        // written in two sessions became two observations. Measured on a live database: 82 rows with
+        // two dates, 29 with a day earlier than the day they were written.
+        return stampDatePrefix(obs, this.calendarTimeZone);
     }
     async createEntities(entities) {
         if (!this.db)
@@ -590,7 +595,7 @@ export class RAGKnowledgeGraphManager {
       INSERT OR IGNORE INTO entities (id, name, entityType, observations, metadata)
       VALUES (?, ?, ?, '[]', ?)
     `);
-        const stripDate = (s2) => s2.replace(/^\[\d{4}-\d{2}-\d{2}\]\s*/, '');
+        const stripDate = stripDatePrefix; // shared with the stamp path — see date-prefix.ts
         for (const entity of entities) {
             const entityId = `entity_${entity.name.toLowerCase().replace(/[^\p{L}\p{N}]/gu, '_')}`;
             const ts = new Date().toISOString();
@@ -700,7 +705,7 @@ export class RAGKnowledgeGraphManager {
             // dedup 기준은 v3.6 과 같다: 날짜 prefix 를 뗀 본문이 active 에 이미 있으면
             // 새 revision 을 만들지 않는다. 다만 v13 에서는 같은 사실이 다른 출처에서 다시
             // 온 것이므로 그 revision 에 source link 를 더한다(spec §8.3 T13).
-            const stripDate = (s2) => s2.replace(/^\[\d{4}-\d{2}-\d{2}\]\s*/, '');
+            const stripDate = stripDatePrefix; // shared with the stamp path — see date-prefix.ts
             const activeRows = this.db.prepare(`SELECT observation_id, content FROM entity_observations
          WHERE entity_id = ? AND status = 'active'`).all(entityId);
             const activeByBare = new Map(activeRows.map(r => [stripDate(r.content), r.observation_id]));
@@ -1963,7 +1968,10 @@ export class RAGKnowledgeGraphManager {
             // report `unchanged` for a different exclusion, which is the silent-wrong case.
             const content = applyExcludePatterns(options.content !== undefined ? options.content : fsSync.readFileSync(filePath, 'utf-8'), options.excludePattern);
             const bytes = Buffer.byteLength(content, 'utf-8');
-            const today = new Date().toISOString().slice(0, 10);
+            // Same calendar as the observation prefix. Deciding this explicitly rather than leaving it
+            // on UTC: both are date-only labels a person reads, and "observations in Seoul days but
+            // documents in UTC days" is not a distinction anyone could explain later.
+            const today = calendarDate(new Date(), this.calendarTimeZone);
             const contentHash = shaHex(content);
             // spec §5.1: content_hash 는 system-owned — user metadata 뒤에 쓴다 (r1: spread 가 덮어쓸 수 있었다).
             const metadata = { source: filePath, updated: today, ...(options.metadata || {}), content_hash: contentHash };
@@ -2915,7 +2923,19 @@ export class RAGKnowledgeGraphManager {
         this.coordinator?.kick();
         return { imported, skipped, observation_order_remap: remapReport };
     }
-    async hybridSearch(query, limit = 5, useGraph = true) {
+    // v5.3.0: the graph re-ranker is OPT-IN (harm-reduced default, not a validated improvement).
+    // Measured 2026-08-17 on three real corpora (self-retrieval, usable samples 120/117/120,
+    // summaries off): with the additive graph boost on, the known-item chunk got WORSE in
+    // 46/49/52 samples and BETTER in 3/2/0 (sign test p < 7e-11 per corpus); 106 targets left the
+    // top-10 entirely. Reproduced on the summaries-on product path (HAL, 20 paired samples:
+    // hit@1 10 -> 7, hit@5 18 -> 13, worse 8 / better 3). Mechanism: only query-matched or
+    // connected entities score, but the per-entity boost (0.5^i decay, cap 0.4) saturates fast, so
+    // heavily-linked chunks get more chances to match and reach the cap — they can outrank the
+    // exact chunk even at vector_similarity 0. Note the graph does not generate candidates: it only
+    // re-orders the vector/FTS candidate pool, so useGraph:true is a legacy/experimental re-ranker
+    // (backward compatibility, controlled evaluation), not a relationship-exploration path — that
+    // contract is openNodes -> getNeighbors. The boost path itself is unchanged.
+    async hybridSearch(query, limit = 5, useGraph = false) {
         if (!this.db)
             throw new Error('Database not initialized');
         if (!this.encoding)
@@ -3452,6 +3472,9 @@ export class RAGKnowledgeGraphManager {
             server: {
                 version: PKG_VERSION,
                 node: process.versions.node,
+                // Which calendar produced the date-only labels in this database. Surfaced because a
+                // wrong value is otherwise invisible: the labels look plausible either way.
+                calendar_timezone: this.calendarTimeZone,
                 embeddings_mode: this.embeddingsMode,
                 model: `${EMBEDDING_MODEL}@${MODEL_REVISION}`,
                 model_state: gs.state,
@@ -3859,7 +3882,8 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
                 return { content: [{ type: "text", text: JSON.stringify(await ragKgManager.linkEntitiesToDocument(validatedArgs.documentId, validatedArgs.entityNames), null, 2) }] };
             case "hybridSearch":
                 const limit = typeof validatedArgs.limit === 'number' ? validatedArgs.limit : 5;
-                const useGraph = validatedArgs.useGraph !== false;
+                // v5.3.0: graph is opt-in — only an explicit true enables the re-ranker (schema default false).
+                const useGraph = validatedArgs.useGraph === true;
                 return { content: [{ type: "text", text: JSON.stringify(await ragKgManager.hybridSearch(validatedArgs.query, limit, useGraph), null, 2) }] };
             case "getDetailedContext":
                 return { content: [{ type: "text", text: JSON.stringify(await ragKgManager.getDetailedContext(validatedArgs.chunkId, validatedArgs.includeSurrounding !== false), null, 2) }] };

package/dist/src/migrations/migrations.js

@@ -9,7 +9,16 @@ export const migrations = [
         version: 1,
         description: 'Complete RAG Knowledge Graph schema - all tables and features',
         up: (db) => {
-            // Disable foreign key enforcement to make deletions easier
+            // ⚠ 이 줄은 **아무 일도 하지 않는다** (2026-08-11 실측). migration-manager 가 각
+            // 마이그레이션을 `db.transaction(...)` 으로 감싸는데, SQLite 에서 `PRAGMA foreign_keys`
+            // 는 **트랜잭션 안에서 no-op** 이기 때문이다. 실측 = 신규 DB 에 13개 마이그레이션을
+            // 전부 적용한 뒤에도 `foreign_keys = 1`.
+            //
+            // **고치지 말 것.** 이 줄이 실제로 동작하게 만들면(트랜잭션 밖으로 빼는 등) FK 는
+            // 마이그레이션 이후 **그 프로세스 수명 내내 꺼진 채로 남는다** — 부팅 게이트(index.ts)
+            // 는 `runMigrations()` **앞**에서 판정하므로 그걸 못 잡는다. 그 상태에서 entity 를
+            // 지우면 observation 계열이 CASCADE 되지 않아 고아·FK 위반이 쌓인다.
+            // 회귀 잠금 = test/observation-cascade.test.mjs T25(신규 DB 부팅 후 FK==1).
             db.pragma('foreign_keys = OFF');
             // Original entities table (enhanced)
             db.exec(`

package/dist/src/observations/date-prefix.d.ts

@@ -0,0 +1,26 @@
+/** The calendar day of `instant` in `timeZone`, as YYYY-MM-DD. */
+export declare function calendarDate(instant: Date, timeZone: string): string;
+/**
+ * Resolve RAG_MEMORY_CALENDAR_TZ. Unset or blank means UTC. An unrecognised zone throws at
+ * boot rather than silently falling back — a wrong-but-quiet calendar writes wrong labels for
+ * as long as nobody looks, which is the failure this whole module exists to end.
+ */
+export declare function resolveCalendarTimeZone(raw: string | undefined | null): string;
+export interface DatePrefix {
+    /** YYYY-MM-DD, already validated as a real calendar day. */
+    date: string;
+    /** Text between the date and the closing bracket, e.g. a session marker. */
+    annotation: string | null;
+    /** The matched prefix including the brackets, without trailing whitespace. */
+    matched: string;
+}
+/** Parse a leading date prefix, or null when the text does not start with one. */
+export declare function parseDatePrefix(content: string): DatePrefix | null;
+/**
+ * The dedup key: the text with its date prefix removed. The annotation comes off with the date
+ * because it records *when and in which session the line was written*, not what the line claims
+ * — the same sentence written in two sessions is one fact, and dedup has to see that.
+ */
+export declare function stripDatePrefix(content: string): string;
+/** Prepend today's calendar day unless the text already carries a valid date prefix. */
+export declare function stampDatePrefix(content: string, timeZone: string, now?: Date): string;

package/dist/src/observations/date-prefix.js

@@ -0,0 +1,89 @@
+// One place that decides what a leading "[date]" on an observation means.
+//
+// Why this is a module and not two regexes inline: the stamp path and the dedup path each had
+// their own copy of the pattern, and they disagreed. Stamping asked "does this already start
+// with a date?" while dedup asked "what is this text without its date?", so widening one without
+// the other let the same sentence be stored twice under two session markers. Both now call in
+// here.
+//
+// Why an explicit timezone instead of the process one: a date-only label has no meaning until
+// you say which calendar produced it. The same database is reached from a laptop, from CI and
+// from another country; deriving the day from the ambient TZ makes the stored string mean
+// something different on each. The default stays UTC so the product does not inherit whichever
+// zone its first author happened to sit in — a deployment that wants local days says so.
+/** The calendar day of `instant` in `timeZone`, as YYYY-MM-DD. */
+export function calendarDate(instant, timeZone) {
+    const parts = new Intl.DateTimeFormat('en-US', {
+        timeZone,
+        year: 'numeric',
+        month: '2-digit',
+        day: '2-digit',
+    }).formatToParts(instant);
+    const part = (type) => {
+        const found = parts.find(p => p.type === type);
+        if (!found)
+            throw new Error(`calendarDate: missing ${type} for timeZone ${timeZone}`);
+        return found.value;
+    };
+    return `${part('year')}-${part('month')}-${part('day')}`;
+}
+/**
+ * Resolve RAG_MEMORY_CALENDAR_TZ. Unset or blank means UTC. An unrecognised zone throws at
+ * boot rather than silently falling back — a wrong-but-quiet calendar writes wrong labels for
+ * as long as nobody looks, which is the failure this whole module exists to end.
+ */
+export function resolveCalendarTimeZone(raw) {
+    const value = (raw ?? '').trim();
+    if (!value)
+        return 'UTC';
+    try {
+        new Intl.DateTimeFormat('en-US', { timeZone: value });
+    }
+    catch {
+        throw new Error(`RAG_MEMORY_CALENDAR_TZ is not a recognised IANA time zone: ${JSON.stringify(value)}. ` +
+            `Use a name like "Asia/Seoul", or leave it unset for UTC.`);
+    }
+    return value;
+}
+// The annotation may not contain a newline or a bracket: an unclosed or multi-line "[2026-…"
+// is prose that happens to start with a digit, not a prefix. Requiring the closing bracket is
+// what keeps "[2026-08-11 unclosed" and "[2026-08-111]" out.
+const PREFIX_RE = /^\[(\d{4})-(\d{2})-(\d{2})(?:[ \t]([^\]\n]*))?\]/;
+function isRealCalendarDay(year, month, day) {
+    const asUtc = new Date(Date.UTC(year, month - 1, day));
+    return asUtc.getUTCFullYear() === year
+        && asUtc.getUTCMonth() === month - 1
+        && asUtc.getUTCDate() === day;
+}
+/** Parse a leading date prefix, or null when the text does not start with one. */
+export function parseDatePrefix(content) {
+    const m = PREFIX_RE.exec(content);
+    if (!m)
+        return null;
+    const [matched, year, month, day, annotation] = m;
+    if (!isRealCalendarDay(Number(year), Number(month), Number(day)))
+        return null;
+    const trimmed = annotation === undefined ? null : annotation.trim();
+    return {
+        date: `${year}-${month}-${day}`,
+        annotation: trimmed === '' ? null : trimmed,
+        matched,
+    };
+}
+/**
+ * The dedup key: the text with its date prefix removed. The annotation comes off with the date
+ * because it records *when and in which session the line was written*, not what the line claims
+ * — the same sentence written in two sessions is one fact, and dedup has to see that.
+ */
+export function stripDatePrefix(content) {
+    const prefix = parseDatePrefix(content);
+    if (!prefix)
+        return content;
+    return content.slice(prefix.matched.length).replace(/^\s+/, '');
+}
+/** Prepend today's calendar day unless the text already carries a valid date prefix. */
+export function stampDatePrefix(content, timeZone, now = new Date()) {
+    if (parseDatePrefix(content))
+        return content;
+    return `[${calendarDate(now, timeZone)}] ${content}`;
+}

package/dist/src/tools/knowledge-graph-tools.js

@@ -200,6 +200,11 @@ Observations provide the factual foundation that supports entity existence and p
 - (!important!) **Entity must exist** - this tool only adds to existing entities
 - (!important!) Duplicates are filtered - repeating existing text with a **new** sources entry
   adds evidence to the existing revision and returns null for that position
+- (!important!) A \`[YYYY-MM-DD]\` prefix is prepended unless the text already starts with a valid
+  one. The day comes from \`RAG_MEMORY_CALENDAR_TZ\` (default UTC) - see \`calendar_timezone\` in
+  getKnowledgeGraphStats. Text may open with \`[YYYY-MM-DD note]\`; the note is treated as recording
+  metadata and is ignored when deduplicating, so the same sentence filed under two session markers
+  stays one observation
 - (!important!) Observations are cumulative. **This tool never replaces anything** - use
   correctObservation to supersede and retractObservation to withdraw
 - (!important!) **Be specific and factual** - observations should be verifiable statements
@@ -287,22 +292,24 @@ const hybridSearchCapability = {
             },
             useGraph: {
                 type: 'boolean',
-                description: 'Whether to enhance results with knowledge graph connections',
-                default: true
+                description: 'Opt-in legacy/experimental graph re-ranker (default false since v5.3.0 — measured to hurt known-item retrieval). It only re-orders the vector/FTS candidate pool; for relationship exploration use openNodes -> getNeighbors instead',
+                default: false
             }
         },
         required: ['query'],
     },
 };
 const hybridSearchDescription = () => `<description>
-Perform sophisticated hybrid search that combines vector similarity with knowledge graph traversal for superior results.
-**The most powerful search tool in the system** - leverages both semantic similarity and structural relationships.
-Perfect for complex queries that benefit from both content matching and conceptual connections.
+Hybrid document search: vector similarity + FTS5 BM25 over document chunks, with an **opt-in** knowledge-graph re-ranker.
+Default (useGraph false) is the harm-reduced default for finding a fact you know exists (known-item retrieval).
+useGraph: true is a legacy/experimental re-ranker kept for backward compatibility and controlled evaluation — it does not
+add candidates, it only re-orders the vector/FTS pool, and measured on three real corpora (2026-08-17) it pushed the exact
+chunk out of the top-5 in ~40% of known-item queries. For relationship exploration use openNodes -> getNeighbors.
 </description>
 
 <importantNotes>
-- (!important!) **Hybrid approach is more powerful** than pure vector or graph search alone
-- (!important!) Graph enhancement finds related concepts even if not directly mentioned
+- (!important!) **Graph re-ranking is opt-in (default false since v5.3.0)** — it re-orders candidates by matched/connected entity links, which is measured to hurt known-item retrieval; enable it only for backward compatibility or controlled evaluation
+- (!important!) Graph re-ranking does not generate candidates — for "what is connected to X" use openNodes -> getNeighbors
 - (!important!) Results include similarity scores, graph boost, and hybrid rankings
 - (!important!) **Best results when knowledge graph is well-populated** with entities and relationships
 </importantNotes>
@@ -327,7 +334,7 @@ Perfect for complex queries that benefit from both content matching and conceptu
 
 <bestPractices>
 - Use natural language queries rather than keywords
-- Enable graph enhancement for better conceptual coverage
+- Keep graph re-ranking off for "find the fact I know exists"; for "what is connected to this" use openNodes -> getNeighbors, not useGraph
 - Start with broader queries, then narrow down based on results
 - Review entity associations to understand why results were selected
 - Use appropriate limits based on your analysis needs
@@ -337,19 +344,18 @@ Perfect for complex queries that benefit from both content matching and conceptu
 <parameters>
 - query: Natural language search query (string, required)
 - limit: Maximum results to return, default 5 (number, optional)
-- useGraph: Enable knowledge graph enhancement, default true (boolean, optional)
+- useGraph: Enable knowledge graph re-ranking, default false / opt-in (boolean, optional)
 </parameters>
 
 <examples>
-- Conceptual search: {"query": "machine learning applications in healthcare", "limit": 10}
-- Technical research: {"query": "React performance optimization techniques", "useGraph": true}
-- Discovery mode: {"query": "Einstein's contributions to modern physics", "limit": 15}
-- Quick lookup: {"query": "quantum computing advantages", "limit": 3, "useGraph": false}
+- Known-item lookup (default, graph off): {"query": "machine learning applications in healthcare", "limit": 10}
+- Legacy re-ranker (opt-in, evaluation/back-compat only): {"query": "React performance optimization techniques", "useGraph": true}
+- Discovery / relationship exploration: use openNodes then getNeighbors on the entity, not this tool's useGraph
 </examples>`;
 const hybridSearchSchema = {
     query: z.string().describe('The search query to find relevant information'),
     limit: z.number().optional().default(5).describe('Maximum number of results to return'),
-    useGraph: z.boolean().optional().default(true).describe('Whether to enhance results with knowledge graph connections'),
+    useGraph: z.boolean().optional().default(false).describe('Opt-in legacy/experimental graph re-ranker (default false since v5.3.0; measured to hurt known-item retrieval; for exploration use openNodes -> getNeighbors)'),
 };
 export const hybridSearchTool = {
     capability: hybridSearchCapability,

package/docs/UPDATING.md

@@ -108,6 +108,28 @@ path and holder pid (e.g. `.download-<key>.lock`). Verify the holder process
 is genuinely gone or hung (`ps -p <pid>`), then remove the lock file manually;
 the next start becomes a clean download owner.
 
+## v5.3.0-rc.1 (schema v14, unchanged): `hybridSearch` graph re-ranking is opt-in
+
+**What changed.** `useGraph` defaults to `false` (was `true`) — in the manager signature, the tool JSON
+schema, the zod schema (`validateToolArgs` fills `false`) and the MCP dispatch (`=== true`). Nothing else
+in the scoring path moved: `useGraph: true` runs exactly the pre-5.3 graph boost.
+
+**Why.** Measured 2026-08-17 on three real corpora (self-retrieval, usable samples 120/117/120, summaries
+off): with the additive graph boost on, the known-item chunk got WORSE in 46/49/52 samples and BETTER in
+3/2/0 (paired sign test p < 7e-11 per corpus); 106 targets fell out of the top-10 entirely. Reproduced on
+the summaries-on product path (HAL, 20 paired samples: hit@1 10→7, hit@5 18→13). Only query-matched or
+connected entities score, but the per-entity boost (0.5^i decay, cap 0.4) saturates fast, so heavily-linked
+chunks can outrank the exact chunk even at `vector_similarity` 0. This is a reversible harm mitigation, not
+a validated graph improvement — the graph's role (candidate generation vs. re-ranking, explicit mode) is
+still open and will be decided on a graph-required query suite. Note that `useGraph: true` does not add
+candidates (it re-orders the vector/FTS pool): for relationship exploration use `openNodes` → `getNeighbors`.
+
+**Fleet rollout.** No migration, no schema change. Published first as a release candidate on the `next`
+dist-tag (omitting an argument changes behavior, so it gets a canary before `latest`): pin one project to
+`rag-memory-epf-mcp@next`, run its normal /start and /sync searches, then promote to stable. A caller that
+relied on graph re-ranking by default must now pass `useGraph: true`. Callers that already choose per query
+see no change. Regression lock: `test/search-graph-default.test.mjs`.
+
 ## v5.1.0 (schema v14, unchanged): destructive-replace reporting + `excludePattern`
 
 **What changes on upgrade**: nothing you have to do. No migration, no schema

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "rag-memory-epf-mcp",
-  "version": "5.1.0",
+  "version": "5.3.0-rc.1",
   "engines": {
     "node": ">=24"
   },
@@ -45,7 +45,7 @@
     "prepare": "npm run build",
     "watch": "tsc --watch",
     "verify:invariants": "node test/chunk-invariants.test.mjs",
-    "verify:engine": "node test/engine-smoke.test.mjs && node test/launch-smoke.test.mjs && node test/sync-atomicity.test.mjs && node test/dedup.test.mjs && node test/search-degradation.test.mjs && node test/entity-embed-cap.test.mjs && node test/migration12.test.mjs && node test/model-cache.test.mjs && node test/embedding-gate.test.mjs && node test/lazy-boot.test.mjs && node test/reconciliation.test.mjs && node test/backfill.test.mjs && node test/fts-query.test.mjs && node test/search-contracts.test.mjs && node test/tool-contracts.test.mjs && node test/bounded-exit.test.mjs && node test/observation-schema.test.mjs && node test/observation-migration.test.mjs && node test/observation-lifecycle.test.mjs && node test/observation-contracts.test.mjs && node test/observation-search.test.mjs && node test/observation-cascade.test.mjs && node test/observation-realdata.test.mjs && node test/chunker-c.test.mjs && node test/migration14.test.mjs && node test/chunk-params-validation.test.mjs && node test/vector-reuse.test.mjs && node test/entity-range-linking.test.mjs && node test/stats-chunking.test.mjs && node test/migration14-realdata.test.mjs && node test/migration14-realdata-sync.test.mjs && node test/search-summaries-off.test.mjs && node test/document-return-contracts.test.mjs",
+    "verify:engine": "node test/engine-smoke.test.mjs && node test/launch-smoke.test.mjs && node test/sync-atomicity.test.mjs && node test/dedup.test.mjs && node test/search-degradation.test.mjs && node test/entity-embed-cap.test.mjs && node test/migration12.test.mjs && node test/model-cache.test.mjs && node test/embedding-gate.test.mjs && node test/lazy-boot.test.mjs && node test/reconciliation.test.mjs && node test/backfill.test.mjs && node test/fts-query.test.mjs && node test/search-contracts.test.mjs && node test/tool-contracts.test.mjs && node test/bounded-exit.test.mjs && node test/observation-schema.test.mjs && node test/search-graph-default.test.mjs && node test/observation-migration.test.mjs && node test/observation-lifecycle.test.mjs && node test/observation-contracts.test.mjs && node test/observation-search.test.mjs && node test/observation-cascade.test.mjs && node test/observation-realdata.test.mjs && node test/chunker-c.test.mjs && node test/migration14.test.mjs && node test/chunk-params-validation.test.mjs && node test/vector-reuse.test.mjs && node test/entity-range-linking.test.mjs && node test/stats-chunking.test.mjs && node test/migration14-realdata.test.mjs && node test/migration14-realdata-sync.test.mjs && node test/search-summaries-off.test.mjs && node test/document-return-contracts.test.mjs && node test/observation-date-prefix.test.mjs && node test/delete-entities-cascade.test.mjs",
     "test": "npm run build && npm run verify:invariants && npm run verify:engine",
     "prepublishOnly": "npm run build && npm run verify:invariants && npm run verify:engine"
   },

package/dist/src/chunkText.d.ts

@@ -1,10 +0,0 @@
-import type { Tiktoken } from 'tiktoken';
-export interface ChunkSegment {
-    text: string;
-    start_pos: number | null;
-    end_pos: number | null;
-    start_token: number;
-    end_token: number;
-}
-export declare function trimIncompleteUtf8(bytes: Uint8Array, trimHead: boolean, trimTail: boolean): Uint8Array;
-export declare function chunkText(text: string, encoding: Tiktoken, maxTokens?: number, overlap?: number): ChunkSegment[];

package/dist/src/chunkText.js

@@ -1,104 +0,0 @@
-// Tokenize and chunk text using a BPE encoder while reporting both token-space
-// and char-space (Unicode codepoint) offsets back into the original string.
-//
-// BPE tokenizers (cl100k_base) split multi-byte UTF-8 sequences across tokens.
-// Slicing token arrays at arbitrary boundaries can leave incomplete UTF-8
-// prefix/suffix bytes, which TextDecoder replaces with U+FFFD (�). We trim the
-// incomplete sequences at chunk boundaries; overlap covers the removed bytes.
-//
-// Each chunk records both token-space offsets (start_token/end_token from the
-// BPE encoder loop) and char-space offsets (start_pos/end_pos into the original
-// text). Char offsets are Unicode codepoint counts — language-neutral, so SQL
-// substr, Python str slicing, and JS [...str] iteration all line up. JS's
-// native UTF-16 indexing differs for supplementary characters (emoji, rare CJK),
-// so the function maintains parallel UTF-16 and codepoint cursors and reports
-// codepoint offsets. On a coincidental indexOf miss the char offsets are NULL.
-//
-// Extracted to a standalone module so publish-time invariant tests can exercise
-// the algorithm directly without booting the full RAG-Memory stack.
-// trimIncompleteUtf8: strip incomplete UTF-8 sequences from the head/tail of a
-// byte buffer produced by decoding an arbitrary token slice. A multi-byte
-// codepoint that begins or ends on the cut edge belongs to an adjacent chunk
-// and must be removed so TextDecoder does not emit U+FFFD. Pass
-// trimHead/trimTail=false to preserve head/tail bytes (first/last chunks).
-export function trimIncompleteUtf8(bytes, trimHead, trimTail) {
-    let start = 0;
-    let end = bytes.length;
-    if (trimHead) {
-        while (start < end && (bytes[start] & 0xC0) === 0x80)
-            start++;
-    }
-    if (trimTail) {
-        let i = end - 1;
-        while (i >= start && (bytes[i] & 0xC0) === 0x80)
-            i--;
-        if (i >= start) {
-            const lead = bytes[i];
-            let needed = 1;
-            if ((lead & 0x80) === 0)
-                needed = 1;
-            else if ((lead & 0xE0) === 0xC0)
-                needed = 2;
-            else if ((lead & 0xF0) === 0xE0)
-                needed = 3;
-            else if ((lead & 0xF8) === 0xF0)
-                needed = 4;
-            if (end - i < needed)
-                end = i;
-        }
-    }
-    return bytes.subarray(start, end);
-}
-export function chunkText(text, encoding, maxTokens = 800, overlap = 160) {
-    const tokens = encoding.encode(text);
-    const segments = [];
-    let utf16Cursor = 0;
-    let cpCursor = 0;
-    for (let i = 0; i < tokens.length; i += maxTokens - overlap) {
-        const chunkTokens = tokens.slice(i, i + maxTokens);
-        const decodedBytes = encoding.decode(chunkTokens);
-        const isFirst = i === 0;
-        const isLast = i + chunkTokens.length >= tokens.length;
-        const safeBytes = trimIncompleteUtf8(decodedBytes, !isFirst, !isLast);
-        const chunkTextStr = new TextDecoder('utf-8').decode(safeBytes);
-        let startPos;
-        let endPos;
-        if (isFirst) {
-            startPos = 0;
-            endPos = [...chunkTextStr].length;
-            utf16Cursor = 0;
-            cpCursor = 0;
-        }
-        else if (chunkTextStr.length === 0) {
-            startPos = null;
-            endPos = null;
-        }
-        else {
-            const utfIdx = text.indexOf(chunkTextStr, utf16Cursor);
-            if (utfIdx >= 0) {
-                // Advance cpCursor by codepoints between the previous cursor and the
-                // new chunk's start (handles overlap by anchoring at the previous
-                // chunk's start, not its end).
-                if (utfIdx > utf16Cursor) {
-                    cpCursor += [...text.slice(utf16Cursor, utfIdx)].length;
-                    utf16Cursor = utfIdx;
-                }
-                const cpLen = [...chunkTextStr].length;
-                startPos = cpCursor;
-                endPos = cpCursor + cpLen;
-            }
-            else {
-                startPos = null;
-                endPos = null;
-            }
-        }
-        segments.push({
-            text: chunkTextStr,
-            start_pos: startPos,
-            end_pos: endPos,
-            start_token: i,
-            end_token: i + chunkTokens.length
-        });
-    }
-    return segments;
-}