rag-memory-epf-mcp 5.3.1 → 6.0.0

package/README.md

@@ -153,6 +153,30 @@ storeDocument(id, content, metadata)
 
 ## Changelog
 
+### v6.0.0
+
+- **Breaking — observation-derived alias links are now gated.** `autoLinkEntities` used to take every
+  `stem.ext` token appearing anywhere in an entity's observations and link that entity to every chunk
+  containing the token as a substring. Measured on a live 2,891-chunk / 604-entity corpus: 65,388 of
+  66,841 `chunk_entities` rows (97.8%) existed only because of such a hit, and one token — `agents.md`,
+  held by 88 entities — accounted for 39,248 of the 80,096 distinct (chunk, entity) pairs any alias
+  could reach (49.0%). The regex also accepted things that are not filenames at all (`v3.3`,
+  `gpt-5.6`, `1.7mb`, `github.com`, `os.path`).
+  A token now has to (a) look like a filename — extension whitelist, stem ≥ 3 chars, non-numeric
+  stem — and (b) be held by at most 3 entities, and it must match on token boundaries so `foo.py` no
+  longer matches inside `notfoo.pyc`. Effect on the same corpus: alias links 100% → 4.0%. Filtering by
+  extension alone leaves 92.3%, so the owner cap is what does the work. The cap is a judgement, not a
+  discovered boundary — the sweep is smooth (`owners<=1` 1.6% … `<=10` 19.9%) — and it is a cap, not a
+  ban: a filename named by one to three records still links, which is what the alias path was for.
+- **What this means for existing databases.** Nothing is rewritten on upgrade: rows already in
+  `chunk_entities` stay, and new ingests simply link far less. Links have always been a function of
+  when a document was last processed (nothing re-links older documents when entities are added), and
+  `chunk_entities` has no provenance column, so old alias rows cannot be told apart from name matches
+  after the fact. If you want the old noise gone you have to clean it offline, before or after
+  upgrading. Callers that assumed dense `chunk_entities` coverage will see sparser graphs.
+- Regression: `test/alias-link-gate.test.mjs` (registered in `verify:engine`), verified to fail
+  without the gate.
+
 ### v5.3.0
 - (Published first as `5.3.0-rc.1` on the `next` dist-tag; promoted to `latest` after a canary run of the published artifact against a real project database: default call carries no `graph_boost` and equals explicit `useGraph:false`, the known-item probe from the 2026-08-17 measurement returns the correct gotcha at rank 1, opt-in `true` still exposes `graph_boost`, schema/MCP defaults read `false`.)
 - **Behavior change — `hybridSearch` graph re-ranking is now opt-in** (`useGraph` default `true` → `false`; tool schema, MCP exposure and the manager signature agree). Omitting the argument now means "no graph re-ranking" — a behavior change for callers that relied on the old default, hence a release-candidate first (`next` dist-tag, fleet canary) before stable. Measured 2026-08-17 on three real corpora (self-retrieval, usable samples 120/117/120, summaries off): with the additive graph boost on, the known-item chunk got worse in 46/49/52 samples and better in 3/2/0 (sign test p < 7e-11 per corpus), 106 targets left the top-10 entirely; reproduced on the summaries-on product path (HAL, 20 paired samples: hit@1 10→7, hit@5 18→13). Mechanism: only query-matched/connected entities score, but the per-entity boost saturates the cap quickly, so heavily-linked chunks can outrank the exact chunk even at `vector_similarity` 0. This is a harm-reduced default, not a validated graph improvement: the boost path is unchanged for `useGraph: true` (legacy/experimental re-ranker for back-compat and evaluation; the graph does not generate candidates — for relationship exploration use `openNodes` → `getNeighbors`). Regression lock: `test/search-graph-default.test.mjs`.

package/dist/index.d.ts

@@ -279,6 +279,8 @@ export declare class RAGKnowledgeGraphManager {
         errors?: string[];
     }>;
     private hasCJK;
+    private static readonly ALIAS_FILE_EXT;
+    private looksLikeFilename;
     private buildEntityRangeFinder;
     private buildEntityMatcher;
     private autoLinkEntities;

package/dist/index.js

@@ -2241,6 +2241,32 @@ export class RAGKnowledgeGraphManager {
     hasCJK(text) {
         return /[\u3000-\u9fff\uac00-\ud7af\uff00-\uffef]/.test(text);
     }
+    // The alias regex in autoLinkEntities matches anything shaped like "stem.ext", which
+    // includes version strings ("v3.3", "gpt-5.6"), measurements ("1.7mb", "0.465") and
+    // domains/module paths ("github.com", "os.path"). Those are not filenames and linking
+    // on them is pure noise. Whitelist the extensions we actually ship and store.
+    static ALIAS_FILE_EXT = new Set([
+        'md', 'py', 'js', 'mjs', 'cjs', 'ts', 'tsx', 'jsx', 'json',
+        'sh', 'bash', 'zsh', 'toml', 'yaml', 'yml', 'ini', 'cfg', 'conf',
+        'txt', 'log', 'csv', 'tsv', 'sql', 'db', 'zip', 'gz', 'tar',
+        'css', 'scss', 'html', 'htm', 'svg', 'png', 'jpg', 'jpeg', 'gif',
+        'pdf', 'docx', 'pptx', 'xlsx', 'hwp', 'hwpx', 'lock', 'bak',
+    ]);
+    // Deliberately absent: any 5+ character extension (jsonl, ipynb, scss is 4 so it stays).
+    // The extractor regex is `\w{1,4}`, so a longer extension never reaches this set — listing
+    // one would be a dead entry that reads as support. Extend the regex first if that changes.
+    looksLikeFilename(token) {
+        const i = token.lastIndexOf('.');
+        if (i < 1)
+            return false;
+        const stem = token.slice(0, i);
+        const ext = token.slice(i + 1);
+        if (stem.length < 3)
+            return false; // "d.ts" is a fragment, not a file
+        if (/^\d+$/.test(stem))
+            return false; // "2026.md" style numeric stems
+        return RAGKnowledgeGraphManager.ALIAS_FILE_EXT.has(ext);
+    }
     // spec §5.4 (r7-2·r8-1·r9): primary name 의 본문 occurrence range [sCp, eCp).
     // 의미 = buildEntityMatcher 와 동일 (CJK substring / Latin word-boundary) — 여기서
     // 어긋나면 'Data' 가 'Database' 에 새로 링크되는 식으로 의미가 확장된다.
@@ -2342,6 +2368,51 @@ export class RAGKnowledgeGraphManager {
             // Minimum name length: 2 for CJK (e.g. "할랄"), 4 for Latin (avoid "API", "Bug")
             const MIN_LEN_CJK = 2;
             const MIN_LEN_LATIN = 4;
+            // Observation-derived aliases are only useful when the token identifies few entities.
+            // Measured on a 2,891-chunk / 604-entity corpus (2026-08-22). Denominators matter here,
+            // so both are stated: 66,841 rows in chunk_entities, of which 65,388 (97.8%) exist only
+            // because of an alias hit; counting distinct (chunk, entity) pairs reachable by any alias
+            // gives 80,096. Against that 80,096, "agents.md" alone accounts for 39,248 (49.0%), and
+            // 35,099 (43.8%) have no other alias reason at all. It is held by 88 entities. A filename
+            // that dozens of entities mention is a stopword, not an identifier.
+            //
+            // The owner cap is what does the work: the extension whitelist alone still leaves 92.3%
+            // of alias links. But the cap value is a judgement call, not a discovered boundary — the
+            // sweep is smooth, with no natural knee (share of the 80,096 that survives):
+            //   owners<=1 1.6% · <=2 3.7% · <=3 5.2% · <=4 7.0% · <=5 8.3% · <=8 14.7% · <=10 19.9%
+            // 3 was chosen to keep the intended behaviour (a file named by one or two records, plus
+            // some slack) while cutting the stopword tail. Raising it is cheap and reversible.
+            //
+            // A chunk-frequency cap was measured (a further 1.4pp) and rejected on COST: it needs a
+            // full-corpus scan on every ingest. Note the honest caveat — it was first rejected for
+            // depending on ingest order, but this owner cap has that same property, and so does the
+            // engine as a whole: autoLinkEntities only ever sees the entities that exist at ingest
+            // time, and nothing re-links older documents when entities are added (see
+            // createEntities / addObservations — neither calls this). Links are a function of when a
+            // document was last processed. That predates this gate; it is not introduced by it.
+            const MAX_ALIAS_OWNERS = 3;
+            const aliasOwners = new Map();
+            for (const e of entities) {
+                if (!e.observations)
+                    continue;
+                let obs;
+                try {
+                    obs = JSON.parse(e.observations);
+                }
+                catch {
+                    continue;
+                }
+                const seen = new Set();
+                for (const ob of obs) {
+                    const pm = String(ob).match(/[\w\-]+\.\w{1,4}\b/g);
+                    if (pm)
+                        for (const p of pm)
+                            if (p.length >= 4)
+                                seen.add(p.toLowerCase());
+                }
+                for (const t of seen)
+                    aliasOwners.set(t, (aliasOwners.get(t) ?? 0) + 1);
+            }
             const insertStmt = this.db.prepare(`
         INSERT OR IGNORE INTO chunk_entities (chunk_rowid, entity_id) VALUES (?, ?)
       `);
@@ -2351,7 +2422,9 @@ export class RAGKnowledgeGraphManager {
                 if (entity.name.length < minLen)
                     continue;
                 const nameMatcher = this.buildEntityMatcher(entity.name);
-                // Also collect observation-derived aliases (short keywords from observations)
+                // Also collect observation-derived aliases (short keywords from observations).
+                // Gated: the token must look like a filename AND identify at most MAX_ALIAS_OWNERS
+                // entities. Ungated, "agents.md" linked 88 entities to every chunk that mentioned it.
                 const aliases = [];
                 if (entity.observations) {
                     let obs;
@@ -2366,9 +2439,30 @@ export class RAGKnowledgeGraphManager {
                         const pathMatch = ob.match(/[\w\-]+\.\w{1,4}\b/g);
                         if (pathMatch) {
                             for (const p of pathMatch) {
-                                if (p.length >= 4) {
-                                    aliases.push((text) => text.toLowerCase().includes(p.toLowerCase()));
-                                }
+                                if (p.length < 4)
+                                    continue;
+                                const tok = p.toLowerCase();
+                                if (!this.looksLikeFilename(tok))
+                                    continue; // "v3.3", "gpt-5.6", "0.465"
+                                if ((aliasOwners.get(tok) ?? 0) > MAX_ALIAS_OWNERS)
+                                    continue; // shared = identifies nothing
+                                // Bare substring matching links "foo.py" to a chunk saying "notfoo.pyc".
+                                // Require the token to stand alone: no filename character on either side.
+                                const boundary = /[\w\-.]/;
+                                aliases.push((text) => {
+                                    const hay = text.toLowerCase();
+                                    let from = 0;
+                                    for (;;) {
+                                        const i = hay.indexOf(tok, from);
+                                        if (i < 0)
+                                            return false;
+                                        const before = i === 0 ? '' : hay[i - 1];
+                                        const after = hay[i + tok.length] ?? '';
+                                        if (!boundary.test(before) && !boundary.test(after))
+                                            return true;
+                                        from = i + 1;
+                                    }
+                                });
                             }
                         }
                     }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "rag-memory-epf-mcp",
-  "version": "5.3.1",
+  "version": "6.0.0",
   "engines": {
     "node": ">=24"
   },
@@ -45,7 +45,7 @@
     "prepare": "npm run build",
     "watch": "tsc --watch",
     "verify:invariants": "node test/chunk-invariants.test.mjs",
-    "verify:engine": "node test/engine-smoke.test.mjs && node test/launch-smoke.test.mjs && node test/sync-atomicity.test.mjs && node test/dedup.test.mjs && node test/search-degradation.test.mjs && node test/entity-embed-cap.test.mjs && node test/migration12.test.mjs && node test/model-cache.test.mjs && node test/embedding-gate.test.mjs && node test/lazy-boot.test.mjs && node test/reconciliation.test.mjs && node test/backfill.test.mjs && node test/fts-query.test.mjs && node test/search-contracts.test.mjs && node test/tool-contracts.test.mjs && node test/bounded-exit.test.mjs && node test/observation-schema.test.mjs && node test/search-graph-default.test.mjs && node test/observation-migration.test.mjs && node test/observation-lifecycle.test.mjs && node test/observation-contracts.test.mjs && node test/observation-search.test.mjs && node test/observation-cascade.test.mjs && node test/observation-realdata.test.mjs && node test/chunker-c.test.mjs && node test/migration14.test.mjs && node test/chunk-params-validation.test.mjs && node test/vector-reuse.test.mjs && node test/entity-range-linking.test.mjs && node test/stats-chunking.test.mjs && node test/migration14-realdata.test.mjs && node test/migration14-realdata-sync.test.mjs && node test/search-summaries-off.test.mjs && node test/document-return-contracts.test.mjs && node test/observation-date-prefix.test.mjs && node test/delete-entities-cascade.test.mjs && node test/backup-publish-portable.test.mjs && node test/graph-context-explain.test.mjs && node test/eval-graph-role-libs.test.mjs && node test/eval-graph-role-t5b.test.mjs && node --test test/eval-graph-role-t8-fix.test.mjs && node --test test/eval-graph-role-t7-upstream.test.mjs && node --test test/eval-graph-role-t11-decision.test.mjs && node --test test/eval-graph-role-prereq-fix.test.mjs",
+    "verify:engine": "node test/engine-smoke.test.mjs && node test/launch-smoke.test.mjs && node test/sync-atomicity.test.mjs && node test/dedup.test.mjs && node test/search-degradation.test.mjs && node test/entity-embed-cap.test.mjs && node test/migration12.test.mjs && node test/model-cache.test.mjs && node test/embedding-gate.test.mjs && node test/lazy-boot.test.mjs && node test/reconciliation.test.mjs && node test/backfill.test.mjs && node test/fts-query.test.mjs && node test/search-contracts.test.mjs && node test/tool-contracts.test.mjs && node test/bounded-exit.test.mjs && node test/observation-schema.test.mjs && node test/search-graph-default.test.mjs && node test/observation-migration.test.mjs && node test/observation-lifecycle.test.mjs && node test/observation-contracts.test.mjs && node test/observation-search.test.mjs && node test/observation-cascade.test.mjs && node test/observation-realdata.test.mjs && node test/chunker-c.test.mjs && node test/migration14.test.mjs && node test/chunk-params-validation.test.mjs && node test/vector-reuse.test.mjs && node test/entity-range-linking.test.mjs && node test/alias-link-gate.test.mjs && node test/stats-chunking.test.mjs && node test/migration14-realdata.test.mjs && node test/migration14-realdata-sync.test.mjs && node test/search-summaries-off.test.mjs && node test/document-return-contracts.test.mjs && node test/observation-date-prefix.test.mjs && node test/delete-entities-cascade.test.mjs && node test/backup-publish-portable.test.mjs && node test/graph-context-explain.test.mjs && node test/eval-graph-role-libs.test.mjs && node test/eval-graph-role-t5b.test.mjs && node --test test/eval-graph-role-t8-fix.test.mjs && node --test test/eval-graph-role-t7-upstream.test.mjs && node --test test/eval-graph-role-t11-decision.test.mjs && node --test test/eval-graph-role-prereq-fix.test.mjs",
     "test": "npm run build && npm run verify:invariants && npm run verify:engine",
     "prepublishOnly": "npm run build && npm run verify:invariants && npm run verify:engine"
   },