rag-memory-epf-mcp 5.3.1 → 6.0.1

package/README.md

@@ -153,6 +153,64 @@ storeDocument(id, content, metadata)
 
 ## Changelog
 
+### v6.0.1
+
+**Versioning note.** Both changes below stop links from being created that should never have been
+created, and restore links that should have been. No tool signature, return shape or schema
+changes, so this is a patch: **derived-link density is not part of the compatibility contract**.
+Read the entries as "linking got more precise", not as an API break.
+
+- **An observation alias must now also appear in the entity's own name.** 6.0.0 capped how many
+  entities may share a token. That answers "does this token point at one entity?" and says nothing
+  about the other direction, so an entity that mentions a common filename *once* still attached to
+  every chunk containing it — and, being the sole owner, sailed through the cap. The cap stopped
+  the explosion, not the magnet. Measured after the 6.0.0 cleanup on a live corpus: 2,014
+  alias-only links remained and **34 entities held 68.1%** of them; the largest had the entity name
+  in **zero** of its chunks. Requiring the token (or its stem) to appear in the name brings that to
+  6 entities / 42.1%, and what remains are entities that really are about that file. Expect far
+  fewer alias links on new ingests; existing rows are not rewritten.
+- **A chunk-frequency cap was measured and rejected on the evidence, not on cost.** The full-corpus
+  scan is 585ms (731 tokens x 2,913 chunks). It was rejected because the sweep has no knee and every
+  cut also removed legitimate links — `log_coverage.py` occurs in 64 chunks and belongs to an entity
+  about exactly that file. The name condition is structural: no threshold, same meaning at any
+  corpus size.
+- **Fixed — entity names whose own edge is punctuation never linked.** The Latin matcher used
+  `\b<name>\b`. `\b` asserts a transition between a word character and a non-word character, so when
+  the name itself starts or ends with punctuation — `Widget Review (2026-05-27)`, `--build-flag` —
+  there is no transition to assert and the pattern cannot match however the text reads. Measured on
+  a live 2,913-chunk / 312-name corpus: **23 standalone occurrences across 13 names** were invisible.
+  Both the chunk matcher and the document range finder now accept an occurrence whose neighbours on
+  both sides are non-word. **Not a widening**: both sides must still be non-word, so `Data` continues
+  not to match inside `Database`. The scan runs on the original text rather than a lowercased copy,
+  because folding can change length (`İ` becomes two units) and the range finder converts these
+  indices to codepoints.
+- Regressions: `test/alias-link-gate.test.mjs` (magnet case) and `test/entity-name-boundary.test.mjs`,
+  both registered in `verify:engine` and both verified to fail without their fix.
+
+### v6.0.0
+
+- **Breaking — observation-derived alias links are now gated.** `autoLinkEntities` used to take every
+  `stem.ext` token appearing anywhere in an entity's observations and link that entity to every chunk
+  containing the token as a substring. Measured on a live 2,891-chunk / 604-entity corpus: 65,388 of
+  66,841 `chunk_entities` rows (97.8%) existed only because of such a hit, and one token — `agents.md`,
+  held by 88 entities — accounted for 39,248 of the 80,096 distinct (chunk, entity) pairs any alias
+  could reach (49.0%). The regex also accepted things that are not filenames at all (`v3.3`,
+  `gpt-5.6`, `1.7mb`, `github.com`, `os.path`).
+  A token now has to (a) look like a filename — extension whitelist, stem ≥ 3 chars, non-numeric
+  stem — and (b) be held by at most 3 entities, and it must match on token boundaries so `foo.py` no
+  longer matches inside `notfoo.pyc`. Effect on the same corpus: alias links 100% → 4.0%. Filtering by
+  extension alone leaves 92.3%, so the owner cap is what does the work. The cap is a judgement, not a
+  discovered boundary — the sweep is smooth (`owners<=1` 1.6% … `<=10` 19.9%) — and it is a cap, not a
+  ban: a filename named by one to three records still links, which is what the alias path was for.
+- **What this means for existing databases.** Nothing is rewritten on upgrade: rows already in
+  `chunk_entities` stay, and new ingests simply link far less. Links have always been a function of
+  when a document was last processed (nothing re-links older documents when entities are added), and
+  `chunk_entities` has no provenance column, so old alias rows cannot be told apart from name matches
+  after the fact. If you want the old noise gone you have to clean it offline, before or after
+  upgrading. Callers that assumed dense `chunk_entities` coverage will see sparser graphs.
+- Regression: `test/alias-link-gate.test.mjs` (registered in `verify:engine`), verified to fail
+  without the gate.
+
 ### v5.3.0
 - (Published first as `5.3.0-rc.1` on the `next` dist-tag; promoted to `latest` after a canary run of the published artifact against a real project database: default call carries no `graph_boost` and equals explicit `useGraph:false`, the known-item probe from the 2026-08-17 measurement returns the correct gotcha at rank 1, opt-in `true` still exposes `graph_boost`, schema/MCP defaults read `false`.)
 - **Behavior change — `hybridSearch` graph re-ranking is now opt-in** (`useGraph` default `true` → `false`; tool schema, MCP exposure and the manager signature agree). Omitting the argument now means "no graph re-ranking" — a behavior change for callers that relied on the old default, hence a release-candidate first (`next` dist-tag, fleet canary) before stable. Measured 2026-08-17 on three real corpora (self-retrieval, usable samples 120/117/120, summaries off): with the additive graph boost on, the known-item chunk got worse in 46/49/52 samples and better in 3/2/0 (sign test p < 7e-11 per corpus), 106 targets left the top-10 entirely; reproduced on the summaries-on product path (HAL, 20 paired samples: hit@1 10→7, hit@5 18→13). Mechanism: only query-matched/connected entities score, but the per-entity boost saturates the cap quickly, so heavily-linked chunks can outrank the exact chunk even at `vector_similarity` 0. This is a harm-reduced default, not a validated graph improvement: the boost path is unchanged for `useGraph: true` (legacy/experimental re-ranker for back-compat and evaluation; the graph does not generate candidates — for relationship exploration use `openNodes` → `getNeighbors`). Regression lock: `test/search-graph-default.test.mjs`.

package/dist/index.d.ts

@@ -279,6 +279,19 @@ export declare class RAGKnowledgeGraphManager {
         errors?: string[];
     }>;
     private hasCJK;
+    private static readonly WORD_CH;
+    /**
+     * Index of an occurrence of `name` in `text` with non-word neighbours, or -1.
+     *
+     * Runs on the ORIGINAL text, not a lowercased copy. Lowercasing can change length —
+     * 'İ' folds to two units — so an index taken from the folded string does not address the
+     * same character in the original, and buildEntityRangeFinder hands these indices straight
+     * to the codepoint table. (r9-1 made the same call for the regex path; a first cut of this
+     * helper folded first and the İ coordinate test caught it.)
+     */
+    private standaloneIndex;
+    private static readonly ALIAS_FILE_EXT;
+    private looksLikeFilename;
     private buildEntityRangeFinder;
     private buildEntityMatcher;
     private autoLinkEntities;

package/dist/index.js

@@ -2241,6 +2241,70 @@ export class RAGKnowledgeGraphManager {
     hasCJK(text) {
         return /[\u3000-\u9fff\uac00-\ud7af\uff00-\uffef]/.test(text);
     }
+    // `\b` asserts a transition between a word char and a non-word char. When the name itself
+    // *ends* (or starts) with a non-word char — and ours routinely do, e.g. "… Review (2026-05-27)"
+    // — there is no transition to assert, so `\bname\b` can never match however the text reads.
+    // Measured 2026-08-23 on a live 2,913-chunk corpus: 23 standalone occurrences across 13 names
+    // were invisible to the regex. This is not a widening: we still require both neighbours to be
+    // non-word, so "Data" continues not to match inside "Database". It is what `\b` was reaching
+    // for, stated in a way that survives a name whose own edges are punctuation.
+    static WORD_CH = /[A-Za-z0-9_]/;
+    /**
+     * Index of an occurrence of `name` in `text` with non-word neighbours, or -1.
+     *
+     * Runs on the ORIGINAL text, not a lowercased copy. Lowercasing can change length —
+     * 'İ' folds to two units — so an index taken from the folded string does not address the
+     * same character in the original, and buildEntityRangeFinder hands these indices straight
+     * to the codepoint table. (r9-1 made the same call for the regex path; a first cut of this
+     * helper folded first and the İ coordinate test caught it.)
+     */
+    standaloneIndex(text, name, from = 0) {
+        let re;
+        try {
+            re = new RegExp(name.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'), 'gi');
+        }
+        catch {
+            return -1;
+        }
+        re.lastIndex = from;
+        let m;
+        while ((m = re.exec(text)) !== null) {
+            const before = m.index === 0 ? '' : text[m.index - 1];
+            const after = text[m.index + m[0].length] ?? '';
+            if (!RAGKnowledgeGraphManager.WORD_CH.test(before)
+                && !RAGKnowledgeGraphManager.WORD_CH.test(after))
+                return m.index;
+            if (re.lastIndex === m.index)
+                re.lastIndex++;
+        }
+        return -1;
+    }
+    // The alias regex in autoLinkEntities matches anything shaped like "stem.ext", which
+    // includes version strings ("v3.3", "gpt-5.6"), measurements ("1.7mb", "0.465") and
+    // domains/module paths ("github.com", "os.path"). Those are not filenames and linking
+    // on them is pure noise. Whitelist the extensions we actually ship and store.
+    static ALIAS_FILE_EXT = new Set([
+        'md', 'py', 'js', 'mjs', 'cjs', 'ts', 'tsx', 'jsx', 'json',
+        'sh', 'bash', 'zsh', 'toml', 'yaml', 'yml', 'ini', 'cfg', 'conf',
+        'txt', 'log', 'csv', 'tsv', 'sql', 'db', 'zip', 'gz', 'tar',
+        'css', 'scss', 'html', 'htm', 'svg', 'png', 'jpg', 'jpeg', 'gif',
+        'pdf', 'docx', 'pptx', 'xlsx', 'hwp', 'hwpx', 'lock', 'bak',
+    ]);
+    // Deliberately absent: any 5+ character extension (jsonl, ipynb, scss is 4 so it stays).
+    // The extractor regex is `\w{1,4}`, so a longer extension never reaches this set — listing
+    // one would be a dead entry that reads as support. Extend the regex first if that changes.
+    looksLikeFilename(token) {
+        const i = token.lastIndexOf('.');
+        if (i < 1)
+            return false;
+        const stem = token.slice(0, i);
+        const ext = token.slice(i + 1);
+        if (stem.length < 3)
+            return false; // "d.ts" is a fragment, not a file
+        if (/^\d+$/.test(stem))
+            return false; // "2026.md" style numeric stems
+        return RAGKnowledgeGraphManager.ALIAS_FILE_EXT.has(ext);
+    }
     // spec §5.4 (r7-2·r8-1·r9): primary name 의 본문 occurrence range [sCp, eCp).
     // 의미 = buildEntityMatcher 와 동일 (CJK substring / Latin word-boundary) — 여기서
     // 어긋나면 'Data' 가 'Database' 에 새로 링크되는 식으로 의미가 확장된다.
@@ -2301,6 +2365,18 @@ export class RAGKnowledgeGraphManager {
                     if (re.lastIndex === m.index)
                         re.lastIndex++;
                 }
+                // Same correction as buildEntityMatcher — the two must agree or a name links at chunk
+                // level but not at range level. Union, deduped: a hit found by both must not be pushed
+                // twice or a chunk gets counted once per path.
+                const seen = new Set(out.map(r => `${r.s}:${r.e}`));
+                for (let at = this.standaloneIndex(content, name); at >= 0; at = this.standaloneIndex(content, name, at + 1)) {
+                    const s2 = origCpAt(at), e2 = origCpAt(at + name.length); // 대소문자 차이는 길이를 바꾸지 않는다
+                    const key = `${s2}:${e2}`;
+                    if (!seen.has(key)) {
+                        seen.add(key);
+                        out.push({ s: s2, e: e2 });
+                    }
+                }
             }
             catch {
                 pushAllSubstr();
@@ -2319,7 +2395,8 @@ export class RAGKnowledgeGraphManager {
         try {
             const escaped = lower.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
             const re = new RegExp(`\\b${escaped}\\b`, 'i');
-            return (text) => re.test(text);
+            // See standaloneIndex: `\b` cannot fire when the name's own edge is punctuation.
+            return (text) => re.test(text) || this.standaloneIndex(text, name) >= 0;
         }
         catch {
             return (text) => text.toLowerCase().includes(lower);
@@ -2342,6 +2419,73 @@ export class RAGKnowledgeGraphManager {
             // Minimum name length: 2 for CJK (e.g. "할랄"), 4 for Latin (avoid "API", "Bug")
             const MIN_LEN_CJK = 2;
             const MIN_LEN_LATIN = 4;
+            // Observation-derived aliases are only useful when the token identifies few entities.
+            // Measured on a 2,891-chunk / 604-entity corpus (2026-08-22). Denominators matter here,
+            // so both are stated: 66,841 rows in chunk_entities, of which 65,388 (97.8%) exist only
+            // because of an alias hit; counting distinct (chunk, entity) pairs reachable by any alias
+            // gives 80,096. Against that 80,096, "agents.md" alone accounts for 39,248 (49.0%), and
+            // 35,099 (43.8%) have no other alias reason at all. It is held by 88 entities. A filename
+            // that dozens of entities mention is a stopword, not an identifier.
+            //
+            // The owner cap is what does the work: the extension whitelist alone still leaves 92.3%
+            // of alias links. But the cap value is a judgement call, not a discovered boundary — the
+            // sweep is smooth, with no natural knee (share of the 80,096 that survives):
+            //   owners<=1 1.6% · <=2 3.7% · <=3 5.2% · <=4 7.0% · <=5 8.3% · <=8 14.7% · <=10 19.9%
+            // 3 was chosen to keep the intended behaviour (a file named by one or two records, plus
+            // some slack) while cutting the stopword tail. Raising it is cheap and reversible.
+            //
+            // A chunk-frequency cap was measured (a further 1.4pp) and rejected on COST: it needs a
+            // full-corpus scan on every ingest. Note the honest caveat — it was first rejected for
+            // depending on ingest order, but this owner cap has that same property, and so does the
+            // engine as a whole: autoLinkEntities only ever sees the entities that exist at ingest
+            // time, and nothing re-links older documents when entities are added (see
+            // createEntities / addObservations — neither calls this). Links are a function of when a
+            // document was last processed. That predates this gate; it is not introduced by it.
+            const MAX_ALIAS_OWNERS = 3;
+            const aliasOwners = new Map();
+            // Owners is only half the mapping. It asks "does this token point at one entity?" and says
+            // nothing about "does this entity point at one token?" — so an entity that mentions a common
+            // filename ONCE in its observations attaches to every chunk containing that filename, and
+            // because it is the only owner, the cap waves it through. The cap stopped the explosion
+            // (tens of thousands of rows) but not the magnet (one entity on dozens of chunks).
+            // Measured 2026-08-23 after the 6.0.0 cleanup: 2,014 alias-only links remained and 34
+            // entities held 68.1% of them; the biggest had the entity name appearing in ZERO of its
+            // chunks — pulled in entirely by a filename someone mentioned in passing.
+            // So require the mapping in both directions: the token must identify the entity (owners)
+            // AND the entity must identify the token (the token, or its stem, appears in the name).
+            // This is a structural condition, not a threshold — there is no knee to tune and it keeps
+            // meaning the same as the corpus grows. Chunk-frequency caps were measured instead
+            // (585ms full scan, so cost was NOT the objection) and rejected because the sweep is smooth
+            // and every cut also removed legitimate links.
+            const aliasNamesTheEntity = (entityName, token) => {
+                const lower = entityName.toLowerCase();
+                if (lower.includes(token))
+                    return true;
+                const dot = token.lastIndexOf('.');
+                const stem = dot > 0 ? token.slice(0, dot) : token;
+                return stem.length >= 4 && lower.includes(stem);
+            };
+            for (const e of entities) {
+                if (!e.observations)
+                    continue;
+                let obs;
+                try {
+                    obs = JSON.parse(e.observations);
+                }
+                catch {
+                    continue;
+                }
+                const seen = new Set();
+                for (const ob of obs) {
+                    const pm = String(ob).match(/[\w\-]+\.\w{1,4}\b/g);
+                    if (pm)
+                        for (const p of pm)
+                            if (p.length >= 4)
+                                seen.add(p.toLowerCase());
+                }
+                for (const t of seen)
+                    aliasOwners.set(t, (aliasOwners.get(t) ?? 0) + 1);
+            }
             const insertStmt = this.db.prepare(`
         INSERT OR IGNORE INTO chunk_entities (chunk_rowid, entity_id) VALUES (?, ?)
       `);
@@ -2351,7 +2495,9 @@ export class RAGKnowledgeGraphManager {
                 if (entity.name.length < minLen)
                     continue;
                 const nameMatcher = this.buildEntityMatcher(entity.name);
-                // Also collect observation-derived aliases (short keywords from observations)
+                // Also collect observation-derived aliases (short keywords from observations).
+                // Gated: the token must look like a filename AND identify at most MAX_ALIAS_OWNERS
+                // entities. Ungated, "agents.md" linked 88 entities to every chunk that mentioned it.
                 const aliases = [];
                 if (entity.observations) {
                     let obs;
@@ -2366,9 +2512,32 @@ export class RAGKnowledgeGraphManager {
                         const pathMatch = ob.match(/[\w\-]+\.\w{1,4}\b/g);
                         if (pathMatch) {
                             for (const p of pathMatch) {
-                                if (p.length >= 4) {
-                                    aliases.push((text) => text.toLowerCase().includes(p.toLowerCase()));
-                                }
+                                if (p.length < 4)
+                                    continue;
+                                const tok = p.toLowerCase();
+                                if (!this.looksLikeFilename(tok))
+                                    continue; // "v3.3", "gpt-5.6", "0.465"
+                                if ((aliasOwners.get(tok) ?? 0) > MAX_ALIAS_OWNERS)
+                                    continue; // shared = identifies nothing
+                                if (!aliasNamesTheEntity(entity.name, tok))
+                                    continue; // mentioned in passing = magnet
+                                // Bare substring matching links "foo.py" to a chunk saying "notfoo.pyc".
+                                // Require the token to stand alone: no filename character on either side.
+                                const boundary = /[\w\-.]/;
+                                aliases.push((text) => {
+                                    const hay = text.toLowerCase();
+                                    let from = 0;
+                                    for (;;) {
+                                        const i = hay.indexOf(tok, from);
+                                        if (i < 0)
+                                            return false;
+                                        const before = i === 0 ? '' : hay[i - 1];
+                                        const after = hay[i + tok.length] ?? '';
+                                        if (!boundary.test(before) && !boundary.test(after))
+                                            return true;
+                                        from = i + 1;
+                                    }
+                                });
                             }
                         }
                     }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "rag-memory-epf-mcp",
-  "version": "5.3.1",
+  "version": "6.0.1",
   "engines": {
     "node": ">=24"
   },
@@ -45,7 +45,7 @@
     "prepare": "npm run build",
     "watch": "tsc --watch",
     "verify:invariants": "node test/chunk-invariants.test.mjs",
-    "verify:engine": "node test/engine-smoke.test.mjs && node test/launch-smoke.test.mjs && node test/sync-atomicity.test.mjs && node test/dedup.test.mjs && node test/search-degradation.test.mjs && node test/entity-embed-cap.test.mjs && node test/migration12.test.mjs && node test/model-cache.test.mjs && node test/embedding-gate.test.mjs && node test/lazy-boot.test.mjs && node test/reconciliation.test.mjs && node test/backfill.test.mjs && node test/fts-query.test.mjs && node test/search-contracts.test.mjs && node test/tool-contracts.test.mjs && node test/bounded-exit.test.mjs && node test/observation-schema.test.mjs && node test/search-graph-default.test.mjs && node test/observation-migration.test.mjs && node test/observation-lifecycle.test.mjs && node test/observation-contracts.test.mjs && node test/observation-search.test.mjs && node test/observation-cascade.test.mjs && node test/observation-realdata.test.mjs && node test/chunker-c.test.mjs && node test/migration14.test.mjs && node test/chunk-params-validation.test.mjs && node test/vector-reuse.test.mjs && node test/entity-range-linking.test.mjs && node test/stats-chunking.test.mjs && node test/migration14-realdata.test.mjs && node test/migration14-realdata-sync.test.mjs && node test/search-summaries-off.test.mjs && node test/document-return-contracts.test.mjs && node test/observation-date-prefix.test.mjs && node test/delete-entities-cascade.test.mjs && node test/backup-publish-portable.test.mjs && node test/graph-context-explain.test.mjs && node test/eval-graph-role-libs.test.mjs && node test/eval-graph-role-t5b.test.mjs && node --test test/eval-graph-role-t8-fix.test.mjs && node --test test/eval-graph-role-t7-upstream.test.mjs && node --test test/eval-graph-role-t11-decision.test.mjs && node --test test/eval-graph-role-prereq-fix.test.mjs",
+    "verify:engine": "node test/engine-smoke.test.mjs && node test/launch-smoke.test.mjs && node test/sync-atomicity.test.mjs && node test/dedup.test.mjs && node test/search-degradation.test.mjs && node test/entity-embed-cap.test.mjs && node test/migration12.test.mjs && node test/model-cache.test.mjs && node test/embedding-gate.test.mjs && node test/lazy-boot.test.mjs && node test/reconciliation.test.mjs && node test/backfill.test.mjs && node test/fts-query.test.mjs && node test/search-contracts.test.mjs && node test/tool-contracts.test.mjs && node test/bounded-exit.test.mjs && node test/observation-schema.test.mjs && node test/search-graph-default.test.mjs && node test/observation-migration.test.mjs && node test/observation-lifecycle.test.mjs && node test/observation-contracts.test.mjs && node test/observation-search.test.mjs && node test/observation-cascade.test.mjs && node test/observation-realdata.test.mjs && node test/chunker-c.test.mjs && node test/migration14.test.mjs && node test/chunk-params-validation.test.mjs && node test/vector-reuse.test.mjs && node test/entity-range-linking.test.mjs && node test/alias-link-gate.test.mjs && node test/entity-name-boundary.test.mjs && node test/stats-chunking.test.mjs && node test/migration14-realdata.test.mjs && node test/migration14-realdata-sync.test.mjs && node test/search-summaries-off.test.mjs && node test/document-return-contracts.test.mjs && node test/observation-date-prefix.test.mjs && node test/delete-entities-cascade.test.mjs && node test/backup-publish-portable.test.mjs && node test/graph-context-explain.test.mjs && node test/eval-graph-role-libs.test.mjs && node test/eval-graph-role-t5b.test.mjs && node --test test/eval-graph-role-t8-fix.test.mjs && node --test test/eval-graph-role-t7-upstream.test.mjs && node --test test/eval-graph-role-t11-decision.test.mjs && node --test test/eval-graph-role-prereq-fix.test.mjs",
     "test": "npm run build && npm run verify:invariants && npm run verify:engine",
     "prepublishOnly": "npm run build && npm run verify:invariants && npm run verify:engine"
   },