mindforge-memory 6.4.8__tar.gz → 6.4.9__tar.gz

{mindforge_memory-6.4.8/mindforge_memory.egg-info → mindforge_memory-6.4.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mindforge-memory
-Version: 6.4.8
+Version: 6.4.9
 Summary: AI Agent 终身记忆系统 - 四层记忆架构 · 知识图谱 · 多模态支持
 License: MIT
 Project-URL: Homepage, https://github.com/opok-ops/MindForge
@@ -36,11 +36,11 @@ Dynamic: license-file
 
 MindForge gives AI agents a persistent, structured long-term memory: a four-tier lifecycle model, knowledge graphs, hybrid retrieval, end-to-end encryption, and MCP / CLI / REST interfaces. All data stays on your machine.
 
-> **The story** — one developer, seven months, from v1.0 (March 2026) to v6.4.8: no team, no funding, 1,252 passing tests. [Read the journey →](https://opok-ops.github.io/MindForge/#developer)
+> **The story** — one developer, seven months, from v1.0 (March 2026) to v6.4.9: no team, no funding, 1,257 passing tests. [Read the journey →](https://opok-ops.github.io/MindForge/#developer)
 
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
-[![Version](https://img.shields.io/badge/version-6.4.8-green.svg)](https://github.com/opok-ops/MindForge/releases)
+[![Version](https://img.shields.io/badge/version-6.4.9-green.svg)](https://github.com/opok-ops/MindForge/releases)
 [![Stars](https://img.shields.io/github/stars/opok-ops/MindForge.svg)](https://github.com/opok-ops/MindForge/stargazers)
 [![Forks](https://img.shields.io/github/forks/opok-ops/MindForge.svg)](https://github.com/opok-ops/MindForge/forks)
 [![Open Issues](https://img.shields.io/github/issues/opok-ops/MindForge.svg)](https://github.com/opok-ops/MindForge/issues)
@@ -95,15 +95,15 @@ MindForge stats
 ```
 
 
-## Scale & Quality Benchmarks (v6.4.8, measured)
+## Scale & Quality Benchmarks (v6.4.9, measured)
 
 | Scale | add throughput | search p50 (rare token) | search p50 (realistic mixed) | search p50 (pathological all-common) | db size |
 |---|---|---|---|---|---|
 | 200 entries (baseline) | — | 1.8 ms | — | — | <1 MB |
 | 10,000 | 1,080/s | ~9.5 ms | ~16 ms | — | 10 MB |
-| 100,000 | ~582/s | 6.8 ms | 135 ms | 122 ms (−44%) | 98 MB |
+| 100,000 (~10M chars) | ~546/s | 6.8 ms | 3.0 ms | 0.4 ms | 98 MB |
 
-- Pathological queries (all-common words, synthetic template data): engine-level steady-state **4.0ms at 30k** (was 3825ms, −99.9%) — inverted-index fallback is now shortest-chain-first with a hard traversal budget; the remaining 100k cost is FTS5 high-frequency-phrase scoring (v6.5 target, honestly documented).
+- **Ten-million-character library, millisecond-class across all query types (v6.4.9)**: pathological 218→**0.4ms** (−99.8%), mixed 143→**3.0ms** (−98%), rare 6.8ms, combined 17.8ms (100k full-pipeline, clean environment). The last bottleneck — FTS5 high-frequency-phrase bm25 scoring — is now pruned query-side via trigram-df pre-check (df upper bound = min trigram df, pruned at 5% corpus size, symmetric to the TF-IDF max_df design; df served from an in-process cache — fts5vocab is a streaming virtual table, per-term probes scan it fully).
 - **Index-blindness fix (v6.4.8)**: TF-IDF vocabulary was locked to the first 5 documents since v6.4.6 — incremental adds indexed as empty vectors (TF-IDF effectively half-dead; previous numbers carried by FTS5/fuzzy). Fixed with incremental vocabulary growth; all pipeline numbers above are now *true full-pipeline* measurements.
 - Retrieval quality (fixed-seed, `benchmarks/quality_bench.py`, 2k entries): needle Recall@1 = 1.0, MRR = 1.0, topic Recall@1 = 1.0, noise false-recall = 0.
 - Full LLM-judged benchmarks (LoCoMo / LongMemEval) — interface ready, not yet run (requires external LLM API key).
@@ -111,7 +111,9 @@ MindForge stats
 
 ## What's New
 
-v6.1.3 – v6.4.8 shipped in ten days (2026-11-01 → 11-11; three releases in the last two days), guarded by **1,253 tests** (1,252 passing, 1 environment-skipped).
+v6.1.3 – v6.4.9 shipped in ten days (2026-11-01 → 11-11; four releases in the last two days), guarded by **1,258 tests** (1,257 passing, 1 environment-skipped).
+
+- **v6.4.9 — ten-million-char library at millisecond latency: FTS5 high-frequency-phrase scoring pruned** (2026-11-11). The last honest bottleneck from v6.4.8's ledger. Root cause (30k per-path timing): under the trigram tokenizer, corpus-wide phrases (template data's "陈述性记忆") match every row — bm25 full-match scoring cost 50.5ms/query, the dominant remainder of 100k mixed latency. Fix, query-side df pre-pruning symmetric to TF-IDF max_df: a phrase's df upper bound = min df of its trigrams (phrase matches ⊆ every trigram's row set); prune at 5% of corpus size; keep the lowest-ub phrase if all pruned. df is served from an in-process cache keyed by `max(rowid)` (add/update invalidate; hard-delete leaves df slightly stale = prunes more conservatively) — **two real pitfalls found en route: fts5vocab is a *streaming virtual table* — per-term probes AND `IN` batches trigger full scans (combined regressed 17→45ms before the cache)**. Plus: FTS5 path skipped on pathological fallback (no discriminating signal). Clean-environment 100k re-bench: pathological 122→**0.4ms** (−99.7%), mixed 134.9→**3.0ms** (−98%), all query types millisecond-class at ~10M chars. AI-flavored comment cleanup (essay-style docstrings → engineer-speak, info preserved). 5 new guards.
 
 - **v6.4.8 — pathological query path & index-blindness fix: O(N) fallback eliminated, TF-IDF actually indexing** (2026-11-11). Two deeper findings beyond the v6.4.6 stress test: (1) the all-common-word fallback traversed *all* inverted chains at full length (O(N×dims), 209ms p50 at 100k), and the 0.5×N max_df threshold still let half-corpus chains into the active set; (2) **the TF-IDF vocabulary had been locked to the first 5 documents** — every incremental add after that indexed as an empty vector (the TF-IDF path was effectively half-dead since v6.4.6; published numbers were carried by FTS5/fuzzy). Fix, six cuts: max_df tightened 0.5×N→**0.05×N**; fallback semantics → **shortest-chain-first** (≤3 dims, per-chain cap 5,000, islice); fuzzy candidate budget (SQL cap 2,000 with exact-substring-first ordering); fuzzy gating (runs only on zero TF-IDF hits — the true typo-correction scenario; encrypted mode unaffected); `incremental_fit()` vocabulary growth for adds after fit (old vectors not recomputed — new tokens contribute zero on old docs mathematically); empty-query fallback-signal reset. 30k repro: pathological steady-state 3825→**4.0ms** (−99.9%), mixed pipeline 1607→1.2ms, fuzzy single-path −93%. 100k re-bench: pathological −44%, all numbers now true full-pipeline. Remaining bottleneck honestly documented: FTS5 high-frequency-phrase scoring (v6.5). New observable `_last_traversed` + deterministic budget contracts. 9 new guards.
 
@@ -186,7 +188,7 @@ v6.1.3 – v6.4.8 shipped in ten days (2026-11-01 → 11-11; three releases in t
 
 ```
 ┌──────────────────────────────────────────────────────┐
-│           MindForge v6.4.8            │
+│           MindForge v6.4.9            │
 │  Cognitive Layer  Personality · KnowledgeGraph       │
 │                   MemoryEvolution · FederatedMemory  │
 ├──────────────────────────────────────────────────────┤

{mindforge_memory-6.4.8 → mindforge_memory-6.4.9}/README.md

@@ -6,11 +6,11 @@
 
 MindForge gives AI agents a persistent, structured long-term memory: a four-tier lifecycle model, knowledge graphs, hybrid retrieval, end-to-end encryption, and MCP / CLI / REST interfaces. All data stays on your machine.
 
-> **The story** — one developer, seven months, from v1.0 (March 2026) to v6.4.8: no team, no funding, 1,252 passing tests. [Read the journey →](https://opok-ops.github.io/MindForge/#developer)
+> **The story** — one developer, seven months, from v1.0 (March 2026) to v6.4.9: no team, no funding, 1,257 passing tests. [Read the journey →](https://opok-ops.github.io/MindForge/#developer)
 
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
-[![Version](https://img.shields.io/badge/version-6.4.8-green.svg)](https://github.com/opok-ops/MindForge/releases)
+[![Version](https://img.shields.io/badge/version-6.4.9-green.svg)](https://github.com/opok-ops/MindForge/releases)
 [![Stars](https://img.shields.io/github/stars/opok-ops/MindForge.svg)](https://github.com/opok-ops/MindForge/stargazers)
 [![Forks](https://img.shields.io/github/forks/opok-ops/MindForge.svg)](https://github.com/opok-ops/MindForge/forks)
 [![Open Issues](https://img.shields.io/github/issues/opok-ops/MindForge.svg)](https://github.com/opok-ops/MindForge/issues)
@@ -65,15 +65,15 @@ MindForge stats
 ```
 
 
-## Scale & Quality Benchmarks (v6.4.8, measured)
+## Scale & Quality Benchmarks (v6.4.9, measured)
 
 | Scale | add throughput | search p50 (rare token) | search p50 (realistic mixed) | search p50 (pathological all-common) | db size |
 |---|---|---|---|---|---|
 | 200 entries (baseline) | — | 1.8 ms | — | — | <1 MB |
 | 10,000 | 1,080/s | ~9.5 ms | ~16 ms | — | 10 MB |
-| 100,000 | ~582/s | 6.8 ms | 135 ms | 122 ms (−44%) | 98 MB |
+| 100,000 (~10M chars) | ~546/s | 6.8 ms | 3.0 ms | 0.4 ms | 98 MB |
 
-- Pathological queries (all-common words, synthetic template data): engine-level steady-state **4.0ms at 30k** (was 3825ms, −99.9%) — inverted-index fallback is now shortest-chain-first with a hard traversal budget; the remaining 100k cost is FTS5 high-frequency-phrase scoring (v6.5 target, honestly documented).
+- **Ten-million-character library, millisecond-class across all query types (v6.4.9)**: pathological 218→**0.4ms** (−99.8%), mixed 143→**3.0ms** (−98%), rare 6.8ms, combined 17.8ms (100k full-pipeline, clean environment). The last bottleneck — FTS5 high-frequency-phrase bm25 scoring — is now pruned query-side via trigram-df pre-check (df upper bound = min trigram df, pruned at 5% corpus size, symmetric to the TF-IDF max_df design; df served from an in-process cache — fts5vocab is a streaming virtual table, per-term probes scan it fully).
 - **Index-blindness fix (v6.4.8)**: TF-IDF vocabulary was locked to the first 5 documents since v6.4.6 — incremental adds indexed as empty vectors (TF-IDF effectively half-dead; previous numbers carried by FTS5/fuzzy). Fixed with incremental vocabulary growth; all pipeline numbers above are now *true full-pipeline* measurements.
 - Retrieval quality (fixed-seed, `benchmarks/quality_bench.py`, 2k entries): needle Recall@1 = 1.0, MRR = 1.0, topic Recall@1 = 1.0, noise false-recall = 0.
 - Full LLM-judged benchmarks (LoCoMo / LongMemEval) — interface ready, not yet run (requires external LLM API key).
@@ -81,7 +81,9 @@ MindForge stats
 
 ## What's New
 
-v6.1.3 – v6.4.8 shipped in ten days (2026-11-01 → 11-11; three releases in the last two days), guarded by **1,253 tests** (1,252 passing, 1 environment-skipped).
+v6.1.3 – v6.4.9 shipped in ten days (2026-11-01 → 11-11; four releases in the last two days), guarded by **1,258 tests** (1,257 passing, 1 environment-skipped).
+
+- **v6.4.9 — ten-million-char library at millisecond latency: FTS5 high-frequency-phrase scoring pruned** (2026-11-11). The last honest bottleneck from v6.4.8's ledger. Root cause (30k per-path timing): under the trigram tokenizer, corpus-wide phrases (template data's "陈述性记忆") match every row — bm25 full-match scoring cost 50.5ms/query, the dominant remainder of 100k mixed latency. Fix, query-side df pre-pruning symmetric to TF-IDF max_df: a phrase's df upper bound = min df of its trigrams (phrase matches ⊆ every trigram's row set); prune at 5% of corpus size; keep the lowest-ub phrase if all pruned. df is served from an in-process cache keyed by `max(rowid)` (add/update invalidate; hard-delete leaves df slightly stale = prunes more conservatively) — **two real pitfalls found en route: fts5vocab is a *streaming virtual table* — per-term probes AND `IN` batches trigger full scans (combined regressed 17→45ms before the cache)**. Plus: FTS5 path skipped on pathological fallback (no discriminating signal). Clean-environment 100k re-bench: pathological 122→**0.4ms** (−99.7%), mixed 134.9→**3.0ms** (−98%), all query types millisecond-class at ~10M chars. AI-flavored comment cleanup (essay-style docstrings → engineer-speak, info preserved). 5 new guards.
 
 - **v6.4.8 — pathological query path & index-blindness fix: O(N) fallback eliminated, TF-IDF actually indexing** (2026-11-11). Two deeper findings beyond the v6.4.6 stress test: (1) the all-common-word fallback traversed *all* inverted chains at full length (O(N×dims), 209ms p50 at 100k), and the 0.5×N max_df threshold still let half-corpus chains into the active set; (2) **the TF-IDF vocabulary had been locked to the first 5 documents** — every incremental add after that indexed as an empty vector (the TF-IDF path was effectively half-dead since v6.4.6; published numbers were carried by FTS5/fuzzy). Fix, six cuts: max_df tightened 0.5×N→**0.05×N**; fallback semantics → **shortest-chain-first** (≤3 dims, per-chain cap 5,000, islice); fuzzy candidate budget (SQL cap 2,000 with exact-substring-first ordering); fuzzy gating (runs only on zero TF-IDF hits — the true typo-correction scenario; encrypted mode unaffected); `incremental_fit()` vocabulary growth for adds after fit (old vectors not recomputed — new tokens contribute zero on old docs mathematically); empty-query fallback-signal reset. 30k repro: pathological steady-state 3825→**4.0ms** (−99.9%), mixed pipeline 1607→1.2ms, fuzzy single-path −93%. 100k re-bench: pathological −44%, all numbers now true full-pipeline. Remaining bottleneck honestly documented: FTS5 high-frequency-phrase scoring (v6.5). New observable `_last_traversed` + deterministic budget contracts. 9 new guards.
 
@@ -156,7 +158,7 @@ v6.1.3 – v6.4.8 shipped in ten days (2026-11-01 → 11-11; three releases in t
 
 ```
 ┌──────────────────────────────────────────────────────┐
-│           MindForge v6.4.8            │
+│           MindForge v6.4.9            │
 │  Cognitive Layer  Personality · KnowledgeGraph       │
 │                   MemoryEvolution · FederatedMemory  │
 ├──────────────────────────────────────────────────────┤

{mindforge_memory-6.4.8 → mindforge_memory-6.4.9}/mindforge/core/indexer.py

@@ -205,20 +205,15 @@ class VectorIndex:
                   if qval > _EPS_REL * query_norm
                   and len(self._postings.get(idx, ())) <= _max_df_len]
         self._last_traversed = 0
-        # v6.4.8: 病态回退信号——查询全部维度超 max_df（全库词）时
-        # 置位，供上层 QueryEngine 跳过 fuzzy 兜底路（全库词查询对
-        # fuzzy 的边际价值为零，详见 query.py 路 2 注释）。
+        # v6.4.8: 全库词回退信号（供 query.py 跳过 fuzzy/FTS5 兜底）
         self.last_fallback = False
         if not active:
             self.last_fallback = True
-            # v6.4.8 perf: 病态回退（全维被剪）不再遍历全部倒排链。
-            # 旧回退 list(q.items()) = O(N×查询维)，100k 全库词查询
-            # p50 209ms 的根因。全库词贡献是常数，top-K 只由相对区分
-            # 度决定，因此回退语义改为「最短链优先」：
-            #   1) 按链长升序取最短 _FALLBACK_DIMS 维（q 中区分度最高
-            #      的信号，哪怕也被 max_df 剪掉）
-            #   2) 单链遍历截断到 _TRAVERSE_CAP 项——全平分数下截断
-            #      不改 top-K 语义，优先守住延迟上界
+            # v6.4.8: 回退不再遍历全部倒排链（旧 O(N×dims) 是 100k
+            # 全库词查询 209ms 的根因）。全库词点积贡献是常数，top-K
+            # 只由相对区分度决定：取最短 _FALLBACK_DIMS 维（区分度最高
+            # 的信号），单链截断 _TRAVERSE_CAP 项（平分数下截断不改
+            # top-K 语义）。
             _FALLBACK_DIMS = 3
             _TRAVERSE_CAP = 5000
             ranked = sorted(
@@ -291,6 +286,9 @@ class IndexEngine:
         self.vector_index = VectorIndex()
         # v6.4.8: 病态回退信号门面透传（见 search()）
         self.last_fallback = False
+        # v6.4.9: FTS5 trigram df 进程内缓存（见 _prune_fts_query）
+        self._fts_vocab_cache: Optional[Dict[str, int]] = None
+        self._fts_vocab_rowid: int = -1
         self._fitted = False
         self._hydrated = False
         self._doc_texts: Dict[str, str] = {}
@@ -348,15 +346,10 @@ class IndexEngine:
         """索引记忆"""
         self._doc_texts[doc_id] = text
 
-        # v5.6.5 P1 #5 的渐进触发点保留（LRU 守卫契约：add 即索引，
-        # _doc_texts/vectors/postings 三结构同步有界）。
-        # v6.4.8 fix（祖传深坑）：旧逻辑这里 fit 一次后，后续增量 add 的
-        # transform 全部 vocab miss → 空向量入索引（文档「失索引」）——
-        # 词表锁死在前 5 条文档的 token，100k 压测的 TF-IDF 路实际长期
-        # 半瘫痪（延迟数字由 FTS5/fuzzy 扛）。修复：fit 后的增量 add 先
-        # 走 incremental_fit 扩词表（新 token 补 vocab + df/idf 增量，
-        # 旧文档向量不回算——新词在旧文档 tf=0 无贡献，旧词 IDF 漂移为
-        # 可接受近似），再 transform 入索引。
+        # v5.6.5 P1 #5: 渐进 fit 触发点（LRU 守卫契约：add 即索引）。
+        # v6.4.8 fix: fit 后的增量 add 先 incremental_fit 扩词表再
+        # transform——旧逻辑词表锁死在前 5 条文档，此后增量 add 的
+        # transform 全 vocab miss（空向量入索引，TF-IDF 路半瘫痪）。
         if not self._fitted and len(self._doc_texts) >= 5:
             self._fit_vectorizer()
         elif self._fitted:
@@ -366,7 +359,7 @@ class IndexEngine:
             vector_dict = self.vectorizer.transform(text)
             self.vector_index.add(doc_id, vector_dict, metadata)
 
-        # v5.6.5 P1 #5：写入后做 LRU 淘汰（覆盖写不增加条目数，故在 add 之后）
+        # v5.6.5 P1 #5: 写入后 LRU 淘汰
         self._evict_if_needed()
 
     def remove_memory(self, doc_id: str):
@@ -446,9 +439,16 @@ class IndexEngine:
 
     def fts_search(self, conn: sqlite3.Connection, query: str,
                    top_k: int = 10) -> List[Tuple[str, float]]:
-        """FTS5 全文搜索（v5.4.7 修复：特殊字符转义）"""
+        """FTS5 全文搜索（v5.4.7 修复：特殊字符转义）
+
+        v6.4.9: 多词查询先经 _prune_fts_query 剔除高频短语——
+        trigram 索引下全库短语的 bm25 评分是 100k 延迟的大头。
+        """
         try:
             escaped = self._escape_fts5_query(query)
+            if not escaped:
+                return []
+            escaped = self._prune_fts_query(conn, query, escaped)
             if not escaped:
                 return []
             rows = conn.execute("""
@@ -474,3 +474,62 @@ class IndexEngine:
             return results
         except sqlite3.OperationalError:
             return []
+
+    # v6.4.9: 高频短语剪枝比例（与 TF-IDF max_df 0.05 对齐）
+    _FTS_MAX_DF_RATIO = 0.05
+
+    def _prune_fts_query(self, conn: sqlite3.Connection, query: str,
+                         escaped: str) -> str:
+        """剔除 MATCH 里必然无区分度的高频短语。
+
+        短语 df 上界 = min(该词全部 trigram 的 df)（短语匹配行 ⊆
+        每个 trigram 行集的交集）。上界超库容 5% 的词剔除；全被剔除
+        时留上界最小的那个，防空 MATCH。
+        df 来自 fts5vocab 的进程内缓存（首查全表读一次，此后 O(1)
+        点查；以 memories.max_rowid 为失效标记，覆盖 add/update；
+        硬删导致的 df 缓存偏大只会让剪枝偏保守，5% 阈值级近似）。
+        """
+        tokens = [t for t in query.split() if t]
+        if len(tokens) <= 1:
+            return escaped
+        try:
+            row = conn.execute(
+                "SELECT max(rowid) FROM memories").fetchone()
+            total = (row[0] or 0) if row else 0
+            if total < 200:
+                return escaped  # 小库 df 语义不稳
+            if (self._fts_vocab_cache is None
+                    or total != self._fts_vocab_rowid):
+                # v6.4.9: vocab 是流式虚拟表，逐 term 点查/IN 批查
+                # 都触发全表扫（combined 100k 45ms 回退的根因）——
+                # 全表读一次进 dict，之后纯内存点查。
+                conn.execute(
+                    "CREATE VIRTUAL TABLE IF NOT EXISTS memory_fts_v "
+                    "USING fts5vocab('memory_fts', 'row')")
+                self._fts_vocab_cache = dict(
+                    conn.execute("SELECT term, doc FROM memory_fts_v"))
+                self._fts_vocab_rowid = total
+            df_map = self._fts_vocab_cache
+            budget = self._FTS_MAX_DF_RATIO * total
+            kept = []
+            best, best_ub = None, None
+            for tok in tokens:
+                if len(tok) < 3:
+                    ub = 0  # trigram 查不了短词，保留
+                else:
+                    ub = min(df_map.get(tok[i:i + 3], 0)
+                             for i in range(len(tok) - 2))
+                if ub <= budget:
+                    kept.append(tok)
+                if best_ub is None or ub < best_ub:
+                    best, best_ub = tok, ub
+            if not kept:
+                kept = [best] if best else []
+                return " OR ".join(
+                    '"' + t.replace('"', '""') + '"' for t in kept)
+            if len(kept) == len(tokens):
+                return escaped
+            return " OR ".join(
+                '"' + t.replace('"', '""') + '"' for t in kept)
+        except sqlite3.Error:
+            return escaped

{mindforge_memory-6.4.8 → mindforge_memory-6.4.9}/mindforge/core/query.py

@@ -257,27 +257,15 @@ class QueryEngine:
                     _bd(doc_id)["winner"] = "tfidf"
                     _bd(doc_id)["final"] = s
 
-        # 路 2：Fuzzy（TF-IDF 召回不足时补充；v5.7.1：加密模式下总是跑，
-        # 供加密兜底复用结果，避免二次全表扫描）
-        # v6.4.8 perf/语义: 触发条件收紧为「TF-IDF 零命中」：
-        #   1) 病态回退（index.last_fallback，全库词查询）→ 跳过。fuzzy
-        #      的设计价值是「拼写纠错/近似词」兜底，全库词查询里不存在
-        #      「没被 TF-IDF 命中的稀有词」，fuzzy 只会给出与全库同分数
-        #      的噪声结果（边际价值为零）；
-        #   2) TF-IDF 已有命中（含低分——cosine 归一化下模板语料的命中
-        #      常在 0.2-0.3 边缘，但词命中本身已证明「查询词在库」）→
-        #      跳过。fuzzy 兜底的真场景是「查询词全不在库」（打错字），
-        #      此时 TF-IDF 必然空结果。mixed 实测（「稀有锚+常见词」）：
-        #      TF-IDF 15 命中 + FTS5 30 命中已足够填 top-K，fuzzy 跑
-        #      cap-2000 候选纯属白税（30k 实测 140ms/查询）。
-        #   3) 性能——fuzzy 候选集对无区分度/长查询会放大成数万条
-        #      （短条目库 LENGTH<4×len(q) 全体进候选），见 storage.py
-        #      v6.4.8 候选预算注释。encrypted 模式不受此影响（仍全跑
-        #      供加密兜底复用）。
+        # 路 2：Fuzzy（拼写纠错兜底；v5.7.1：加密模式下总是跑供加密
+        # 兜底复用，避免二次全表扫描）
+        # v6.4.8: 触发收紧为「TF-IDF 零有效命中」。fuzzy 的价值在
+        # 「查询词全不在库」（打错字场景），此时 TF-IDF 必空。有命中
+        # （含低分）时 fuzzy 只补凑数条目还白扫候选集（30k 实测
+        # 140ms/查询）。全库词回退（tfidf_fallback）同理跳过。
+        # 注意按非零分判定：空查询走零分 top-K 兼容分支，
+        # tfidf_results 恒非空，不看分会死锁拼写兜底。
         tfidf_fallback = getattr(self.index, "last_fallback", False)
-        # v6.4.8: 「有效命中」= TF-IDF 存在非零分结果。空查询（全 miss/
-        # 零范数）走文档序 0 分 top-K 兼容分支，tfidf_results 恒非空但
-        # 全零分——必须按分数判定，否则 fuzzy 拼写兜底被死锁。
         tfidf_effective = any(s > 0.0 for _, s in tfidf_results)
         fuzzy_hits = []
         if (self.storage.encrypted
@@ -296,27 +284,29 @@ class QueryEngine:
                         _bd(entry.id)["final"] = s
 
         # 路 3：FTS5 全文检索
-        # v6.4.8 perf note: 尝试过「TF-IDF 有命中时跳过 FTS5」的门控，
-        # 被混合质量基线测试拦下（hybrid 0.875 < single 0.915，FTS5 的
-        # 补充分对融合质量有真实贡献）。正确方向是 FTS5 内部的
-        # ORDER BY rank top-k 优化（见 indexer.fts_search）。
+        # v6.4.9: 全库词回退时跳过（FTS5 对高频短语的 bm25 全匹配行
+        # 评分是病态查询的主要剩余成本，且无区分度信号）。正常查询
+        # 走 fts_search 的 df 剪枝。
         fts5_used = False
-        # v5.6.2 安全修复：初始化默认值，防止异常路径下 UnboundLocalError
+        # v5.6.2: 防异常路径下 UnboundLocalError
         strategy_used = "tfidf+fuzzy"
         try:
-            conn = self.storage._get_conn()
-            fts_results = self.index.fts_search(conn, query, top_k=max_results * 3)
-            for doc_id, s in fts_results:
-                if explain:
-                    _bd(doc_id)["fts5"] = max(s, _bd(doc_id)["fts5"])
-                if s >= score_map.get(doc_id, 0.0):
-                    score_map[doc_id] = s
+            if not tfidf_fallback:
+                conn = self.storage._get_conn()
+                fts_results = self.index.fts_search(conn, query, top_k=max_results * 3)
+                for doc_id, s in fts_results:
                     if explain:
-                        _bd(doc_id)["winner"] = "fts5"
-                        _bd(doc_id)["final"] = s
-            if fts_results:
-                fts5_used = True
-                strategy_used = "tfidf+fuzzy+fts5"
+                        _bd(doc_id)["fts5"] = max(s, _bd(doc_id)["fts5"])
+                    if s >= score_map.get(doc_id, 0.0):
+                        score_map[doc_id] = s
+                        if explain:
+                            _bd(doc_id)["winner"] = "fts5"
+                            _bd(doc_id)["final"] = s
+                if fts_results:
+                    fts5_used = True
+                    strategy_used = "tfidf+fuzzy+fts5"
+                else:
+                    strategy_used = "tfidf+fuzzy"
             else:
                 strategy_used = "tfidf+fuzzy"
         except Exception:

{mindforge_memory-6.4.8 → mindforge_memory-6.4.9}/mindforge/core/storage.py

@@ -9127,19 +9127,10 @@ class StorageEngine:
                     params_or.append(f"%{_ch}%")
         cand_query = base_query + " AND (" + " OR ".join(conds) + ")"
 
-        # v6.4.8 perf: 候选集预算上限 + 区分度优先排序。
-        # 病理：短条目库（content < 4×len(query) 全体进候选）+ 16 字级
-        # 中文长查询 → 候选≈全库，30k 实测 3 万条逐条反序列化+SequenceMatcher
-        # = 1642ms（100k 语料条目 50-70 字时约半库过长度阈值，候选 ~5 万
-        # = 212ms 白税）。预算语义（与 TF-IDF max_df 0.05×N 同一哲学）：
-        #   - 正常查询（含区分词）LIKE 命中 + 短文本全体仍远小于预算，
-        #     结果零变化；
-        #   - 候选超预算只发生在「查询对全库无区分度」的病态场景——此时
-        #     fuzzy 全体候选分数趋同，截断不改 top-K 语义（保延迟优先）。
-        # 排序：content LIKE %q%（精确子串命中，最强 fuzzy 信号——打错字
-        # 场景「fact-7770」vs「fact-777」也走这条路先保住）排最前，其余
-        # 按 rowid（插入序）。截断发生在「短文本兜底路」而不是真命中。
-        # encrypted 兜底路径同享此预算（候选爆炸对加密库同样不可接受）。
+        # v6.4.8: 候选集预算。短条目库 + 16 字级中文查询会让候选
+        # ≈ 全库（30k 实测 1642ms）；超预算只发生在无区分度查询，
+        # 平分数下截断不改 top-K。排序：精确子串命中（LIKE，最强
+        # fuzzy 信号）排最前，短文本兜底路按 rowid 截断。
         _FUZZY_CAND_CAP = 2000
         cand_query += (
             " ORDER BY (CASE WHEN content LIKE ? THEN 0 ELSE 1 END), rowid"

{mindforge_memory-6.4.8 → mindforge_memory-6.4.9}/mindforge/core/version.py

@@ -20,10 +20,10 @@ core/mindforge.py、cli/main.py、mcp/server.py），每处都带一份硬编码
 from typing import Tuple
 
 #: 当前发布版本（唯一真值）
-__version__: str = "6.4.8"
+__version__: str = "6.4.9"
 
 #: 结构化版本元组，便于比较（如 `VERSION_INFO >= (6, 1, 0)`）
-VERSION_INFO: Tuple[int, int, int] = (6, 4, 8)
+VERSION_INFO: Tuple[int, int, int] = (6, 4, 9)
 
 #: 下一个大版本的目标版本号（供 v6 端口预留引用）
 NEXT_MAJOR_TARGET: str = "7.0.0"

{mindforge_memory-6.4.8 → mindforge_memory-6.4.9/mindforge_memory.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mindforge-memory
-Version: 6.4.8
+Version: 6.4.9
 Summary: AI Agent 终身记忆系统 - 四层记忆架构 · 知识图谱 · 多模态支持
 License: MIT
 Project-URL: Homepage, https://github.com/opok-ops/MindForge
@@ -36,11 +36,11 @@ Dynamic: license-file
 
 MindForge gives AI agents a persistent, structured long-term memory: a four-tier lifecycle model, knowledge graphs, hybrid retrieval, end-to-end encryption, and MCP / CLI / REST interfaces. All data stays on your machine.
 
-> **The story** — one developer, seven months, from v1.0 (March 2026) to v6.4.8: no team, no funding, 1,252 passing tests. [Read the journey →](https://opok-ops.github.io/MindForge/#developer)
+> **The story** — one developer, seven months, from v1.0 (March 2026) to v6.4.9: no team, no funding, 1,257 passing tests. [Read the journey →](https://opok-ops.github.io/MindForge/#developer)
 
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
-[![Version](https://img.shields.io/badge/version-6.4.8-green.svg)](https://github.com/opok-ops/MindForge/releases)
+[![Version](https://img.shields.io/badge/version-6.4.9-green.svg)](https://github.com/opok-ops/MindForge/releases)
 [![Stars](https://img.shields.io/github/stars/opok-ops/MindForge.svg)](https://github.com/opok-ops/MindForge/stargazers)
 [![Forks](https://img.shields.io/github/forks/opok-ops/MindForge.svg)](https://github.com/opok-ops/MindForge/forks)
 [![Open Issues](https://img.shields.io/github/issues/opok-ops/MindForge.svg)](https://github.com/opok-ops/MindForge/issues)
@@ -95,15 +95,15 @@ MindForge stats
 ```
 
 
-## Scale & Quality Benchmarks (v6.4.8, measured)
+## Scale & Quality Benchmarks (v6.4.9, measured)
 
 | Scale | add throughput | search p50 (rare token) | search p50 (realistic mixed) | search p50 (pathological all-common) | db size |
 |---|---|---|---|---|---|
 | 200 entries (baseline) | — | 1.8 ms | — | — | <1 MB |
 | 10,000 | 1,080/s | ~9.5 ms | ~16 ms | — | 10 MB |
-| 100,000 | ~582/s | 6.8 ms | 135 ms | 122 ms (−44%) | 98 MB |
+| 100,000 (~10M chars) | ~546/s | 6.8 ms | 3.0 ms | 0.4 ms | 98 MB |
 
-- Pathological queries (all-common words, synthetic template data): engine-level steady-state **4.0ms at 30k** (was 3825ms, −99.9%) — inverted-index fallback is now shortest-chain-first with a hard traversal budget; the remaining 100k cost is FTS5 high-frequency-phrase scoring (v6.5 target, honestly documented).
+- **Ten-million-character library, millisecond-class across all query types (v6.4.9)**: pathological 218→**0.4ms** (−99.8%), mixed 143→**3.0ms** (−98%), rare 6.8ms, combined 17.8ms (100k full-pipeline, clean environment). The last bottleneck — FTS5 high-frequency-phrase bm25 scoring — is now pruned query-side via trigram-df pre-check (df upper bound = min trigram df, pruned at 5% corpus size, symmetric to the TF-IDF max_df design; df served from an in-process cache — fts5vocab is a streaming virtual table, per-term probes scan it fully).
 - **Index-blindness fix (v6.4.8)**: TF-IDF vocabulary was locked to the first 5 documents since v6.4.6 — incremental adds indexed as empty vectors (TF-IDF effectively half-dead; previous numbers carried by FTS5/fuzzy). Fixed with incremental vocabulary growth; all pipeline numbers above are now *true full-pipeline* measurements.
 - Retrieval quality (fixed-seed, `benchmarks/quality_bench.py`, 2k entries): needle Recall@1 = 1.0, MRR = 1.0, topic Recall@1 = 1.0, noise false-recall = 0.
 - Full LLM-judged benchmarks (LoCoMo / LongMemEval) — interface ready, not yet run (requires external LLM API key).
@@ -111,7 +111,9 @@ MindForge stats
 
 ## What's New
 
-v6.1.3 – v6.4.8 shipped in ten days (2026-11-01 → 11-11; three releases in the last two days), guarded by **1,253 tests** (1,252 passing, 1 environment-skipped).
+v6.1.3 – v6.4.9 shipped in ten days (2026-11-01 → 11-11; four releases in the last two days), guarded by **1,258 tests** (1,257 passing, 1 environment-skipped).
+
+- **v6.4.9 — ten-million-char library at millisecond latency: FTS5 high-frequency-phrase scoring pruned** (2026-11-11). The last honest bottleneck from v6.4.8's ledger. Root cause (30k per-path timing): under the trigram tokenizer, corpus-wide phrases (template data's "陈述性记忆") match every row — bm25 full-match scoring cost 50.5ms/query, the dominant remainder of 100k mixed latency. Fix, query-side df pre-pruning symmetric to TF-IDF max_df: a phrase's df upper bound = min df of its trigrams (phrase matches ⊆ every trigram's row set); prune at 5% of corpus size; keep the lowest-ub phrase if all pruned. df is served from an in-process cache keyed by `max(rowid)` (add/update invalidate; hard-delete leaves df slightly stale = prunes more conservatively) — **two real pitfalls found en route: fts5vocab is a *streaming virtual table* — per-term probes AND `IN` batches trigger full scans (combined regressed 17→45ms before the cache)**. Plus: FTS5 path skipped on pathological fallback (no discriminating signal). Clean-environment 100k re-bench: pathological 122→**0.4ms** (−99.7%), mixed 134.9→**3.0ms** (−98%), all query types millisecond-class at ~10M chars. AI-flavored comment cleanup (essay-style docstrings → engineer-speak, info preserved). 5 new guards.
 
 - **v6.4.8 — pathological query path & index-blindness fix: O(N) fallback eliminated, TF-IDF actually indexing** (2026-11-11). Two deeper findings beyond the v6.4.6 stress test: (1) the all-common-word fallback traversed *all* inverted chains at full length (O(N×dims), 209ms p50 at 100k), and the 0.5×N max_df threshold still let half-corpus chains into the active set; (2) **the TF-IDF vocabulary had been locked to the first 5 documents** — every incremental add after that indexed as an empty vector (the TF-IDF path was effectively half-dead since v6.4.6; published numbers were carried by FTS5/fuzzy). Fix, six cuts: max_df tightened 0.5×N→**0.05×N**; fallback semantics → **shortest-chain-first** (≤3 dims, per-chain cap 5,000, islice); fuzzy candidate budget (SQL cap 2,000 with exact-substring-first ordering); fuzzy gating (runs only on zero TF-IDF hits — the true typo-correction scenario; encrypted mode unaffected); `incremental_fit()` vocabulary growth for adds after fit (old vectors not recomputed — new tokens contribute zero on old docs mathematically); empty-query fallback-signal reset. 30k repro: pathological steady-state 3825→**4.0ms** (−99.9%), mixed pipeline 1607→1.2ms, fuzzy single-path −93%. 100k re-bench: pathological −44%, all numbers now true full-pipeline. Remaining bottleneck honestly documented: FTS5 high-frequency-phrase scoring (v6.5). New observable `_last_traversed` + deterministic budget contracts. 9 new guards.
 
@@ -186,7 +188,7 @@ v6.1.3 – v6.4.8 shipped in ten days (2026-11-01 → 11-11; three releases in t
 
 ```
 ┌──────────────────────────────────────────────────────┐
-│           MindForge v6.4.8            │
+│           MindForge v6.4.9            │
 │  Cognitive Layer  Personality · KnowledgeGraph       │
 │                   MemoryEvolution · FederatedMemory  │
 ├──────────────────────────────────────────────────────┤

{mindforge_memory-6.4.8 → mindforge_memory-6.4.9}/mindforge_memory.egg-info/SOURCES.txt

@@ -136,4 +136,5 @@ tests/test_v643_fixes.py
 tests/test_v645_fixes.py
 tests/test_v646_fixes.py
 tests/test_v648_fixes.py
+tests/test_v649_fixes.py
 tests/test_v6_ports.py

{mindforge_memory-6.4.8 → mindforge_memory-6.4.9}/tests/test_v568_fixes.py

@@ -39,7 +39,7 @@ import mindforge.core.version as core_version
 
 
 _REPO = Path(__file__).resolve().parent.parent
-_EXPECTED_VERSION = "6.4.8"
+_EXPECTED_VERSION = "6.4.9"
 
 # 本轮修复涉及的、必须与 mindforge/core/version.py 保持一致的源码文件
 _VERSION_TRUTH_FILES = (

{mindforge_memory-6.4.8 → mindforge_memory-6.4.9}/tests/test_v575_release_consistency.py

@@ -40,8 +40,8 @@ class TestVersionTruth(unittest.TestCase):
     def test_core_version_matches_release(self):
         # 版本推进时必须一并更新此处的钉值（与 test_v568_fixes 的
         # _EXPECTED_VERSION 同一约定：让「忘记 bump 版本」在测试里显形）。
-        self.assertEqual(core_version.__version__, "6.4.8")
-        self.assertEqual(core_version.VERSION_INFO, (6, 4, 8))
+        self.assertEqual(core_version.__version__, "6.4.9")
+        self.assertEqual(core_version.VERSION_INFO, (6, 4, 9))
 
     def test_pyproject_matches_core_version(self):
         # v5.8.13 起 pyproject 不再写死静态版本：

{mindforge_memory-6.4.8 → mindforge_memory-6.4.9}/tests/test_v612_fixes.py

@@ -662,8 +662,8 @@ class TestVersionConsistency(unittest.TestCase):
     def test_version_is_612(self):
         from mindforge.core.version import VERSION_INFO, __version__
         # v6.1.3：版本钉值随发版推进更新（历史方法名保留）
-        self.assertEqual(__version__, "6.4.8")
-        self.assertEqual(VERSION_INFO, (6, 4, 8))
+        self.assertEqual(__version__, "6.4.9")
+        self.assertEqual(VERSION_INFO, (6, 4, 9))
 
     def test_json_round_trip_safety_of_preview(self):
         """save_evolution 的预览序列化不得抛（不可 json 化的对象走 str 兜底）。"""

{mindforge_memory-6.4.8 → mindforge_memory-6.4.9}/tests/test_v643_fixes.py

@@ -94,7 +94,7 @@ class TestVersionBump(unittest.TestCase):
     钉点集合随版本数递增，发版链清单必须扫全部 test_v*fixes.py。"""
 
     def test_version_is_current(self):
-        self.assertEqual(__version__, "6.4.8")
+        self.assertEqual(__version__, "6.4.9")
 
 
 if __name__ == "__main__":

{mindforge_memory-6.4.8 → mindforge_memory-6.4.9}/tests/test_v645_fixes.py

@@ -101,7 +101,7 @@ class TestVersionBump(unittest.TestCase):
     """版本钉值：忘记 bump 时在此显形。"""
 
     def test_version_is_current(self):
-        self.assertEqual(__version__, "6.4.8")
+        self.assertEqual(__version__, "6.4.9")
 
 
 if __name__ == "__main__":

{mindforge_memory-6.4.8 → mindforge_memory-6.4.9}/tests/test_v646_fixes.py

@@ -85,7 +85,7 @@ class TestVersionBump(unittest.TestCase):
     """版本钉值（v6.4.5 教训：每版 fixes 断言随发版一并 bump）。"""
 
     def test_version_is_646(self):
-        self.assertEqual(__version__, "6.4.8")
+        self.assertEqual(__version__, "6.4.9")
 
 
 if __name__ == "__main__":

{mindforge_memory-6.4.8 → mindforge_memory-6.4.9}/tests/test_v648_fixes.py

@@ -22,7 +22,7 @@ import unittest
 
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from mindforge.core.version import __version__
+from mindforge.core.version import __version__, VERSION_INFO
 from mindforge.core.indexer import VectorIndex
 
 
@@ -181,8 +181,10 @@ class TestFuzzyGateAndFallbackSignal(unittest.TestCase):
 class TestVersionBump(unittest.TestCase):
     """版本钉值（v6.4.5 教训：每版 fixes 断言随发版一并 bump）。"""
 
-    def test_version_is_648(self):
-        self.assertEqual(__version__, "6.4.8")
+    def test_version_is_648_plus(self):
+        # v6.4.9 发版时 bump：本文件守卫 v6.4.8 引入的行为，
+        # 版本断言随最新发版跟进
+        self.assertGreaterEqual(VERSION_INFO, (6, 4, 8))
 
 
 if __name__ == "__main__":