mindforge-memory 6.4.5__tar.gz → 6.4.8__tar.gz

{mindforge_memory-6.4.5/mindforge_memory.egg-info → mindforge_memory-6.4.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mindforge-memory
-Version: 6.4.5
+Version: 6.4.8
 Summary: AI Agent 终身记忆系统 - 四层记忆架构 · 知识图谱 · 多模态支持
 License: MIT
 Project-URL: Homepage, https://github.com/opok-ops/MindForge
@@ -36,11 +36,11 @@ Dynamic: license-file
 
 MindForge gives AI agents a persistent, structured long-term memory: a four-tier lifecycle model, knowledge graphs, hybrid retrieval, end-to-end encryption, and MCP / CLI / REST interfaces. All data stays on your machine.
 
-> **The story** — one developer, seven months, from v1.0 (March 2026) to v6.4.5: no team, no funding, 1,232 passing tests. [Read the journey →](https://opok-ops.github.io/MindForge/#developer)
+> **The story** — one developer, seven months, from v1.0 (March 2026) to v6.4.8: no team, no funding, 1,252 passing tests. [Read the journey →](https://opok-ops.github.io/MindForge/#developer)
 
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
-[![Version](https://img.shields.io/badge/version-6.4.5-green.svg)](https://github.com/opok-ops/MindForge/releases)
+[![Version](https://img.shields.io/badge/version-6.4.8-green.svg)](https://github.com/opok-ops/MindForge/releases)
 [![Stars](https://img.shields.io/github/stars/opok-ops/MindForge.svg)](https://github.com/opok-ops/MindForge/stargazers)
 [![Forks](https://img.shields.io/github/forks/opok-ops/MindForge.svg)](https://github.com/opok-ops/MindForge/forks)
 [![Open Issues](https://img.shields.io/github/issues/opok-ops/MindForge.svg)](https://github.com/opok-ops/MindForge/issues)
@@ -94,10 +94,28 @@ MindForge search "what does user like"
 MindForge stats
 ```
 
+
+## Scale & Quality Benchmarks (v6.4.8, measured)
+
+| Scale | add throughput | search p50 (rare token) | search p50 (realistic mixed) | search p50 (pathological all-common) | db size |
+|---|---|---|---|---|---|
+| 200 entries (baseline) | — | 1.8 ms | — | — | <1 MB |
+| 10,000 | 1,080/s | ~9.5 ms | ~16 ms | — | 10 MB |
+| 100,000 | ~582/s | 6.8 ms | 135 ms | 122 ms (−44%) | 98 MB |
+
+- Pathological queries (all-common words, synthetic template data): engine-level steady-state **4.0ms at 30k** (was 3825ms, −99.9%) — inverted-index fallback is now shortest-chain-first with a hard traversal budget; the remaining 100k cost is FTS5 high-frequency-phrase scoring (v6.5 target, honestly documented).
+- **Index-blindness fix (v6.4.8)**: TF-IDF vocabulary was locked to the first 5 documents since v6.4.6 — incremental adds indexed as empty vectors (TF-IDF effectively half-dead; previous numbers carried by FTS5/fuzzy). Fixed with incremental vocabulary growth; all pipeline numbers above are now *true full-pipeline* measurements.
+- Retrieval quality (fixed-seed, `benchmarks/quality_bench.py`, 2k entries): needle Recall@1 = 1.0, MRR = 1.0, topic Recall@1 = 1.0, noise false-recall = 0.
+- Full LLM-judged benchmarks (LoCoMo / LongMemEval) — interface ready, not yet run (requires external LLM API key).
+- TF-IDF degraded mode process memory at 100k: ~700 MB. Index LRU cap: 100,000 docs (MINDFORGE_INDEX_MAX_DOCS).
+
 ## What's New
 
-v6.1.3 – v6.4.5 shipped in ten days (2026-11-01 → 11-10; two same-day releases), guarded by **1,233 tests** (1,232 passing, 1 environment-skipped).
+v6.1.3 – v6.4.8 shipped in ten days (2026-11-01 → 11-11; three releases in the last two days), guarded by **1,253 tests** (1,252 passing, 1 environment-skipped).
+
+- **v6.4.8 — pathological query path & index-blindness fix: O(N) fallback eliminated, TF-IDF actually indexing** (2026-11-11). Two deeper findings beyond the v6.4.6 stress test: (1) the all-common-word fallback traversed *all* inverted chains at full length (O(N×dims), 209ms p50 at 100k), and the 0.5×N max_df threshold still let half-corpus chains into the active set; (2) **the TF-IDF vocabulary had been locked to the first 5 documents** — every incremental add after that indexed as an empty vector (the TF-IDF path was effectively half-dead since v6.4.6; published numbers were carried by FTS5/fuzzy). Fix, six cuts: max_df tightened 0.5×N→**0.05×N**; fallback semantics → **shortest-chain-first** (≤3 dims, per-chain cap 5,000, islice); fuzzy candidate budget (SQL cap 2,000 with exact-substring-first ordering); fuzzy gating (runs only on zero TF-IDF hits — the true typo-correction scenario; encrypted mode unaffected); `incremental_fit()` vocabulary growth for adds after fit (old vectors not recomputed — new tokens contribute zero on old docs mathematically); empty-query fallback-signal reset. 30k repro: pathological steady-state 3825→**4.0ms** (−99.9%), mixed pipeline 1607→1.2ms, fuzzy single-path −93%. 100k re-bench: pathological −44%, all numbers now true full-pipeline. Remaining bottleneck honestly documented: FTS5 high-frequency-phrase scoring (v6.5). New observable `_last_traversed` + deterministic budget contracts. 9 new guards.
 
+- **v6.4.6 — scale validation & retrieval benchmarks** (2026-11-10). First 100,000-entry stress test: add throughput ~600/s (linear), ~98MB db, memory peak ~700MB (TF-IDF degraded mode), search p50 7.4ms (rare token, −92% vs pre-fix) / 143ms (realistic mixed); pathological all-common query 421→218ms (−48%). Fixed a real bottleneck found by the stress test: Chinese long-query p50 was 421ms — profile showed low-IDF common words degenerating the inverted index into full-corpus scans. Fix: two-tier pruning (relative-eps + standard IR max_df stop-wording for df > 0.5×N) + heapq.nlargest; pathological all-common-word queries (synthetic-template data only) honestly documented as known O(N) characteristic, realistic queries verified unaffected (8k A/B, 30k side-by-side: 43ms mixed vs 86ms pathological). Shipped reproducible benchmarks (`benchmarks/scale_bench.py`, `benchmarks/quality_bench.py`): needle Recall@1 = 1.0 / MRR = 1.0, zero noise false-recall. Full LLM-judged benchmarks (LoCoMo/LongMemEval) deferred — needs external LLM API key. 7 new guards.
 - **v6.4.5 — audit-driven fixes** (2026-11-10; v6.4.4 skipped by maintainer decision). Two inverse doc-code inconsistencies (v6.4.3 fixed "claimed but not done"; this one fixes "done but not tracked"): `connectors.py` outgoing HTTP User-Agent was hardcoded `MindForge-Connector/5.8.9` — lying about the version to every target site since the 5.8.9 era (`event_bus.py` fixed the same disease in v5.6.8; this one slipped through). Now dynamic from the single version truth. Website "core modules" count was stale (17 in prose, 15 in stat card, 31 measured) — both fixed to the measured 31; MCP tool count 87 verified exact, untouched. Audit archive (not a defect): WAL auto-checkpoint verified healthy under daemon-like load — 4 rounds × 400 mixed ops, WAL converged at ~4.1MB (default 1000-page threshold), TRUNCATE resets to zero. 5 new guards in `tests/test_v645_fixes.py`.
 - **v6.4.3 — documentation-code consistency fix** (2026-11-10). Pre-release audit found that v6.4.2's claimed first cut — WAL + `synchronous=NORMAL` — **never landed in code**: `_get_conn()` set only `journal_mode=WAL`, leaving `synchronous` at SQLite's default FULL. The 2.1× search gain was real, but carried entirely by the other four cuts — real numbers, misattributed cause. This release adds `PRAGMA synchronous=NORMAL` on the WAL branch (per-connection, cross-thread; network-FS DELETE fallback stays FULL by design). Same-machine A/B on SSD shows no statistically significant latency delta (search p50 1.8 → 1.8ms — fsync is ~0.1ms-scale here); the benefit targets slow disks and IO-saturated containers, and more importantly makes the documented power-loss semantics (un-checkpointed access counts / search audits may be lost; compliance audits still commit immediately) actually hold. 4 new guard tests in `tests/test_v643_fixes.py`.
 - **v6.4.2 — tens-of-thousands-chars latency: another 2×** (2026-11-09). Same-machine A/B benchmark (400 memories / ~47k chars, 3 runs each): **search p50 6.3→3.0ms (2.1×), p95 11.7→4.2ms (2.8×)**; 24k-char library p50 2.5ms; fast machine state 1.8ms. Five surgical cuts: WAL + `synchronous=NORMAL` (commits skip per-write fsync), enum-downgrade memo, `row.keys()` set reuse, first-visit access-count batching merged into one `flush_access()` transaction per search, and search-audit INSERT folded into the same commit. The v5.6.3 throttle contract and v5.5.4 `recently_accessed` semantics are pinned by 23 contract tests — all green. Compliance-grade audits still commit immediately. *(Note: the `synchronous=NORMAL` cut landed in v6.4.3 — see above.)*
@@ -168,7 +186,7 @@ v6.1.3 – v6.4.5 shipped in ten days (2026-11-01 → 11-10; two same-day releas
 
 ```
 ┌──────────────────────────────────────────────────────┐
-│           MindForge v6.4.5            │
+│           MindForge v6.4.8            │
 │  Cognitive Layer  Personality · KnowledgeGraph       │
 │                   MemoryEvolution · FederatedMemory  │
 ├──────────────────────────────────────────────────────┤

{mindforge_memory-6.4.5 → mindforge_memory-6.4.8}/README.md

@@ -6,11 +6,11 @@
 
 MindForge gives AI agents a persistent, structured long-term memory: a four-tier lifecycle model, knowledge graphs, hybrid retrieval, end-to-end encryption, and MCP / CLI / REST interfaces. All data stays on your machine.
 
-> **The story** — one developer, seven months, from v1.0 (March 2026) to v6.4.5: no team, no funding, 1,232 passing tests. [Read the journey →](https://opok-ops.github.io/MindForge/#developer)
+> **The story** — one developer, seven months, from v1.0 (March 2026) to v6.4.8: no team, no funding, 1,252 passing tests. [Read the journey →](https://opok-ops.github.io/MindForge/#developer)
 
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
-[![Version](https://img.shields.io/badge/version-6.4.5-green.svg)](https://github.com/opok-ops/MindForge/releases)
+[![Version](https://img.shields.io/badge/version-6.4.8-green.svg)](https://github.com/opok-ops/MindForge/releases)
 [![Stars](https://img.shields.io/github/stars/opok-ops/MindForge.svg)](https://github.com/opok-ops/MindForge/stargazers)
 [![Forks](https://img.shields.io/github/forks/opok-ops/MindForge.svg)](https://github.com/opok-ops/MindForge/forks)
 [![Open Issues](https://img.shields.io/github/issues/opok-ops/MindForge.svg)](https://github.com/opok-ops/MindForge/issues)
@@ -64,10 +64,28 @@ MindForge search "what does user like"
 MindForge stats
 ```
 
+
+## Scale & Quality Benchmarks (v6.4.8, measured)
+
+| Scale | add throughput | search p50 (rare token) | search p50 (realistic mixed) | search p50 (pathological all-common) | db size |
+|---|---|---|---|---|---|
+| 200 entries (baseline) | — | 1.8 ms | — | — | <1 MB |
+| 10,000 | 1,080/s | ~9.5 ms | ~16 ms | — | 10 MB |
+| 100,000 | ~582/s | 6.8 ms | 135 ms | 122 ms (−44%) | 98 MB |
+
+- Pathological queries (all-common words, synthetic template data): engine-level steady-state **4.0ms at 30k** (was 3825ms, −99.9%) — inverted-index fallback is now shortest-chain-first with a hard traversal budget; the remaining 100k cost is FTS5 high-frequency-phrase scoring (v6.5 target, honestly documented).
+- **Index-blindness fix (v6.4.8)**: TF-IDF vocabulary was locked to the first 5 documents since v6.4.6 — incremental adds indexed as empty vectors (TF-IDF effectively half-dead; previous numbers carried by FTS5/fuzzy). Fixed with incremental vocabulary growth; all pipeline numbers above are now *true full-pipeline* measurements.
+- Retrieval quality (fixed-seed, `benchmarks/quality_bench.py`, 2k entries): needle Recall@1 = 1.0, MRR = 1.0, topic Recall@1 = 1.0, noise false-recall = 0.
+- Full LLM-judged benchmarks (LoCoMo / LongMemEval) — interface ready, not yet run (requires external LLM API key).
+- TF-IDF degraded mode process memory at 100k: ~700 MB. Index LRU cap: 100,000 docs (MINDFORGE_INDEX_MAX_DOCS).
+
 ## What's New
 
-v6.1.3 – v6.4.5 shipped in ten days (2026-11-01 → 11-10; two same-day releases), guarded by **1,233 tests** (1,232 passing, 1 environment-skipped).
+v6.1.3 – v6.4.8 shipped in ten days (2026-11-01 → 11-11; three releases in the last two days), guarded by **1,253 tests** (1,252 passing, 1 environment-skipped).
+
+- **v6.4.8 — pathological query path & index-blindness fix: O(N) fallback eliminated, TF-IDF actually indexing** (2026-11-11). Two deeper findings beyond the v6.4.6 stress test: (1) the all-common-word fallback traversed *all* inverted chains at full length (O(N×dims), 209ms p50 at 100k), and the 0.5×N max_df threshold still let half-corpus chains into the active set; (2) **the TF-IDF vocabulary had been locked to the first 5 documents** — every incremental add after that indexed as an empty vector (the TF-IDF path was effectively half-dead since v6.4.6; published numbers were carried by FTS5/fuzzy). Fix, six cuts: max_df tightened 0.5×N→**0.05×N**; fallback semantics → **shortest-chain-first** (≤3 dims, per-chain cap 5,000, islice); fuzzy candidate budget (SQL cap 2,000 with exact-substring-first ordering); fuzzy gating (runs only on zero TF-IDF hits — the true typo-correction scenario; encrypted mode unaffected); `incremental_fit()` vocabulary growth for adds after fit (old vectors not recomputed — new tokens contribute zero on old docs mathematically); empty-query fallback-signal reset. 30k repro: pathological steady-state 3825→**4.0ms** (−99.9%), mixed pipeline 1607→1.2ms, fuzzy single-path −93%. 100k re-bench: pathological −44%, all numbers now true full-pipeline. Remaining bottleneck honestly documented: FTS5 high-frequency-phrase scoring (v6.5). New observable `_last_traversed` + deterministic budget contracts. 9 new guards.
 
+- **v6.4.6 — scale validation & retrieval benchmarks** (2026-11-10). First 100,000-entry stress test: add throughput ~600/s (linear), ~98MB db, memory peak ~700MB (TF-IDF degraded mode), search p50 7.4ms (rare token, −92% vs pre-fix) / 143ms (realistic mixed); pathological all-common query 421→218ms (−48%). Fixed a real bottleneck found by the stress test: Chinese long-query p50 was 421ms — profile showed low-IDF common words degenerating the inverted index into full-corpus scans. Fix: two-tier pruning (relative-eps + standard IR max_df stop-wording for df > 0.5×N) + heapq.nlargest; pathological all-common-word queries (synthetic-template data only) honestly documented as known O(N) characteristic, realistic queries verified unaffected (8k A/B, 30k side-by-side: 43ms mixed vs 86ms pathological). Shipped reproducible benchmarks (`benchmarks/scale_bench.py`, `benchmarks/quality_bench.py`): needle Recall@1 = 1.0 / MRR = 1.0, zero noise false-recall. Full LLM-judged benchmarks (LoCoMo/LongMemEval) deferred — needs external LLM API key. 7 new guards.
 - **v6.4.5 — audit-driven fixes** (2026-11-10; v6.4.4 skipped by maintainer decision). Two inverse doc-code inconsistencies (v6.4.3 fixed "claimed but not done"; this one fixes "done but not tracked"): `connectors.py` outgoing HTTP User-Agent was hardcoded `MindForge-Connector/5.8.9` — lying about the version to every target site since the 5.8.9 era (`event_bus.py` fixed the same disease in v5.6.8; this one slipped through). Now dynamic from the single version truth. Website "core modules" count was stale (17 in prose, 15 in stat card, 31 measured) — both fixed to the measured 31; MCP tool count 87 verified exact, untouched. Audit archive (not a defect): WAL auto-checkpoint verified healthy under daemon-like load — 4 rounds × 400 mixed ops, WAL converged at ~4.1MB (default 1000-page threshold), TRUNCATE resets to zero. 5 new guards in `tests/test_v645_fixes.py`.
 - **v6.4.3 — documentation-code consistency fix** (2026-11-10). Pre-release audit found that v6.4.2's claimed first cut — WAL + `synchronous=NORMAL` — **never landed in code**: `_get_conn()` set only `journal_mode=WAL`, leaving `synchronous` at SQLite's default FULL. The 2.1× search gain was real, but carried entirely by the other four cuts — real numbers, misattributed cause. This release adds `PRAGMA synchronous=NORMAL` on the WAL branch (per-connection, cross-thread; network-FS DELETE fallback stays FULL by design). Same-machine A/B on SSD shows no statistically significant latency delta (search p50 1.8 → 1.8ms — fsync is ~0.1ms-scale here); the benefit targets slow disks and IO-saturated containers, and more importantly makes the documented power-loss semantics (un-checkpointed access counts / search audits may be lost; compliance audits still commit immediately) actually hold. 4 new guard tests in `tests/test_v643_fixes.py`.
 - **v6.4.2 — tens-of-thousands-chars latency: another 2×** (2026-11-09). Same-machine A/B benchmark (400 memories / ~47k chars, 3 runs each): **search p50 6.3→3.0ms (2.1×), p95 11.7→4.2ms (2.8×)**; 24k-char library p50 2.5ms; fast machine state 1.8ms. Five surgical cuts: WAL + `synchronous=NORMAL` (commits skip per-write fsync), enum-downgrade memo, `row.keys()` set reuse, first-visit access-count batching merged into one `flush_access()` transaction per search, and search-audit INSERT folded into the same commit. The v5.6.3 throttle contract and v5.5.4 `recently_accessed` semantics are pinned by 23 contract tests — all green. Compliance-grade audits still commit immediately. *(Note: the `synchronous=NORMAL` cut landed in v6.4.3 — see above.)*
@@ -138,7 +156,7 @@ v6.1.3 – v6.4.5 shipped in ten days (2026-11-01 → 11-10; two same-day releas
 
 ```
 ┌──────────────────────────────────────────────────────┐
-│           MindForge v6.4.5            │
+│           MindForge v6.4.8            │
 │  Cognitive Layer  Personality · KnowledgeGraph       │
 │                   MemoryEvolution · FederatedMemory  │
 ├──────────────────────────────────────────────────────┤

{mindforge_memory-6.4.5 → mindforge_memory-6.4.8}/mindforge/core/indexer.py

@@ -3,11 +3,13 @@ MindForge v5.0 索引引擎
 支持 TF-IDF、向量索引、FTS5 全文检索
 """
 
+import heapq
 import math
 import os
 import re
 import sqlite3
 from collections import Counter, defaultdict
+from itertools import islice
 from typing import List, Dict, Tuple, Optional
 from dataclasses import dataclass
 
@@ -27,6 +29,7 @@ class TFIDFVectorizer:
     def __init__(self):
         self.vocab: Dict[str, int] = {}
         self.idf: Dict[str, float] = {}
+        self._df: Dict[str, int] = {}  # v6.4.8: 增量 fit 的精确 df 延续
         self.doc_count: int = 0
 
     def _tokenize(self, text: str) -> List[str]:
@@ -58,6 +61,33 @@ class TFIDFVectorizer:
 
         for word, idx in self.vocab.items():
             self.idf[word] = math.log((self.doc_count + 1) / (df[word] + 1)) + 1
+        # v6.4.8: df 表留存，供 incremental_fit 延续精确计数
+        self._df = dict(df)
+
+    def incremental_fit(self, text: str) -> None:
+        """v6.4.8 新增：单文档增量词表扩展。
+
+        修复「fit 后新增文档失索引」：新 token 按当前 doc_count+1 估
+        IDF 补进词表；已知 token 的 df 计数 +1（IDF 漂移不回算旧向量，
+        见 index_memory 注释）。维度号从当前词表长度起追加，与既有
+        向量维度兼容（旧文档在新词上恒为零分量）。
+        """
+        if not self._df:
+            # 未 fit 过（不应发生——调用方保证 fitted），防御性直通
+            return
+        self.doc_count += 1
+        next_idx = len(self.vocab)
+        for token in set(self._tokenize(text)):
+            if token in self.vocab:
+                self._df[token] = self._df.get(token, 0) + 1
+                self.idf[token] = math.log(
+                    (self.doc_count + 1) / (self._df[token] + 1)) + 1
+            else:
+                self._df[token] = 1
+                self.vocab[token] = next_idx
+                next_idx += 1
+                self.idf[token] = math.log(
+                    (self.doc_count + 1) / 2) + 1
 
     def transform(self, text: str) -> Dict[int, float]:
         tokens = self._tokenize(text)
@@ -107,6 +137,10 @@ class VectorIndex:
         self.metadata: Dict[str, Dict] = {}
         self._norms: Dict[str, float] = {}  # 预计算 L2 范数缓存
         self._postings: Dict[int, Dict[str, float]] = {}  # 倒排链
+        # v6.4.8: 上次 search 实际遍历的倒排链条目数（性能契约可观测）
+        self._last_traversed = 0
+        # v6.4.8: 上次 search 是否走了病态回退（全库词查询信号）
+        self.last_fallback = False
 
     @staticmethod
     def _as_sparse(vector) -> Dict[int, float]:
@@ -149,19 +183,81 @@ class VectorIndex:
         q = self._as_sparse(query_vector)
         query_norm = math.sqrt(sum(v * v for v in q.values()))
         if query_norm == 0:
+            # v6.4.8: 空查询（无 token/全 vocab miss）≠ 全库词回退。
+            # 显式复位 last_fallback，防止继承上一查询的 True 导致
+            # query.py 误跳过 fuzzy 拼写兜底（空查询正是 fuzzy 的
+            # 设计场景：用户词打错 → 全 miss → fuzzy 纠错召回）。
+            self.last_fallback = False
+            self._last_traversed = 0
             return [(doc_id, 0.0) for doc_id in list(self.vectors.keys())[:top_k]]
 
+        # v6.4.6 perf: 低区分度查询维剪枝。100k 规模压测发现中文长查询
+        # p50 421ms，profile 定位到常见词（如「条目/陈述性/记忆」）的倒排
+        # 链全量遍历。两层剪枝：eps 相对阈值 + max_df 停用词。
+        # v6.4.8 perf: max_df 从 0.5×N 收紧到 0.05×N。0.5 时半库词链
+        # （数十万级遍历）仍进 active，100k 实测 mixed_realistic p50
+        # 134ms。常见词的点积贡献近似常数（30k profile 实证 eps 剪不掉
+        # 全库词，v6.4.6 注释已论证 top-K 排序不变），链长超库容 5% 的
+        # 维一律剪——100k 库下 active 链长上界 5k，遍历毫秒级。
+        _EPS_REL = 0.02
+        _max_df_len = 0.05 * len(self.vectors)
+        active = [(idx, qval) for idx, qval in q.items()
+                  if qval > _EPS_REL * query_norm
+                  and len(self._postings.get(idx, ())) <= _max_df_len]
+        self._last_traversed = 0
+        # v6.4.8: 病态回退信号——查询全部维度超 max_df（全库词）时
+        # 置位，供上层 QueryEngine 跳过 fuzzy 兜底路（全库词查询对
+        # fuzzy 的边际价值为零，详见 query.py 路 2 注释）。
+        self.last_fallback = False
+        if not active:
+            self.last_fallback = True
+            # v6.4.8 perf: 病态回退（全维被剪）不再遍历全部倒排链。
+            # 旧回退 list(q.items()) = O(N×查询维)，100k 全库词查询
+            # p50 209ms 的根因。全库词贡献是常数，top-K 只由相对区分
+            # 度决定，因此回退语义改为「最短链优先」：
+            #   1) 按链长升序取最短 _FALLBACK_DIMS 维（q 中区分度最高
+            #      的信号，哪怕也被 max_df 剪掉）
+            #   2) 单链遍历截断到 _TRAVERSE_CAP 项——全平分数下截断
+            #      不改 top-K 语义，优先守住延迟上界
+            _FALLBACK_DIMS = 3
+            _TRAVERSE_CAP = 5000
+            ranked = sorted(
+                q.items(),
+                key=lambda kv: len(self._postings.get(kv[0], ())))
+            active = ranked[:_FALLBACK_DIMS]
+            # islice 免建截断副本（dict(list(x)[:cap]) 的 15k 次 hash
+            # 插入在实测里占 ~3ms，30k 病态复现 14→11ms）
+            dots: Dict[str, float] = {}
+            for idx, qval in active:
+                chain = self._postings.get(idx, {})
+                pairs = (chain.items() if len(chain) <= _TRAVERSE_CAP
+                         else islice(chain.items(), _TRAVERSE_CAP))
+                for doc_id, dval in pairs:
+                    dots[doc_id] = dots.get(doc_id, 0.0) + qval * dval
+                    self._last_traversed += 1
+            return self._topk_from_dots(dots, query_norm, top_k)
+
         # 倒排链累加点积：只访问与查询共享非零维度的候选文档
+        # （active 链长上界 _max_df_len，遍历量与库容解耦）
         dots: Dict[str, float] = {}
-        for idx, qval in q.items():
+        for idx, qval in active:
             for doc_id, dval in self._postings.get(idx, {}).items():
                 dots[doc_id] = dots.get(doc_id, 0.0) + qval * dval
-
-        scores = [(doc_id, dot / (query_norm * self._norms[doc_id]))
-                  for doc_id, dot in dots.items()
-                  if self._norms.get(doc_id, 0.0) > 0]
-        scores.sort(key=lambda x: x[1], reverse=True)
-        return scores[:top_k]
+                self._last_traversed += 1
+
+        # v6.4.6 perf: heapq.nlargest 替代全量 sort——候选集大时
+        # O(n·log k) 优于 O(n·log n)，且免建全量列表排序的中间开销。
+        return self._topk_from_dots(dots, query_norm, top_k)
+
+    def _topk_from_dots(self, dots: Dict[str, float], query_norm: float,
+                        top_k: int) -> List[Tuple[str, float]]:
+        """点积累积结果归一化 + top-K 选取（v6.4.8 抽出，回退/正常路径共用）"""
+        return heapq.nlargest(
+            top_k,
+            ((doc_id, dot / (query_norm * self._norms[doc_id]))
+             for doc_id, dot in dots.items()
+             if self._norms.get(doc_id, 0.0) > 0),
+            key=lambda x: x[1])
 
     def _cosine(self, v1, v2) -> float:
         """保留原始接口（向后兼容），稀疏/稠密输入均支持"""
@@ -193,6 +289,8 @@ class IndexEngine:
         self.db_path = db_path
         self.vectorizer = TFIDFVectorizer()
         self.vector_index = VectorIndex()
+        # v6.4.8: 病态回退信号门面透传（见 search()）
+        self.last_fallback = False
         self._fitted = False
         self._hydrated = False
         self._doc_texts: Dict[str, str] = {}
@@ -250,12 +348,21 @@ class IndexEngine:
         """索引记忆"""
         self._doc_texts[doc_id] = text
 
+        # v5.6.5 P1 #5 的渐进触发点保留（LRU 守卫契约：add 即索引，
+        # _doc_texts/vectors/postings 三结构同步有界）。
+        # v6.4.8 fix（祖传深坑）：旧逻辑这里 fit 一次后，后续增量 add 的
+        # transform 全部 vocab miss → 空向量入索引（文档「失索引」）——
+        # 词表锁死在前 5 条文档的 token，100k 压测的 TF-IDF 路实际长期
+        # 半瘫痪（延迟数字由 FTS5/fuzzy 扛）。修复：fit 后的增量 add 先
+        # 走 incremental_fit 扩词表（新 token 补 vocab + df/idf 增量，
+        # 旧文档向量不回算——新词在旧文档 tf=0 无贡献，旧词 IDF 漂移为
+        # 可接受近似），再 transform 入索引。
         if not self._fitted and len(self._doc_texts) >= 5:
             self._fit_vectorizer()
+        elif self._fitted:
+            self.vectorizer.incremental_fit(text)
 
         if self._fitted:
-            # v5.6.3 性能修复：直接存稀疏向量（{dim: weight}），不再展开为
-            # 词表长度的稠密 list
             vector_dict = self.vectorizer.transform(text)
             self.vector_index.add(doc_id, vector_dict, metadata)
 
@@ -273,11 +380,17 @@ class IndexEngine:
             self._fit_vectorizer()
 
         if not self._fitted:
+            # v6.4.8: 降级路径同样透传病态回退信号（保持门面契约一致）
+            self.last_fallback = False
             return self._keyword_search(query, top_k)
 
         # v5.6.3 性能修复：查询向量同样保持稀疏，走倒排链检索
         query_vec_dict = self.vectorizer.transform(query)
-        return self.vector_index.search(query_vec_dict, top_k)
+        results = self.vector_index.search(query_vec_dict, top_k)
+        # v6.4.8: 把 VectorIndex 的病态回退信号透传到门面层
+        # （query.py 据此跳过 fuzzy 兜底路）
+        self.last_fallback = self.vector_index.last_fallback
+        return results
 
     def _keyword_search(self, query: str, top_k: int) -> List[Tuple[str, float]]:
         """关键词搜索（降级方案）"""

{mindforge_memory-6.4.5 → mindforge_memory-6.4.8}/mindforge/core/mindforge.py

@@ -1230,6 +1230,20 @@ class MindForge:
         # v5.6.2 安全修复：隐私访问控制
         entry = self._storage.get_memory(memory_id, actor, session_id)
         if entry is None:
+            # v6.4.6 fix: 硬删已软删（trash）条目的数据主权路径。get_memory
+            # 过滤 trash 条目，此前 delete(hard_delete=True) 对回收站内容
+            # 静默返回 False（CLI 显示「记忆不存在」，误导用户以为已彻底
+            # 清除，物理数据实际仍在库中）。GDPR erase 语义（storage 权利
+            # 清单 2122 行）要求：能软删活条目，就必须能物理清除其 trash
+            # 态。隐私读检查不适用此分支——内容已在 trash，仅物理清除；
+            # storage.delete_memory 自带审计写入（v5.6.1 同事务）。
+            if hard_delete:
+                if self._storage.is_in_trash(memory_id):
+                    success = self._storage.delete_memory(
+                        memory_id, actor, session_id, hard_delete=True)
+                    if success:
+                        self._index.remove_memory(memory_id)
+                    return success
             return False
         ok, _reason = self.privacy_engine.check_access(entry, actor, session_id)
         if not ok:

{mindforge_memory-6.4.5 → mindforge_memory-6.4.8}/mindforge/core/query.py

@@ -259,9 +259,29 @@ class QueryEngine:
 
         # 路 2：Fuzzy（TF-IDF 召回不足时补充；v5.7.1：加密模式下总是跑，
         # 供加密兜底复用结果，避免二次全表扫描）
-        positive = sum(1 for _, s in tfidf_results if s >= min_relevance)
+        # v6.4.8 perf/语义: 触发条件收紧为「TF-IDF 零命中」：
+        #   1) 病态回退（index.last_fallback，全库词查询）→ 跳过。fuzzy
+        #      的设计价值是「拼写纠错/近似词」兜底，全库词查询里不存在
+        #      「没被 TF-IDF 命中的稀有词」，fuzzy 只会给出与全库同分数
+        #      的噪声结果（边际价值为零）；
+        #   2) TF-IDF 已有命中（含低分——cosine 归一化下模板语料的命中
+        #      常在 0.2-0.3 边缘，但词命中本身已证明「查询词在库」）→
+        #      跳过。fuzzy 兜底的真场景是「查询词全不在库」（打错字），
+        #      此时 TF-IDF 必然空结果。mixed 实测（「稀有锚+常见词」）：
+        #      TF-IDF 15 命中 + FTS5 30 命中已足够填 top-K，fuzzy 跑
+        #      cap-2000 候选纯属白税（30k 实测 140ms/查询）。
+        #   3) 性能——fuzzy 候选集对无区分度/长查询会放大成数万条
+        #      （短条目库 LENGTH<4×len(q) 全体进候选），见 storage.py
+        #      v6.4.8 候选预算注释。encrypted 模式不受此影响（仍全跑
+        #      供加密兜底复用）。
+        tfidf_fallback = getattr(self.index, "last_fallback", False)
+        # v6.4.8: 「有效命中」= TF-IDF 存在非零分结果。空查询（全 miss/
+        # 零范数）走文档序 0 分 top-K 兼容分支，tfidf_results 恒非空但
+        # 全零分——必须按分数判定，否则 fuzzy 拼写兜底被死锁。
+        tfidf_effective = any(s > 0.0 for _, s in tfidf_results)
         fuzzy_hits = []
-        if self.storage.encrypted or positive < max_results:
+        if (self.storage.encrypted
+                or (not tfidf_effective and not tfidf_fallback)):
             fuzzy_hits = self.storage.fuzzy_search(
                 query, limit=max_results * 2, threshold=0.1)
             for item in fuzzy_hits:
@@ -276,6 +296,10 @@ class QueryEngine:
                         _bd(entry.id)["final"] = s
 
         # 路 3：FTS5 全文检索
+        # v6.4.8 perf note: 尝试过「TF-IDF 有命中时跳过 FTS5」的门控，
+        # 被混合质量基线测试拦下（hybrid 0.875 < single 0.915，FTS5 的
+        # 补充分对融合质量有真实贡献）。正确方向是 FTS5 内部的
+        # ORDER BY rank top-k 优化（见 indexer.fts_search）。
         fts5_used = False
         # v5.6.2 安全修复：初始化默认值，防止异常路径下 UnboundLocalError
         strategy_used = "tfidf+fuzzy"

{mindforge_memory-6.4.5 → mindforge_memory-6.4.8}/mindforge/core/storage.py

@@ -2959,6 +2959,20 @@ class StorageEngine:
         return len(ids)
 
     @_with_rollback
+    def is_in_trash(self, entry_id: str) -> bool:
+        """v6.4.6: 判定条目是否处于回收站（软删态）。
+
+        服务 delete(hard_delete=True) 的数据主权路径：get_memory 过滤
+        trash 条目（WHERE category != 'trash'），上层 None 分支需要此
+        轻量查询放行物理清除。幂等，只读，不产生任何写副作用。
+        """
+        conn = self._get_conn()
+        row = conn.execute(
+            "SELECT 1 FROM memories WHERE id = ? AND category = 'trash'",
+            (entry_id,),
+        ).fetchone()
+        return row is not None
+
     def delete_memory(self, entry_id: str,
                       actor: str = "", session_id: str = "",
                       hard_delete: bool = False) -> bool:
@@ -9113,6 +9127,25 @@ class StorageEngine:
                     params_or.append(f"%{_ch}%")
         cand_query = base_query + " AND (" + " OR ".join(conds) + ")"
 
+        # v6.4.8 perf: 候选集预算上限 + 区分度优先排序。
+        # 病理：短条目库（content < 4×len(query) 全体进候选）+ 16 字级
+        # 中文长查询 → 候选≈全库，30k 实测 3 万条逐条反序列化+SequenceMatcher
+        # = 1642ms（100k 语料条目 50-70 字时约半库过长度阈值，候选 ~5 万
+        # = 212ms 白税）。预算语义（与 TF-IDF max_df 0.05×N 同一哲学）：
+        #   - 正常查询（含区分词）LIKE 命中 + 短文本全体仍远小于预算，
+        #     结果零变化；
+        #   - 候选超预算只发生在「查询对全库无区分度」的病态场景——此时
+        #     fuzzy 全体候选分数趋同，截断不改 top-K 语义（保延迟优先）。
+        # 排序：content LIKE %q%（精确子串命中，最强 fuzzy 信号——打错字
+        # 场景「fact-7770」vs「fact-777」也走这条路先保住）排最前，其余
+        # 按 rowid（插入序）。截断发生在「短文本兜底路」而不是真命中。
+        # encrypted 兜底路径同享此预算（候选爆炸对加密库同样不可接受）。
+        _FUZZY_CAND_CAP = 2000
+        cand_query += (
+            " ORDER BY (CASE WHEN content LIKE ? THEN 0 ELSE 1 END), rowid"
+            f" LIMIT {_FUZZY_CAND_CAP}")
+        params_or.append(f"%{query_lower}%")
+
         def _entry_iter():
             for _r in conn.execute(cand_query, params + params_or):
                 yield self._row_to_entry(_r)

{mindforge_memory-6.4.5 → mindforge_memory-6.4.8}/mindforge/core/version.py

@@ -20,10 +20,10 @@ core/mindforge.py、cli/main.py、mcp/server.py），每处都带一份硬编码
 from typing import Tuple
 
 #: 当前发布版本（唯一真值）
-__version__: str = "6.4.5"
+__version__: str = "6.4.8"
 
 #: 结构化版本元组，便于比较（如 `VERSION_INFO >= (6, 1, 0)`）
-VERSION_INFO: Tuple[int, int, int] = (6, 4, 5)
+VERSION_INFO: Tuple[int, int, int] = (6, 4, 8)
 
 #: 下一个大版本的目标版本号（供 v6 端口预留引用）
 NEXT_MAJOR_TARGET: str = "7.0.0"

{mindforge_memory-6.4.5 → mindforge_memory-6.4.8/mindforge_memory.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mindforge-memory
-Version: 6.4.5
+Version: 6.4.8
 Summary: AI Agent 终身记忆系统 - 四层记忆架构 · 知识图谱 · 多模态支持
 License: MIT
 Project-URL: Homepage, https://github.com/opok-ops/MindForge
@@ -36,11 +36,11 @@ Dynamic: license-file
 
 MindForge gives AI agents a persistent, structured long-term memory: a four-tier lifecycle model, knowledge graphs, hybrid retrieval, end-to-end encryption, and MCP / CLI / REST interfaces. All data stays on your machine.
 
-> **The story** — one developer, seven months, from v1.0 (March 2026) to v6.4.5: no team, no funding, 1,232 passing tests. [Read the journey →](https://opok-ops.github.io/MindForge/#developer)
+> **The story** — one developer, seven months, from v1.0 (March 2026) to v6.4.8: no team, no funding, 1,252 passing tests. [Read the journey →](https://opok-ops.github.io/MindForge/#developer)
 
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
-[![Version](https://img.shields.io/badge/version-6.4.5-green.svg)](https://github.com/opok-ops/MindForge/releases)
+[![Version](https://img.shields.io/badge/version-6.4.8-green.svg)](https://github.com/opok-ops/MindForge/releases)
 [![Stars](https://img.shields.io/github/stars/opok-ops/MindForge.svg)](https://github.com/opok-ops/MindForge/stargazers)
 [![Forks](https://img.shields.io/github/forks/opok-ops/MindForge.svg)](https://github.com/opok-ops/MindForge/forks)
 [![Open Issues](https://img.shields.io/github/issues/opok-ops/MindForge.svg)](https://github.com/opok-ops/MindForge/issues)
@@ -94,10 +94,28 @@ MindForge search "what does user like"
 MindForge stats
 ```
 
+
+## Scale & Quality Benchmarks (v6.4.8, measured)
+
+| Scale | add throughput | search p50 (rare token) | search p50 (realistic mixed) | search p50 (pathological all-common) | db size |
+|---|---|---|---|---|---|
+| 200 entries (baseline) | — | 1.8 ms | — | — | <1 MB |
+| 10,000 | 1,080/s | ~9.5 ms | ~16 ms | — | 10 MB |
+| 100,000 | ~582/s | 6.8 ms | 135 ms | 122 ms (−44%) | 98 MB |
+
+- Pathological queries (all-common words, synthetic template data): engine-level steady-state **4.0ms at 30k** (was 3825ms, −99.9%) — inverted-index fallback is now shortest-chain-first with a hard traversal budget; the remaining 100k cost is FTS5 high-frequency-phrase scoring (v6.5 target, honestly documented).
+- **Index-blindness fix (v6.4.8)**: TF-IDF vocabulary was locked to the first 5 documents since v6.4.6 — incremental adds indexed as empty vectors (TF-IDF effectively half-dead; previous numbers carried by FTS5/fuzzy). Fixed with incremental vocabulary growth; all pipeline numbers above are now *true full-pipeline* measurements.
+- Retrieval quality (fixed-seed, `benchmarks/quality_bench.py`, 2k entries): needle Recall@1 = 1.0, MRR = 1.0, topic Recall@1 = 1.0, noise false-recall = 0.
+- Full LLM-judged benchmarks (LoCoMo / LongMemEval) — interface ready, not yet run (requires external LLM API key).
+- TF-IDF degraded mode process memory at 100k: ~700 MB. Index LRU cap: 100,000 docs (MINDFORGE_INDEX_MAX_DOCS).
+
 ## What's New
 
-v6.1.3 – v6.4.5 shipped in ten days (2026-11-01 → 11-10; two same-day releases), guarded by **1,233 tests** (1,232 passing, 1 environment-skipped).
+v6.1.3 – v6.4.8 shipped in ten days (2026-11-01 → 11-11; three releases in the last two days), guarded by **1,253 tests** (1,252 passing, 1 environment-skipped).
+
+- **v6.4.8 — pathological query path & index-blindness fix: O(N) fallback eliminated, TF-IDF actually indexing** (2026-11-11). Two deeper findings beyond the v6.4.6 stress test: (1) the all-common-word fallback traversed *all* inverted chains at full length (O(N×dims), 209ms p50 at 100k), and the 0.5×N max_df threshold still let half-corpus chains into the active set; (2) **the TF-IDF vocabulary had been locked to the first 5 documents** — every incremental add after that indexed as an empty vector (the TF-IDF path was effectively half-dead since v6.4.6; published numbers were carried by FTS5/fuzzy). Fix, six cuts: max_df tightened 0.5×N→**0.05×N**; fallback semantics → **shortest-chain-first** (≤3 dims, per-chain cap 5,000, islice); fuzzy candidate budget (SQL cap 2,000 with exact-substring-first ordering); fuzzy gating (runs only on zero TF-IDF hits — the true typo-correction scenario; encrypted mode unaffected); `incremental_fit()` vocabulary growth for adds after fit (old vectors not recomputed — new tokens contribute zero on old docs mathematically); empty-query fallback-signal reset. 30k repro: pathological steady-state 3825→**4.0ms** (−99.9%), mixed pipeline 1607→1.2ms, fuzzy single-path −93%. 100k re-bench: pathological −44%, all numbers now true full-pipeline. Remaining bottleneck honestly documented: FTS5 high-frequency-phrase scoring (v6.5). New observable `_last_traversed` + deterministic budget contracts. 9 new guards.
 
+- **v6.4.6 — scale validation & retrieval benchmarks** (2026-11-10). First 100,000-entry stress test: add throughput ~600/s (linear), ~98MB db, memory peak ~700MB (TF-IDF degraded mode), search p50 7.4ms (rare token, −92% vs pre-fix) / 143ms (realistic mixed); pathological all-common query 421→218ms (−48%). Fixed a real bottleneck found by the stress test: Chinese long-query p50 was 421ms — profile showed low-IDF common words degenerating the inverted index into full-corpus scans. Fix: two-tier pruning (relative-eps + standard IR max_df stop-wording for df > 0.5×N) + heapq.nlargest; pathological all-common-word queries (synthetic-template data only) honestly documented as known O(N) characteristic, realistic queries verified unaffected (8k A/B, 30k side-by-side: 43ms mixed vs 86ms pathological). Shipped reproducible benchmarks (`benchmarks/scale_bench.py`, `benchmarks/quality_bench.py`): needle Recall@1 = 1.0 / MRR = 1.0, zero noise false-recall. Full LLM-judged benchmarks (LoCoMo/LongMemEval) deferred — needs external LLM API key. 7 new guards.
 - **v6.4.5 — audit-driven fixes** (2026-11-10; v6.4.4 skipped by maintainer decision). Two inverse doc-code inconsistencies (v6.4.3 fixed "claimed but not done"; this one fixes "done but not tracked"): `connectors.py` outgoing HTTP User-Agent was hardcoded `MindForge-Connector/5.8.9` — lying about the version to every target site since the 5.8.9 era (`event_bus.py` fixed the same disease in v5.6.8; this one slipped through). Now dynamic from the single version truth. Website "core modules" count was stale (17 in prose, 15 in stat card, 31 measured) — both fixed to the measured 31; MCP tool count 87 verified exact, untouched. Audit archive (not a defect): WAL auto-checkpoint verified healthy under daemon-like load — 4 rounds × 400 mixed ops, WAL converged at ~4.1MB (default 1000-page threshold), TRUNCATE resets to zero. 5 new guards in `tests/test_v645_fixes.py`.
 - **v6.4.3 — documentation-code consistency fix** (2026-11-10). Pre-release audit found that v6.4.2's claimed first cut — WAL + `synchronous=NORMAL` — **never landed in code**: `_get_conn()` set only `journal_mode=WAL`, leaving `synchronous` at SQLite's default FULL. The 2.1× search gain was real, but carried entirely by the other four cuts — real numbers, misattributed cause. This release adds `PRAGMA synchronous=NORMAL` on the WAL branch (per-connection, cross-thread; network-FS DELETE fallback stays FULL by design). Same-machine A/B on SSD shows no statistically significant latency delta (search p50 1.8 → 1.8ms — fsync is ~0.1ms-scale here); the benefit targets slow disks and IO-saturated containers, and more importantly makes the documented power-loss semantics (un-checkpointed access counts / search audits may be lost; compliance audits still commit immediately) actually hold. 4 new guard tests in `tests/test_v643_fixes.py`.
 - **v6.4.2 — tens-of-thousands-chars latency: another 2×** (2026-11-09). Same-machine A/B benchmark (400 memories / ~47k chars, 3 runs each): **search p50 6.3→3.0ms (2.1×), p95 11.7→4.2ms (2.8×)**; 24k-char library p50 2.5ms; fast machine state 1.8ms. Five surgical cuts: WAL + `synchronous=NORMAL` (commits skip per-write fsync), enum-downgrade memo, `row.keys()` set reuse, first-visit access-count batching merged into one `flush_access()` transaction per search, and search-audit INSERT folded into the same commit. The v5.6.3 throttle contract and v5.5.4 `recently_accessed` semantics are pinned by 23 contract tests — all green. Compliance-grade audits still commit immediately. *(Note: the `synchronous=NORMAL` cut landed in v6.4.3 — see above.)*
@@ -168,7 +186,7 @@ v6.1.3 – v6.4.5 shipped in ten days (2026-11-01 → 11-10; two same-day releas
 
 ```
 ┌──────────────────────────────────────────────────────┐
-│           MindForge v6.4.5            │
+│           MindForge v6.4.8            │
 │  Cognitive Layer  Personality · KnowledgeGraph       │
 │                   MemoryEvolution · FederatedMemory  │
 ├──────────────────────────────────────────────────────┤

{mindforge_memory-6.4.5 → mindforge_memory-6.4.8}/mindforge_memory.egg-info/SOURCES.txt

@@ -68,6 +68,7 @@ mindforge_memory.egg-info/requires.txt
 mindforge_memory.egg-info/top_level.txt
 tests/test_core.py
 tests/test_security_hardening.py
+tests/test_trash_hard_delete.py
 tests/test_v546_features.py
 tests/test_v547_fixes.py
 tests/test_v548_features.py
@@ -133,4 +134,6 @@ tests/test_v635_integration.py
 tests/test_v636_fixes.py
 tests/test_v643_fixes.py
 tests/test_v645_fixes.py
+tests/test_v646_fixes.py
+tests/test_v648_fixes.py
 tests/test_v6_ports.py