superlocalmemory 3.4.58 → 3.4.59

package/CHANGELOG.md

@@ -5,6 +5,66 @@ All notable changes to SuperLocalMemory V3 will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [3.4.59] - 2026-05-31 — Graph Edge Cap + Recall Reliability
+
+**Fixes SLM falling into degraded FTS5 mode on every session start**, and stops
+the knowledge graph from growing into an O(n²) edge explosion that made recall
+slow as the fact corpus scaled beyond 10K facts.
+
+### Root Causes Fixed
+
+**1. MCP timeout too aggressive for dense graphs (degraded mode bug)**
+The v3.4.57 timeout reduction (60s→8s) was correct for small graphs but
+backfired at scale. With a 17K-fact corpus and 2.1M graph edges, full 6-channel
+recall (including spreading activation + Hopfield) takes 13–15s. The 8s timeout
+always expired → `pool_recall` raised `PoolError` → session_init fell back to
+emergency FTS5 BM25. Every session started in degraded mode silently.
+
+**Fix:** `DaemonPoolProxy.timeout_s` raised from 8s to 30s. Covers the current
+worst-case (13.4s full recall) with 2× headroom. The orphan-flood concern from
+v3.4.57 is mitigated by the ingest cap reductions below, which stop the graph
+from growing further.
+
+**2. Knowledge graph edge explosion (O(n²) growth)**
+At 17K facts, the corpus hit 2.1M graph edges (avg 121 per node). Root cause:
+`_MAX_ENTITY_EDGES_PER_ENTITY = 20` was designed for ~500-fact graphs. At scale,
+hub entities (appearing in 1000s of facts) accumulated 5000+ edges per node.
+Spreading activation must fan out across all edges per node, causing 9s SA time.
+
+**Fixes:**
+- `_MAX_ENTITY_EDGES_PER_ENTITY` lowered from 20 → 5
+- `_MAX_CAUSAL_EDGES_PER_ENTITY` lowered from 20 → 5
+- **Hub node filter added:** nodes already at ≥ 200 total edges are skipped
+  during ingest. High-frequency hub nodes (e.g. a term appearing in every fact)
+  link everything to everything — they are graph noise, not graph signal.
+- Hub cache shared across entity/causal edge builders per `build_edges` call to
+  avoid redundant DB queries.
+
+**3. Spreading activation UNION query not using indexes (SA slow path)**
+The `_get_unified_neighbors` UNION ALL query fetched all edges for a node then
+sorted them, preventing the `idx_edges_source_weight` and `idx_edges_target_weight`
+covering indexes from terminating early. Fix: push `ORDER BY weight DESC LIMIT ?`
+inside each UNION branch (wrapped in `SELECT * FROM (...)` per SQLite compound
+SELECT syntax). SQLite now stops after `max_neighbors_per_node` rows per branch
+using the covering index instead of materialising the full edge set.
+
+**4. Degree-cap pruner added to graph_pruner.py**
+New `_cap_node_degree()` function using `ROW_NUMBER() OVER (PARTITION BY source_id
+ORDER BY weight DESC)` — single-pass window function, no Python loops. Integrated
+into `prune_graph()` as `cap_degree=True` (default). Automatically runs during
+scheduled maintenance cycles to keep hub nodes bounded.
+
+### Changed
+- `mcp/_daemon_proxy.py`: `timeout_s` default 8.0 → 30.0
+- `encoding/graph_builder.py`: entity + causal caps 20 → 5; hub filter at 200 edges
+- `core/graph_pruner.py`: `_cap_node_degree()` added; `prune_graph()` gains `cap_degree` param
+- `retrieval/spreading_activation.py`: UNION LIMIT pushed inside each branch
+
+### Tests
+4053 passed, 15 skipped — no regressions.
+
+---
+
 ## [3.4.58] - 2026-05-30 — Permanent OpenMP SIGSEGV Fix
 
 **Eliminates the recurring Python crash popup on macOS Apple Silicon.** Any user

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "superlocalmemory",
-  "version": "3.4.58",
+  "version": "3.4.59",
   "description": "Information-geometric agent memory with mathematical guarantees. 4-channel retrieval, Fisher-Rao similarity, zero-LLM mode, EU AI Act compliant. Works with Claude, Cursor, Windsurf, and 17+ AI tools.",
   "keywords": [
     "ai-memory",

package/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "superlocalmemory"
-version = "3.4.58"
+version = "3.4.59"
 description = "Information-geometric agent memory with mathematical guarantees"
 readme = "README.md"
 license = {text = "AGPL-3.0-or-later"}

package/src/superlocalmemory/__init__.py

@@ -28,7 +28,7 @@ if "OMP_NUM_THREADS" not in os.environ:
     os.environ["OMP_NUM_THREADS"] = "2"
 # ---------------------------------------------------------------------------
 
-__version__ = "3.4.58"
+__version__ = "3.4.59"
 
 _REQUIRED_VERSIONS = {
     "sentence_transformers": "5.3.0",

package/src/superlocalmemory/core/graph_pruner.py

@@ -28,12 +28,18 @@ from pathlib import Path
 logger = logging.getLogger("superlocalmemory.graph_pruner")
 
 _CHAIN_BATCH_LIMIT = 10_000
+# v3.4.59: Nodes with more than this many total edges (in+out) are hub nodes.
+# SA and entity_graph channels cap fan-out at 30/node, so edges beyond this
+# threshold provide zero additional recall signal while bloating the graph.
+_MAX_DEGREE_PER_NODE: int = 100
+_HUB_PRUNE_BATCH: int = 500  # delete edges in batches to avoid giant IN clauses
 
 
 def prune_graph(
     db_path: str | Path,
     profile_id: str = "default",
     dry_run: bool = False,
+    cap_degree: bool = True,
 ) -> dict:
     """Run all graph pruning strategies for a specific profile.
 
@@ -41,7 +47,7 @@ def prune_graph(
     """
     conn = sqlite3.connect(str(db_path))
     conn.execute("PRAGMA journal_mode=WAL")
-    conn.execute("PRAGMA busy_timeout=10000")
+    conn.execute("PRAGMA busy_timeout=30000")
     conn.row_factory = sqlite3.Row
 
     stats = {
@@ -49,6 +55,7 @@ def prune_graph(
         "supersedes_collapsed": 0,
         "self_loops_removed": 0,
         "duplicates_removed": 0,
+        "hub_edges_removed": 0,
         "total_before": 0,
         "total_after": 0,
     }
@@ -72,6 +79,10 @@ def prune_graph(
         stats["supersedes_collapsed"] = _collapse_supersedes_chains(
             c, profile_id, dry_run,
         )
+        if cap_degree:
+            stats["hub_edges_removed"] = _cap_node_degree(
+                c, profile_id, _MAX_DEGREE_PER_NODE, dry_run,
+            )
 
         if dry_run:
             c.execute("ROLLBACK")
@@ -91,10 +102,11 @@ def prune_graph(
         prefix = "(dry-run) " if dry_run else ""
         logger.info(
             "%sGraph pruning: removed %d edges (%.1f%%) in %.1fs — "
-            "orphans=%d, supersedes=%d, self_loops=%d, duplicates=%d",
+            "orphans=%d, supersedes=%d, self_loops=%d, duplicates=%d, hub_cap=%d",
             prefix, total_removed, pct, elapsed,
             stats["orphans_removed"], stats["supersedes_collapsed"],
             stats["self_loops_removed"], stats["duplicates_removed"],
+            stats["hub_edges_removed"],
         )
 
     except Exception as exc:
@@ -288,3 +300,81 @@ def _collapse_supersedes_chains(
     )
 
     return len(delete_ids)
+
+
+def _cap_node_degree(
+    c: sqlite3.Cursor,
+    profile_id: str,
+    max_degree: int,
+    dry_run: bool,
+) -> int:
+    """Remove low-weight edges from hub nodes (nodes with degree > max_degree).
+
+    v3.4.59: At 17K facts, popular entities (SLM, Claude, AgentAssert) caused
+    1.4M+ entity edges — avg 121 per node, some nodes with 5000+. SA fan-out
+    is capped at 30/node during traversal, so edges beyond max_degree add zero
+    recall signal while making every graph query scan millions of rows.
+
+    Algorithm (single-pass window function — no Python loops):
+      1. ROW_NUMBER() OVER (PARTITION BY source_id ORDER BY weight DESC) ranks
+         every edge per node in one full table scan.
+      2. Edges with rn > max_degree are deleted in a single DELETE statement.
+      Requires SQLite 3.25+ (window functions). System is on 3.53.1.
+    """
+    if dry_run:
+        c.execute(
+            """
+            SELECT COUNT(*) as cnt FROM (
+                SELECT edge_id,
+                       ROW_NUMBER() OVER (
+                           PARTITION BY source_id ORDER BY weight DESC
+                       ) as rn
+                FROM graph_edges
+                WHERE profile_id = ?
+            ) WHERE rn > ?
+            """,
+            (profile_id, max_degree),
+        )
+        excess = c.fetchone()["cnt"]
+        logger.info(
+            "(dry-run) _cap_node_degree: ~%d edges would be removed (max_degree=%d)",
+            excess, max_degree,
+        )
+        return excess
+
+    # Step 1: build temp keep-list in one pass (ROW_NUMBER ranks by weight DESC)
+    c.execute("CREATE TEMP TABLE IF NOT EXISTS _slm_keep_edges (edge_id TEXT PRIMARY KEY)")
+    c.execute("DELETE FROM _slm_keep_edges")  # idempotent if called twice
+    c.execute(
+        """
+        INSERT INTO _slm_keep_edges (edge_id)
+        SELECT edge_id FROM (
+            SELECT edge_id,
+                   ROW_NUMBER() OVER (
+                       PARTITION BY source_id ORDER BY weight DESC
+                   ) as rn
+            FROM graph_edges
+            WHERE profile_id = ?
+        ) WHERE rn <= ?
+        """,
+        (profile_id, max_degree),
+    )
+
+    # Step 2: delete everything not in keep-list (single DELETE)
+    c.execute(
+        """
+        DELETE FROM graph_edges
+        WHERE profile_id = ?
+          AND edge_id NOT IN (SELECT edge_id FROM _slm_keep_edges)
+        """,
+        (profile_id,),
+    )
+    deleted = c.rowcount
+
+    c.execute("DROP TABLE IF EXISTS _slm_keep_edges")
+
+    logger.info(
+        "_cap_node_degree: deleted %d low-weight edges (max_degree=%d)",
+        deleted, max_degree,
+    )
+    return deleted

package/src/superlocalmemory/encoding/graph_builder.py

@@ -76,11 +76,12 @@ class GraphBuilder:
 
     def build_edges(self, new_fact: AtomicFact, profile_id: str) -> list[GraphEdge]:
         """Create ALL relevant edges for *new_fact*. Persists and returns them."""
+        hub_cache: dict[str, int] = {}  # shared across edge types to avoid redundant queries
         edges: list[GraphEdge] = []
-        edges.extend(self._build_entity_edges(new_fact, profile_id))
+        edges.extend(self._build_entity_edges(new_fact, profile_id, hub_cache))
         edges.extend(self._build_temporal_edges(new_fact, profile_id))
         edges.extend(self._build_semantic_edges(new_fact, profile_id))
-        edges.extend(self._build_causal_edges(new_fact, profile_id))
+        edges.extend(self._build_causal_edges(new_fact, profile_id, hub_cache))
 
         for edge in edges:
             self._db.store_edge(edge)
@@ -142,17 +143,33 @@ class GraphBuilder:
 
     # -- Edge builders (private) -------------------------------------------
 
-    # V3.3.12: Cap entity edges per entity to prevent O(n²) explosion.
-    # With 500+ facts sharing a popular entity, creating an edge to each
-    # produced 44K+ edges and 22-min ingestion. Cap to 20 most recent per entity.
-    _MAX_ENTITY_EDGES_PER_ENTITY: int = 20
+    # v3.4.59: Lowered from 20→5. At 17K facts, 20 edges × many entities per fact
+    # produces 1.4M+ entity edges and 13s recalls. 5 is sufficient graph signal.
+    _MAX_ENTITY_EDGES_PER_ENTITY: int = 5
+    # v3.4.59: Hub filter — nodes with > 200 total edges are "gravity wells"
+    # (e.g. every fact mentions "SLM"). Skip adding more edges to them.
+    _MAX_HUB_DEGREE: int = 200
+
+    def _node_degree(self, fact_id: str, profile_id: str, cache: dict[str, int]) -> int:
+        """Cached total degree (in + out) for a node."""
+        if fact_id not in cache:
+            rows = self._db.execute(
+                "SELECT COUNT(*) as cnt FROM graph_edges "
+                "WHERE profile_id = ? AND (source_id = ? OR target_id = ?)",
+                (profile_id, fact_id, fact_id),
+            )
+            cache[fact_id] = int(dict(rows[0])["cnt"]) if rows else 0
+        return cache[fact_id]
 
     def _build_entity_edges(
         self, new_fact: AtomicFact, profile_id: str,
+        hub_cache: dict[str, int] | None = None,
     ) -> list[GraphEdge]:
         """ENTITY edges: shared canonical entity — capped to most recent per entity."""
         if not new_fact.canonical_entities:
             return []
+        if hub_cache is None:
+            hub_cache = {}
         edges: list[GraphEdge] = []
         seen: set[str] = set()
 
@@ -163,6 +180,8 @@ class GraphBuilder:
                     break
                 if other.fact_id == new_fact.fact_id or other.fact_id in seen:
                     continue
+                if self._node_degree(other.fact_id, profile_id, hub_cache) >= self._MAX_HUB_DEGREE:
+                    continue  # skip hub nodes — they link everything to everything
                 if self._edge_exists(new_fact.fact_id, other.fact_id, EdgeType.ENTITY, profile_id):
                     continue
                 seen.add(other.fact_id)
@@ -261,17 +280,20 @@ class GraphBuilder:
                 break
         return edges
 
-    # V3.3.13: Cap causal edges per entity to prevent O(n²) explosion (same as entity/temporal).
-    _MAX_CAUSAL_EDGES_PER_ENTITY: int = 20
+    # v3.4.59: Lowered from 20→5, same reasoning as entity cap.
+    _MAX_CAUSAL_EDGES_PER_ENTITY: int = 5
 
     def _build_causal_edges(
         self, new_fact: AtomicFact, profile_id: str,
+        hub_cache: dict[str, int] | None = None,
     ) -> list[GraphEdge]:
         """CAUSAL edges: causal markers + shared entity. Direction: cause -> effect."""
         if not any(p.search(new_fact.content) for p in _CAUSAL_CUES):
             return []
         if not new_fact.canonical_entities:
             return []
+        if hub_cache is None:
+            hub_cache = {}
 
         edges: list[GraphEdge] = []
         seen: set[str] = set()
@@ -282,6 +304,8 @@ class GraphBuilder:
                     break
                 if other.fact_id == new_fact.fact_id or other.fact_id in seen:
                     continue
+                if self._node_degree(other.fact_id, profile_id, hub_cache) >= self._MAX_HUB_DEGREE:
+                    continue
                 if self._edge_exists(other.fact_id, new_fact.fact_id, EdgeType.CAUSAL, profile_id):
                     continue
                 seen.add(other.fact_id)

package/src/superlocalmemory/mcp/_daemon_proxy.py

@@ -36,7 +36,7 @@ class DaemonPoolProxy:
     envelopes — the adapter is responsible for surfacing those.
     """
 
-    def __init__(self, port: int, *, timeout_s: float = 8.0) -> None:  # v3.4.57: 60s→8s — prevents orphan flood from blocking daemon event loop
+    def __init__(self, port: int, *, timeout_s: float = 30.0) -> None:  # v3.4.59: 8s→30s — observed recall takes 13.4s on dense graph (2.1M edges); 8s always timed out → degraded mode
         self._port = port
         self._timeout = timeout_s
 

package/src/superlocalmemory/retrieval/spreading_activation.py

@@ -234,27 +234,47 @@ class SpreadingActivation:
         the highest-signal edges.
         """
         try:
+            # v3.4.59: LIMIT pushed inside each UNION branch so SQLite can use
+            # idx_edges_source_weight / idx_edges_target_weight and stop after
+            # max_neighbors_per_node rows per branch instead of materializing
+            # all 2.1M edges then sorting. Each branch wrapped in SELECT * FROM (...)
+            # because SQLite requires parentheses for ORDER BY+LIMIT in compound SELECTs.
+            lim = self._config.max_neighbors_per_node
             rows = self._db.execute(
                 """
                 SELECT neighbor_id, weight FROM (
-                    SELECT target_id AS neighbor_id, weight FROM graph_edges
+                    SELECT * FROM (
+                        SELECT target_id AS neighbor_id, weight FROM graph_edges
                         WHERE source_id = ? AND profile_id = ?
+                        ORDER BY weight DESC LIMIT ?
+                    )
                     UNION ALL
-                    SELECT target_fact_id AS neighbor_id, weight FROM association_edges
+                    SELECT * FROM (
+                        SELECT target_fact_id AS neighbor_id, weight FROM association_edges
                         WHERE source_fact_id = ? AND profile_id = ?
+                        ORDER BY weight DESC LIMIT ?
+                    )
                     UNION ALL
-                    SELECT source_id AS neighbor_id, weight FROM graph_edges
+                    SELECT * FROM (
+                        SELECT source_id AS neighbor_id, weight FROM graph_edges
                         WHERE target_id = ? AND profile_id = ?
+                        ORDER BY weight DESC LIMIT ?
+                    )
                     UNION ALL
-                    SELECT source_fact_id AS neighbor_id, weight FROM association_edges
+                    SELECT * FROM (
+                        SELECT source_fact_id AS neighbor_id, weight FROM association_edges
                         WHERE target_fact_id = ? AND profile_id = ?
+                        ORDER BY weight DESC LIMIT ?
+                    )
                 )
                 ORDER BY weight DESC
                 LIMIT ?
                 """,
-                (node_id, profile_id, node_id, profile_id,
-                 node_id, profile_id, node_id, profile_id,
-                 self._config.max_neighbors_per_node),
+                (node_id, profile_id, lim,
+                 node_id, profile_id, lim,
+                 node_id, profile_id, lim,
+                 node_id, profile_id, lim,
+                 lim),
             )
             return [
                 (dict(r)["neighbor_id"], dict(r)["weight"]) for r in rows

package/src/superlocalmemory.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: superlocalmemory
-Version: 3.4.58
+Version: 3.4.59
 Summary: Information-geometric agent memory with mathematical guarantees
 Author-email: Varun Pratap Bhardwaj <admin@superlocalmemory.com>
 License: AGPL-3.0-or-later