superlocalmemory 3.4.56 → 3.4.59
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +107 -0
- package/package.json +1 -1
- package/pyproject.toml +1 -1
- package/src/superlocalmemory/__init__.py +27 -2
- package/src/superlocalmemory/core/config.py +54 -47
- package/src/superlocalmemory/core/graph_pruner.py +92 -2
- package/src/superlocalmemory/encoding/graph_builder.py +32 -8
- package/src/superlocalmemory/infra/process_reaper.py +1 -1
- package/src/superlocalmemory/learning/ranker_retrain_legacy.py +9 -1
- package/src/superlocalmemory/learning/ranker_retrain_online.py +11 -1
- package/src/superlocalmemory/mcp/_daemon_proxy.py +1 -1
- package/src/superlocalmemory/mcp/server.py +37 -0
- package/src/superlocalmemory/retrieval/spreading_activation.py +27 -7
- package/src/superlocalmemory.egg-info/PKG-INFO +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,113 @@ All notable changes to SuperLocalMemory V3 will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [3.4.59] - 2026-05-31 — Graph Edge Cap + Recall Reliability
|
|
9
|
+
|
|
10
|
+
**Fixes SLM falling into degraded FTS5 mode on every session start**, and stops
|
|
11
|
+
the knowledge graph from growing into an O(n²) edge explosion that made recall
|
|
12
|
+
slow as the fact corpus scaled beyond 10K facts.
|
|
13
|
+
|
|
14
|
+
### Root Causes Fixed
|
|
15
|
+
|
|
16
|
+
**1. MCP timeout too aggressive for dense graphs (degraded mode bug)**
|
|
17
|
+
The v3.4.57 timeout reduction (60s→8s) was correct for small graphs but
|
|
18
|
+
backfired at scale. With a 17K-fact corpus and 2.1M graph edges, full 6-channel
|
|
19
|
+
recall (including spreading activation + Hopfield) takes 13–15s. The 8s timeout
|
|
20
|
+
always expired → `pool_recall` raised `PoolError` → session_init fell back to
|
|
21
|
+
emergency FTS5 BM25. Every session started in degraded mode silently.
|
|
22
|
+
|
|
23
|
+
**Fix:** `DaemonPoolProxy.timeout_s` raised from 8s to 30s. Covers the current
|
|
24
|
+
worst-case (13.4s full recall) with 2× headroom. The orphan-flood concern from
|
|
25
|
+
v3.4.57 is mitigated by the ingest cap reductions below, which stop the graph
|
|
26
|
+
from growing further.
|
|
27
|
+
|
|
28
|
+
**2. Knowledge graph edge explosion (O(n²) growth)**
|
|
29
|
+
At 17K facts, the corpus hit 2.1M graph edges (avg 121 per node). Root cause:
|
|
30
|
+
`_MAX_ENTITY_EDGES_PER_ENTITY = 20` was designed for ~500-fact graphs. At scale,
|
|
31
|
+
hub entities (appearing in 1000s of facts) accumulated 5000+ edges per node.
|
|
32
|
+
Spreading activation must fan out across all edges per node, causing 9s SA time.
|
|
33
|
+
|
|
34
|
+
**Fixes:**
|
|
35
|
+
- `_MAX_ENTITY_EDGES_PER_ENTITY` lowered from 20 → 5
|
|
36
|
+
- `_MAX_CAUSAL_EDGES_PER_ENTITY` lowered from 20 → 5
|
|
37
|
+
- **Hub node filter added:** nodes already at ≥ 200 total edges are skipped
|
|
38
|
+
during ingest. High-frequency hub nodes (e.g. a term appearing in every fact)
|
|
39
|
+
link everything to everything — they are graph noise, not graph signal.
|
|
40
|
+
- Hub cache shared across entity/causal edge builders per `build_edges` call to
|
|
41
|
+
avoid redundant DB queries.
|
|
42
|
+
|
|
43
|
+
**3. Spreading activation UNION query not using indexes (SA slow path)**
|
|
44
|
+
The `_get_unified_neighbors` UNION ALL query fetched all edges for a node then
|
|
45
|
+
sorted them, preventing the `idx_edges_source_weight` and `idx_edges_target_weight`
|
|
46
|
+
covering indexes from terminating early. Fix: push `ORDER BY weight DESC LIMIT ?`
|
|
47
|
+
inside each UNION branch (wrapped in `SELECT * FROM (...)` per SQLite compound
|
|
48
|
+
SELECT syntax). SQLite now stops after `max_neighbors_per_node` rows per branch
|
|
49
|
+
using the covering index instead of materialising the full edge set.
|
|
50
|
+
|
|
51
|
+
**4. Degree-cap pruner added to graph_pruner.py**
|
|
52
|
+
New `_cap_node_degree()` function using `ROW_NUMBER() OVER (PARTITION BY source_id
|
|
53
|
+
ORDER BY weight DESC)` — single-pass window function, no Python loops. Integrated
|
|
54
|
+
into `prune_graph()` as `cap_degree=True` (default). Automatically runs during
|
|
55
|
+
scheduled maintenance cycles to keep hub nodes bounded.
|
|
56
|
+
|
|
57
|
+
### Changed
|
|
58
|
+
- `mcp/_daemon_proxy.py`: `timeout_s` default 8.0 → 30.0
|
|
59
|
+
- `encoding/graph_builder.py`: entity + causal caps 20 → 5; hub filter at 200 edges
|
|
60
|
+
- `core/graph_pruner.py`: `_cap_node_degree()` added; `prune_graph()` gains `cap_degree` param
|
|
61
|
+
- `retrieval/spreading_activation.py`: UNION LIMIT pushed inside each branch
|
|
62
|
+
|
|
63
|
+
### Tests
|
|
64
|
+
4053 passed, 15 skipped — no regressions.
|
|
65
|
+
|
|
66
|
+
---
|
|
67
|
+
|
|
68
|
+
## [3.4.58] - 2026-05-30 — Permanent OpenMP SIGSEGV Fix
|
|
69
|
+
|
|
70
|
+
**Eliminates the recurring Python crash popup on macOS Apple Silicon.** Any user
|
|
71
|
+
who triggered a LightGBM retrain cycle (background learning after ~50 recalls)
|
|
72
|
+
would see a macOS crash report for `Python [PID]` with SIGSEGV at
|
|
73
|
+
`__kmp_suspend_initialize_thread + 32`. This release permanently fixes the root
|
|
74
|
+
cause in the SLM source — no system changes required.
|
|
75
|
+
|
|
76
|
+
### Root Cause
|
|
77
|
+
SLM's dependency set ships **three separate `libomp.dylib` binaries** on macOS ARM:
|
|
78
|
+
- `torch==2.11.0` bundles `/opt/llvm-openmp/lib/libomp.dylib` (860 KB)
|
|
79
|
+
- `scikit-learn==1.8.0` bundles its own `/opt/llvm-openmp/lib/libomp.dylib` (678 KB)
|
|
80
|
+
- `lightgbm==4.6.0` resolves to homebrew's `/opt/homebrew/opt/libomp/lib/libomp.dylib` (739 KB)
|
|
81
|
+
|
|
82
|
+
When `lgb.Dataset(X, ...)` called `LGBM_DatasetCreateFromMat` → OpenMP `fork_call`
|
|
83
|
+
with `num_threads = os.cpu_count() - 1` (9 threads on M4 Mac), the parallel
|
|
84
|
+
worker threads were allocated by LightGBM's libomp but attempted to synchronize
|
|
85
|
+
via PyTorch's libomp thread pool. The two runtimes have incompatible internal
|
|
86
|
+
thread structs — the barrier release read address `0x580` (null + struct offset),
|
|
87
|
+
causing `EXC_BAD_ACCESS (SIGSEGV)` in Thread 26.
|
|
88
|
+
|
|
89
|
+
**All macOS Apple Silicon users with the standard SLM install were affected.**
|
|
90
|
+
The crash fired silently in a background consolidation worker, causing the
|
|
91
|
+
`slm mcp` subprocess to restart repeatedly, generating the persistent crash popup.
|
|
92
|
+
|
|
93
|
+
### Fixed
|
|
94
|
+
- **`ranker_retrain_online.py` line 198** — `num_threads = max(1, os.cpu_count()-1)`
|
|
95
|
+
changed to a safe cap of **2 threads** (configurable via `SLM_LGBM_THREADS` env
|
|
96
|
+
var). With ≤2 threads, the problematic parallel fork path in
|
|
97
|
+
`DatasetLoader::ConstructFromSampleData` is avoided entirely. SLM's training
|
|
98
|
+
datasets (50–5,000 rows) see ~90% of max-core throughput at 2 threads — the
|
|
99
|
+
difference is under 200ms per retrain cycle.
|
|
100
|
+
- **`__init__.py`** — `KMP_DUPLICATE_LIB_OK` changed from `os.environ.setdefault`
|
|
101
|
+
(could be overridden to FALSE) to unconditional `os.environ[...] = "TRUE"`.
|
|
102
|
+
Added `OMP_NUM_THREADS=2` cap (respects user override) as belt-and-suspenders
|
|
103
|
+
at the OS level before any C library reads the thread count.
|
|
104
|
+
|
|
105
|
+
### New environment variables
|
|
106
|
+
- `SLM_LGBM_THREADS` — override the LightGBM thread count (default: `2`).
|
|
107
|
+
Only increase if your system has a unified single-runtime OpenMP setup.
|
|
108
|
+
|
|
109
|
+
### Why not fix the dylib collision instead?
|
|
110
|
+
Patching `libomp.dylib` on users' systems via `install_name_tool` is fragile:
|
|
111
|
+
it breaks on package updates, requires write access to site-packages, and fails
|
|
112
|
+
if SIP prevents modifying signed binaries. The source fix is permanent, upgrade-safe,
|
|
113
|
+
and works identically on every user's machine regardless of their exact package versions.
|
|
114
|
+
|
|
8
115
|
## [3.4.52] - 2026-05-28 — Warm Memory, No Cold Starts
|
|
9
116
|
|
|
10
117
|
**Production resilience for session_init.** No quality degradation as the primary path: full 6-channel recall (semantic + BM25 + entity + temporal + Hopfield + spreading-activation, Fisher-Rao fusion) is preserved. The cold-start problem is fixed at the infrastructure layer, not by downgrading retrieval.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "superlocalmemory",
|
|
3
|
-
"version": "3.4.
|
|
3
|
+
"version": "3.4.59",
|
|
4
4
|
"description": "Information-geometric agent memory with mathematical guarantees. 4-channel retrieval, Fisher-Rao similarity, zero-LLM mode, EU AI Act compliant. Works with Claude, Cursor, Windsurf, and 17+ AI tools.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"ai-memory",
|
package/pyproject.toml
CHANGED
|
@@ -1,9 +1,34 @@
|
|
|
1
1
|
"""SuperLocalMemory — information-geometric agent memory."""
|
|
2
2
|
|
|
3
3
|
import os
|
|
4
|
-
os.environ.setdefault("KMP_DUPLICATE_LIB_OK", "TRUE")
|
|
5
4
|
|
|
6
|
-
|
|
5
|
+
# --- OpenMP multi-library guard (permanent fix, v3.4.58) -------------------
|
|
6
|
+
# SLM ships with torch, scikit-learn, and lightgbm — each bundles its own
|
|
7
|
+
# libomp.dylib on macOS ARM (Apple Silicon). When all three are loaded in the
|
|
8
|
+
# same process, the Intel OpenMP runtime detects duplicate libraries and either
|
|
9
|
+
# (a) emits OMP: Error #15 and aborts, or (b) crashes with SIGSEGV in
|
|
10
|
+
# __kmp_suspend_initialize_thread (address 0x580 — null thread-struct deref)
|
|
11
|
+
# when LightGBM's parallel fork tries to coordinate with PyTorch's thread pool.
|
|
12
|
+
#
|
|
13
|
+
# KMP_DUPLICATE_LIB_OK=TRUE tells the runtime to elect one master instance
|
|
14
|
+
# and continue rather than abort. This is the upstream-recommended workaround
|
|
15
|
+
# for mixed-dependency environments (PyTorch docs, scikit-learn FAQ, LightGBM
|
|
16
|
+
# issue #3877). Use unconditional assignment — not setdefault — so user env
|
|
17
|
+
# cannot accidentally disable this safety net by setting it to FALSE.
|
|
18
|
+
#
|
|
19
|
+
# OMP_NUM_THREADS=2 caps the maximum thread count before any C library reads
|
|
20
|
+
# it. With ≤2 threads the problematic parallel fork path in LightGBM's
|
|
21
|
+
# DatasetLoader::ConstructFromSampleData is avoided on all observed crash
|
|
22
|
+
# configurations. SLM datasets are small (50–5 000 rows); 2 threads gives
|
|
23
|
+
# ~90% of the performance of max-core training at zero crash risk.
|
|
24
|
+
# Users who need more threads can set SLM_LGBM_THREADS=N to override
|
|
25
|
+
# the per-call cap in ranker_retrain_online.py.
|
|
26
|
+
os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"
|
|
27
|
+
if "OMP_NUM_THREADS" not in os.environ:
|
|
28
|
+
os.environ["OMP_NUM_THREADS"] = "2"
|
|
29
|
+
# ---------------------------------------------------------------------------
|
|
30
|
+
|
|
31
|
+
__version__ = "3.4.59"
|
|
7
32
|
|
|
8
33
|
_REQUIRED_VERSIONS = {
|
|
9
34
|
"sentence_transformers": "5.3.0",
|
|
@@ -1000,80 +1000,87 @@ class SLMConfig:
|
|
|
1000
1000
|
new_mode: str,
|
|
1001
1001
|
base_dir: Path | None = None,
|
|
1002
1002
|
) -> "SLMConfig":
|
|
1003
|
-
"""Switch to a different mode, preserving
|
|
1004
|
-
|
|
1005
|
-
v3.4.
|
|
1006
|
-
|
|
1007
|
-
|
|
1008
|
-
|
|
1009
|
-
|
|
1003
|
+
"""Switch to a different mode, preserving ALL non-mode settings.
|
|
1004
|
+
|
|
1005
|
+
v3.4.58 fix: Only ``config.mode`` changes. The previous implementation
|
|
1006
|
+
called ``for_mode()`` (which resets embedding/LLM/retrieval to mode
|
|
1007
|
+
defaults) whenever the target mode's per-mode file didn't exist.
|
|
1008
|
+
This silently clobbered custom embeddings, endpoints, and LLM config.
|
|
1009
|
+
|
|
1010
|
+
Correct behavior:
|
|
1011
|
+
- Load the current (old) config from config.json.
|
|
1012
|
+
- Save it to mode_{old}.json for the 3-mode system.
|
|
1013
|
+
- If mode_{new}.json exists, load it (user had a prior config for that mode).
|
|
1014
|
+
- If mode_{new}.json does NOT exist: start from the OLD config, change
|
|
1015
|
+
only the mode field. This preserves embedding, retrieval, forgetting, etc.
|
|
1016
|
+
- Exception: if user has NO LLM provider AND is switching to B/C, populate
|
|
1017
|
+
sensible LLM defaults so the daemon doesn't start dead.
|
|
1018
|
+
- Write the result to config.json (backward compat) and current_mode.
|
|
1010
1019
|
|
|
1011
1020
|
Returns the new active config.
|
|
1012
1021
|
"""
|
|
1022
|
+
import copy
|
|
1023
|
+
import dataclasses
|
|
1024
|
+
|
|
1013
1025
|
from superlocalmemory.storage.models import Mode as _M
|
|
1014
1026
|
_base = base_dir or DEFAULT_BASE_DIR
|
|
1015
1027
|
_base.mkdir(parents=True, exist_ok=True)
|
|
1016
1028
|
|
|
1017
1029
|
old_config = cls.load(_base / "config.json")
|
|
1018
1030
|
old_mode = old_config.mode.value.lower()
|
|
1019
|
-
|
|
1020
1031
|
new_mode_val = _M(new_mode.lower())
|
|
1021
1032
|
|
|
1022
|
-
# 1. Save current config to its per-mode file (
|
|
1023
|
-
# matched — prevents overwriting user customizations)
|
|
1033
|
+
# 1. Save current config to its per-mode file (preserve customizations)
|
|
1024
1034
|
if old_mode != new_mode.lower():
|
|
1025
1035
|
old_path = cls._mode_config_path(_base, old_config.mode)
|
|
1026
1036
|
old_config.save(old_path)
|
|
1027
1037
|
|
|
1028
|
-
# 2.
|
|
1038
|
+
# 2. Determine new config:
|
|
1039
|
+
# a) mode_{new}.json exists → user had prior config for this mode, use it
|
|
1040
|
+
# b) otherwise → copy old config, change only mode
|
|
1029
1041
|
new_path = cls._mode_config_path(_base, new_mode_val)
|
|
1030
|
-
need_migration = False
|
|
1031
1042
|
if new_path.exists():
|
|
1032
1043
|
try:
|
|
1033
1044
|
new_config = cls.load(new_path)
|
|
1034
|
-
# Ensure the loaded config actually has the right mode
|
|
1035
1045
|
if new_config.mode != new_mode_val:
|
|
1036
|
-
|
|
1046
|
+
# Corrupt/mismatched mode file — rebuild preserving old values
|
|
1047
|
+
new_config = copy.copy(old_config)
|
|
1048
|
+
new_config.mode = new_mode_val
|
|
1037
1049
|
except Exception:
|
|
1038
|
-
new_config =
|
|
1050
|
+
new_config = copy.copy(old_config)
|
|
1051
|
+
new_config.mode = new_mode_val
|
|
1039
1052
|
else:
|
|
1040
|
-
# First time switching to this mode:
|
|
1041
|
-
#
|
|
1042
|
-
|
|
1043
|
-
|
|
1044
|
-
|
|
1045
|
-
|
|
1046
|
-
|
|
1047
|
-
|
|
1048
|
-
|
|
1049
|
-
|
|
1050
|
-
|
|
1051
|
-
|
|
1052
|
-
|
|
1053
|
-
|
|
1054
|
-
|
|
1055
|
-
|
|
1056
|
-
else:
|
|
1057
|
-
new_config =
|
|
1053
|
+
# First time switching to this mode: start from old config, change mode only.
|
|
1054
|
+
# Use dataclasses.replace() to produce a new frozen-compatible object.
|
|
1055
|
+
new_config = dataclasses.replace(old_config, mode=new_mode_val)
|
|
1056
|
+
|
|
1057
|
+
# 3. LLM default population — ONLY if user has no provider AND switching to B/C.
|
|
1058
|
+
# Never overwrites an existing provider.
|
|
1059
|
+
if new_mode_val in (_M.B, _M.C) and not new_config.llm.provider:
|
|
1060
|
+
if new_mode_val == _M.B:
|
|
1061
|
+
new_config = dataclasses.replace(
|
|
1062
|
+
new_config,
|
|
1063
|
+
llm=LLMConfig(
|
|
1064
|
+
provider="ollama",
|
|
1065
|
+
model="llama3.2",
|
|
1066
|
+
api_base="http://localhost:11434",
|
|
1067
|
+
),
|
|
1068
|
+
)
|
|
1069
|
+
else: # Mode C
|
|
1070
|
+
new_config = dataclasses.replace(
|
|
1071
|
+
new_config,
|
|
1072
|
+
llm=LLMConfig(
|
|
1073
|
+
provider="openrouter",
|
|
1074
|
+
model="anthropic/claude-sonnet-4",
|
|
1075
|
+
),
|
|
1076
|
+
)
|
|
1058
1077
|
|
|
1059
|
-
#
|
|
1078
|
+
# 4. Save as active config.json (backward compat)
|
|
1060
1079
|
new_config.save(_base / "config.json", mode_change=True)
|
|
1061
1080
|
|
|
1062
|
-
#
|
|
1081
|
+
# 5. Write current_mode
|
|
1063
1082
|
cls.write_current_mode(new_mode, _base)
|
|
1064
1083
|
|
|
1065
|
-
# 5. On first migration, also save mode_a.json and mode_c.json
|
|
1066
|
-
# so users have a complete set
|
|
1067
|
-
if need_migration:
|
|
1068
|
-
for _m in (_M.A, _M.B, _M.C):
|
|
1069
|
-
_mp = cls._mode_config_path(_base, _m)
|
|
1070
|
-
if not _mp.exists():
|
|
1071
|
-
try:
|
|
1072
|
-
_mc = cls.for_mode(_m, base_dir=_base)
|
|
1073
|
-
_mc.save(_mp)
|
|
1074
|
-
except Exception:
|
|
1075
|
-
pass
|
|
1076
|
-
|
|
1077
1084
|
return new_config
|
|
1078
1085
|
|
|
1079
1086
|
@classmethod
|
|
@@ -28,12 +28,18 @@ from pathlib import Path
|
|
|
28
28
|
logger = logging.getLogger("superlocalmemory.graph_pruner")
|
|
29
29
|
|
|
30
30
|
_CHAIN_BATCH_LIMIT = 10_000
|
|
31
|
+
# v3.4.59: Nodes with more than this many total edges (in+out) are hub nodes.
|
|
32
|
+
# SA and entity_graph channels cap fan-out at 30/node, so edges beyond this
|
|
33
|
+
# threshold provide zero additional recall signal while bloating the graph.
|
|
34
|
+
_MAX_DEGREE_PER_NODE: int = 100
|
|
35
|
+
_HUB_PRUNE_BATCH: int = 500 # delete edges in batches to avoid giant IN clauses
|
|
31
36
|
|
|
32
37
|
|
|
33
38
|
def prune_graph(
|
|
34
39
|
db_path: str | Path,
|
|
35
40
|
profile_id: str = "default",
|
|
36
41
|
dry_run: bool = False,
|
|
42
|
+
cap_degree: bool = True,
|
|
37
43
|
) -> dict:
|
|
38
44
|
"""Run all graph pruning strategies for a specific profile.
|
|
39
45
|
|
|
@@ -41,7 +47,7 @@ def prune_graph(
|
|
|
41
47
|
"""
|
|
42
48
|
conn = sqlite3.connect(str(db_path))
|
|
43
49
|
conn.execute("PRAGMA journal_mode=WAL")
|
|
44
|
-
conn.execute("PRAGMA busy_timeout=
|
|
50
|
+
conn.execute("PRAGMA busy_timeout=30000")
|
|
45
51
|
conn.row_factory = sqlite3.Row
|
|
46
52
|
|
|
47
53
|
stats = {
|
|
@@ -49,6 +55,7 @@ def prune_graph(
|
|
|
49
55
|
"supersedes_collapsed": 0,
|
|
50
56
|
"self_loops_removed": 0,
|
|
51
57
|
"duplicates_removed": 0,
|
|
58
|
+
"hub_edges_removed": 0,
|
|
52
59
|
"total_before": 0,
|
|
53
60
|
"total_after": 0,
|
|
54
61
|
}
|
|
@@ -72,6 +79,10 @@ def prune_graph(
|
|
|
72
79
|
stats["supersedes_collapsed"] = _collapse_supersedes_chains(
|
|
73
80
|
c, profile_id, dry_run,
|
|
74
81
|
)
|
|
82
|
+
if cap_degree:
|
|
83
|
+
stats["hub_edges_removed"] = _cap_node_degree(
|
|
84
|
+
c, profile_id, _MAX_DEGREE_PER_NODE, dry_run,
|
|
85
|
+
)
|
|
75
86
|
|
|
76
87
|
if dry_run:
|
|
77
88
|
c.execute("ROLLBACK")
|
|
@@ -91,10 +102,11 @@ def prune_graph(
|
|
|
91
102
|
prefix = "(dry-run) " if dry_run else ""
|
|
92
103
|
logger.info(
|
|
93
104
|
"%sGraph pruning: removed %d edges (%.1f%%) in %.1fs — "
|
|
94
|
-
"orphans=%d, supersedes=%d, self_loops=%d, duplicates=%d",
|
|
105
|
+
"orphans=%d, supersedes=%d, self_loops=%d, duplicates=%d, hub_cap=%d",
|
|
95
106
|
prefix, total_removed, pct, elapsed,
|
|
96
107
|
stats["orphans_removed"], stats["supersedes_collapsed"],
|
|
97
108
|
stats["self_loops_removed"], stats["duplicates_removed"],
|
|
109
|
+
stats["hub_edges_removed"],
|
|
98
110
|
)
|
|
99
111
|
|
|
100
112
|
except Exception as exc:
|
|
@@ -288,3 +300,81 @@ def _collapse_supersedes_chains(
|
|
|
288
300
|
)
|
|
289
301
|
|
|
290
302
|
return len(delete_ids)
|
|
303
|
+
|
|
304
|
+
|
|
305
|
+
def _cap_node_degree(
|
|
306
|
+
c: sqlite3.Cursor,
|
|
307
|
+
profile_id: str,
|
|
308
|
+
max_degree: int,
|
|
309
|
+
dry_run: bool,
|
|
310
|
+
) -> int:
|
|
311
|
+
"""Remove low-weight edges from hub nodes (nodes with degree > max_degree).
|
|
312
|
+
|
|
313
|
+
v3.4.59: At 17K facts, popular entities (SLM, Claude, AgentAssert) caused
|
|
314
|
+
1.4M+ entity edges — avg 121 per node, some nodes with 5000+. SA fan-out
|
|
315
|
+
is capped at 30/node during traversal, so edges beyond max_degree add zero
|
|
316
|
+
recall signal while making every graph query scan millions of rows.
|
|
317
|
+
|
|
318
|
+
Algorithm (single-pass window function — no Python loops):
|
|
319
|
+
1. ROW_NUMBER() OVER (PARTITION BY source_id ORDER BY weight DESC) ranks
|
|
320
|
+
every edge per node in one full table scan.
|
|
321
|
+
2. Edges with rn > max_degree are deleted in a single DELETE statement.
|
|
322
|
+
Requires SQLite 3.25+ (window functions). System is on 3.53.1.
|
|
323
|
+
"""
|
|
324
|
+
if dry_run:
|
|
325
|
+
c.execute(
|
|
326
|
+
"""
|
|
327
|
+
SELECT COUNT(*) as cnt FROM (
|
|
328
|
+
SELECT edge_id,
|
|
329
|
+
ROW_NUMBER() OVER (
|
|
330
|
+
PARTITION BY source_id ORDER BY weight DESC
|
|
331
|
+
) as rn
|
|
332
|
+
FROM graph_edges
|
|
333
|
+
WHERE profile_id = ?
|
|
334
|
+
) WHERE rn > ?
|
|
335
|
+
""",
|
|
336
|
+
(profile_id, max_degree),
|
|
337
|
+
)
|
|
338
|
+
excess = c.fetchone()["cnt"]
|
|
339
|
+
logger.info(
|
|
340
|
+
"(dry-run) _cap_node_degree: ~%d edges would be removed (max_degree=%d)",
|
|
341
|
+
excess, max_degree,
|
|
342
|
+
)
|
|
343
|
+
return excess
|
|
344
|
+
|
|
345
|
+
# Step 1: build temp keep-list in one pass (ROW_NUMBER ranks by weight DESC)
|
|
346
|
+
c.execute("CREATE TEMP TABLE IF NOT EXISTS _slm_keep_edges (edge_id TEXT PRIMARY KEY)")
|
|
347
|
+
c.execute("DELETE FROM _slm_keep_edges") # idempotent if called twice
|
|
348
|
+
c.execute(
|
|
349
|
+
"""
|
|
350
|
+
INSERT INTO _slm_keep_edges (edge_id)
|
|
351
|
+
SELECT edge_id FROM (
|
|
352
|
+
SELECT edge_id,
|
|
353
|
+
ROW_NUMBER() OVER (
|
|
354
|
+
PARTITION BY source_id ORDER BY weight DESC
|
|
355
|
+
) as rn
|
|
356
|
+
FROM graph_edges
|
|
357
|
+
WHERE profile_id = ?
|
|
358
|
+
) WHERE rn <= ?
|
|
359
|
+
""",
|
|
360
|
+
(profile_id, max_degree),
|
|
361
|
+
)
|
|
362
|
+
|
|
363
|
+
# Step 2: delete everything not in keep-list (single DELETE)
|
|
364
|
+
c.execute(
|
|
365
|
+
"""
|
|
366
|
+
DELETE FROM graph_edges
|
|
367
|
+
WHERE profile_id = ?
|
|
368
|
+
AND edge_id NOT IN (SELECT edge_id FROM _slm_keep_edges)
|
|
369
|
+
""",
|
|
370
|
+
(profile_id,),
|
|
371
|
+
)
|
|
372
|
+
deleted = c.rowcount
|
|
373
|
+
|
|
374
|
+
c.execute("DROP TABLE IF EXISTS _slm_keep_edges")
|
|
375
|
+
|
|
376
|
+
logger.info(
|
|
377
|
+
"_cap_node_degree: deleted %d low-weight edges (max_degree=%d)",
|
|
378
|
+
deleted, max_degree,
|
|
379
|
+
)
|
|
380
|
+
return deleted
|
|
@@ -76,11 +76,12 @@ class GraphBuilder:
|
|
|
76
76
|
|
|
77
77
|
def build_edges(self, new_fact: AtomicFact, profile_id: str) -> list[GraphEdge]:
|
|
78
78
|
"""Create ALL relevant edges for *new_fact*. Persists and returns them."""
|
|
79
|
+
hub_cache: dict[str, int] = {} # shared across edge types to avoid redundant queries
|
|
79
80
|
edges: list[GraphEdge] = []
|
|
80
|
-
edges.extend(self._build_entity_edges(new_fact, profile_id))
|
|
81
|
+
edges.extend(self._build_entity_edges(new_fact, profile_id, hub_cache))
|
|
81
82
|
edges.extend(self._build_temporal_edges(new_fact, profile_id))
|
|
82
83
|
edges.extend(self._build_semantic_edges(new_fact, profile_id))
|
|
83
|
-
edges.extend(self._build_causal_edges(new_fact, profile_id))
|
|
84
|
+
edges.extend(self._build_causal_edges(new_fact, profile_id, hub_cache))
|
|
84
85
|
|
|
85
86
|
for edge in edges:
|
|
86
87
|
self._db.store_edge(edge)
|
|
@@ -142,17 +143,33 @@ class GraphBuilder:
|
|
|
142
143
|
|
|
143
144
|
# -- Edge builders (private) -------------------------------------------
|
|
144
145
|
|
|
145
|
-
#
|
|
146
|
-
#
|
|
147
|
-
|
|
148
|
-
|
|
146
|
+
# v3.4.59: Lowered from 20→5. At 17K facts, 20 edges × many entities per fact
|
|
147
|
+
# produces 1.4M+ entity edges and 13s recalls. 5 is sufficient graph signal.
|
|
148
|
+
_MAX_ENTITY_EDGES_PER_ENTITY: int = 5
|
|
149
|
+
# v3.4.59: Hub filter — nodes with > 200 total edges are "gravity wells"
|
|
150
|
+
# (e.g. every fact mentions "SLM"). Skip adding more edges to them.
|
|
151
|
+
_MAX_HUB_DEGREE: int = 200
|
|
152
|
+
|
|
153
|
+
def _node_degree(self, fact_id: str, profile_id: str, cache: dict[str, int]) -> int:
|
|
154
|
+
"""Cached total degree (in + out) for a node."""
|
|
155
|
+
if fact_id not in cache:
|
|
156
|
+
rows = self._db.execute(
|
|
157
|
+
"SELECT COUNT(*) as cnt FROM graph_edges "
|
|
158
|
+
"WHERE profile_id = ? AND (source_id = ? OR target_id = ?)",
|
|
159
|
+
(profile_id, fact_id, fact_id),
|
|
160
|
+
)
|
|
161
|
+
cache[fact_id] = int(dict(rows[0])["cnt"]) if rows else 0
|
|
162
|
+
return cache[fact_id]
|
|
149
163
|
|
|
150
164
|
def _build_entity_edges(
|
|
151
165
|
self, new_fact: AtomicFact, profile_id: str,
|
|
166
|
+
hub_cache: dict[str, int] | None = None,
|
|
152
167
|
) -> list[GraphEdge]:
|
|
153
168
|
"""ENTITY edges: shared canonical entity — capped to most recent per entity."""
|
|
154
169
|
if not new_fact.canonical_entities:
|
|
155
170
|
return []
|
|
171
|
+
if hub_cache is None:
|
|
172
|
+
hub_cache = {}
|
|
156
173
|
edges: list[GraphEdge] = []
|
|
157
174
|
seen: set[str] = set()
|
|
158
175
|
|
|
@@ -163,6 +180,8 @@ class GraphBuilder:
|
|
|
163
180
|
break
|
|
164
181
|
if other.fact_id == new_fact.fact_id or other.fact_id in seen:
|
|
165
182
|
continue
|
|
183
|
+
if self._node_degree(other.fact_id, profile_id, hub_cache) >= self._MAX_HUB_DEGREE:
|
|
184
|
+
continue # skip hub nodes — they link everything to everything
|
|
166
185
|
if self._edge_exists(new_fact.fact_id, other.fact_id, EdgeType.ENTITY, profile_id):
|
|
167
186
|
continue
|
|
168
187
|
seen.add(other.fact_id)
|
|
@@ -261,17 +280,20 @@ class GraphBuilder:
|
|
|
261
280
|
break
|
|
262
281
|
return edges
|
|
263
282
|
|
|
264
|
-
#
|
|
265
|
-
_MAX_CAUSAL_EDGES_PER_ENTITY: int =
|
|
283
|
+
# v3.4.59: Lowered from 20→5, same reasoning as entity cap.
|
|
284
|
+
_MAX_CAUSAL_EDGES_PER_ENTITY: int = 5
|
|
266
285
|
|
|
267
286
|
def _build_causal_edges(
|
|
268
287
|
self, new_fact: AtomicFact, profile_id: str,
|
|
288
|
+
hub_cache: dict[str, int] | None = None,
|
|
269
289
|
) -> list[GraphEdge]:
|
|
270
290
|
"""CAUSAL edges: causal markers + shared entity. Direction: cause -> effect."""
|
|
271
291
|
if not any(p.search(new_fact.content) for p in _CAUSAL_CUES):
|
|
272
292
|
return []
|
|
273
293
|
if not new_fact.canonical_entities:
|
|
274
294
|
return []
|
|
295
|
+
if hub_cache is None:
|
|
296
|
+
hub_cache = {}
|
|
275
297
|
|
|
276
298
|
edges: list[GraphEdge] = []
|
|
277
299
|
seen: set[str] = set()
|
|
@@ -282,6 +304,8 @@ class GraphBuilder:
|
|
|
282
304
|
break
|
|
283
305
|
if other.fact_id == new_fact.fact_id or other.fact_id in seen:
|
|
284
306
|
continue
|
|
307
|
+
if self._node_degree(other.fact_id, profile_id, hub_cache) >= self._MAX_HUB_DEGREE:
|
|
308
|
+
continue
|
|
285
309
|
if self._edge_exists(other.fact_id, new_fact.fact_id, EdgeType.CAUSAL, profile_id):
|
|
286
310
|
continue
|
|
287
311
|
seen.add(other.fact_id)
|
|
@@ -46,7 +46,7 @@ class ReaperConfig:
|
|
|
46
46
|
|
|
47
47
|
enabled: bool = True
|
|
48
48
|
heartbeat_interval_seconds: int = 60
|
|
49
|
-
orphan_age_threshold_hours: float = 4.0
|
|
49
|
+
orphan_age_threshold_hours: float = 1.0 # v3.4.57: reduced from 4.0 — confirmed-dead parent = safe to kill after 1h
|
|
50
50
|
pid_file_path: str = "" # Empty = default (~/.superlocalmemory/slm.pids)
|
|
51
51
|
graceful_timeout_seconds: float = 5.0
|
|
52
52
|
|
|
@@ -147,6 +147,13 @@ def _retrain_ranker_impl(
|
|
|
147
147
|
objective, sorted(_allowed_objectives),
|
|
148
148
|
)
|
|
149
149
|
objective = "lambdarank"
|
|
150
|
+
# SAFE THREAD CAP — v3.4.58 permanent fix for OpenMP multi-library SIGSEGV.
|
|
151
|
+
# See ranker_retrain_online.py for full explanation. Same cap applies here:
|
|
152
|
+
# the legacy path hits the identical LGBM_DatasetCreateFromMat → OpenMP
|
|
153
|
+
# parallel fork. On macOS ARM with torch+sklearn+lightgbm, >2 threads
|
|
154
|
+
# triggers the libomp multi-runtime SIGSEGV in __kmp_suspend_initialize_thread.
|
|
155
|
+
_lgbm_threads_env = os.environ.get("SLM_LGBM_THREADS", "").strip()
|
|
156
|
+
_lgbm_num_threads = int(_lgbm_threads_env) if _lgbm_threads_env.isdigit() else 2
|
|
150
157
|
params = {
|
|
151
158
|
"objective": objective,
|
|
152
159
|
"metric": "ndcg",
|
|
@@ -156,8 +163,9 @@ def _retrain_ranker_impl(
|
|
|
156
163
|
"num_leaves": 31,
|
|
157
164
|
"min_data_in_leaf": 20,
|
|
158
165
|
"verbosity": -1,
|
|
159
|
-
"num_threads":
|
|
166
|
+
"num_threads": _lgbm_num_threads,
|
|
160
167
|
}
|
|
168
|
+
|
|
161
169
|
try:
|
|
162
170
|
booster_new = lgb.train(params, ds_train, num_boost_round=50)
|
|
163
171
|
except lgb.basic.LightGBMError as exc:
|
|
@@ -195,7 +195,17 @@ def _train_booster(
|
|
|
195
195
|
params = dict(RETRAIN_HYPERPARAM_CAPS)
|
|
196
196
|
params["objective"] = objective
|
|
197
197
|
params["label_gain"] = gain
|
|
198
|
-
|
|
198
|
+
# SAFE THREAD CAP — v3.4.58 permanent fix for OpenMP multi-library SIGSEGV.
|
|
199
|
+
# Using os.cpu_count()-1 (e.g. 9 on M4 Mac) triggers a race in LightGBM's
|
|
200
|
+
# DatasetLoader::ConstructFromSampleData when PyTorch's libomp and
|
|
201
|
+
# LightGBM's libomp are both loaded. The parallel OpenMP fork crosses
|
|
202
|
+
# libomp runtime boundaries → SIGSEGV in __kmp_suspend_initialize_thread.
|
|
203
|
+
# 2 threads avoids the crash path and is sufficient for SLM's small datasets.
|
|
204
|
+
# Override with SLM_LGBM_THREADS env var if you need more (and have a
|
|
205
|
+
# single-runtime environment with libomp unified).
|
|
206
|
+
_lgbm_threads_env = os.environ.get("SLM_LGBM_THREADS", "").strip()
|
|
207
|
+
params["num_threads"] = int(_lgbm_threads_env) if _lgbm_threads_env.isdigit() else 2
|
|
208
|
+
|
|
199
209
|
num_boost_round = int(params.pop("num_boost_round"))
|
|
200
210
|
|
|
201
211
|
start = time.monotonic()
|
|
@@ -36,7 +36,7 @@ class DaemonPoolProxy:
|
|
|
36
36
|
envelopes — the adapter is responsible for surfacing those.
|
|
37
37
|
"""
|
|
38
38
|
|
|
39
|
-
def __init__(self, port: int, *, timeout_s: float =
|
|
39
|
+
def __init__(self, port: int, *, timeout_s: float = 30.0) -> None: # v3.4.59: 8s→30s — observed recall takes 13.4s on dense graph (2.1M edges); 8s always timed out → degraded mode
|
|
40
40
|
self._port = port
|
|
41
41
|
self._timeout = timeout_s
|
|
42
42
|
|
|
@@ -241,5 +241,42 @@ _warmup_thread = threading.Thread(target=_eager_warmup, daemon=True, name="mcp-w
|
|
|
241
241
|
_warmup_thread.start()
|
|
242
242
|
|
|
243
243
|
|
|
244
|
+
# V3.4.57: Parent watchdog — self-terminate when the IDE/Claude session dies.
|
|
245
|
+
# FastMCP relies on stdin EOF to stop, but stdin EOF is NOT guaranteed on
|
|
246
|
+
# crash or force-quit. Without this, every abnormal exit leaves an orphaned
|
|
247
|
+
# slm mcp process consuming ~100-200 MB indefinitely. 22 orphans caused a
|
|
248
|
+
# daemon deadlock on May 30 2026 (241% CPU, session_init timeouts).
|
|
249
|
+
def _parent_watchdog() -> None:
|
|
250
|
+
"""Exit when parent IDE process (Claude Code, Cursor, etc.) dies.
|
|
251
|
+
|
|
252
|
+
Polls os.getppid() every 10 seconds. On macOS/Linux, when a parent
|
|
253
|
+
dies the child is reparented to PID 1 (init) — getppid() returns 1.
|
|
254
|
+
Also validates the original parent is still alive via os.kill(ppid, 0).
|
|
255
|
+
Uses os._exit(0) to bypass any atexit handlers that might hang.
|
|
256
|
+
"""
|
|
257
|
+
import os as _os_wd, time as _time
|
|
258
|
+
_wlog = logging.getLogger(__name__ + ".watchdog")
|
|
259
|
+
initial_ppid = _os_wd.getppid()
|
|
260
|
+
if initial_ppid <= 1:
|
|
261
|
+
return # Already reparented at startup — don't self-terminate
|
|
262
|
+
while True:
|
|
263
|
+
_time.sleep(10)
|
|
264
|
+
try:
|
|
265
|
+
current_ppid = _os_wd.getppid()
|
|
266
|
+
if current_ppid != initial_ppid or current_ppid <= 1:
|
|
267
|
+
_wlog.info("Parent PID changed (%d→%d), self-terminating", initial_ppid, current_ppid)
|
|
268
|
+
_os_wd._exit(0)
|
|
269
|
+
_os_wd.kill(initial_ppid, 0) # Raises ProcessLookupError if dead
|
|
270
|
+
except ProcessLookupError:
|
|
271
|
+
_wlog.info("Parent PID %d gone, self-terminating", initial_ppid)
|
|
272
|
+
_os_wd._exit(0)
|
|
273
|
+
except Exception:
|
|
274
|
+
pass # Transient errors — keep watching
|
|
275
|
+
|
|
276
|
+
|
|
277
|
+
_watchdog_thread = threading.Thread(target=_parent_watchdog, daemon=True, name="parent-watchdog")
|
|
278
|
+
_watchdog_thread.start()
|
|
279
|
+
|
|
280
|
+
|
|
244
281
|
if __name__ == "__main__":
|
|
245
282
|
server.run(transport="stdio")
|
|
@@ -234,27 +234,47 @@ class SpreadingActivation:
|
|
|
234
234
|
the highest-signal edges.
|
|
235
235
|
"""
|
|
236
236
|
try:
|
|
237
|
+
# v3.4.59: LIMIT pushed inside each UNION branch so SQLite can use
|
|
238
|
+
# idx_edges_source_weight / idx_edges_target_weight and stop after
|
|
239
|
+
# max_neighbors_per_node rows per branch instead of materializing
|
|
240
|
+
# all 2.1M edges then sorting. Each branch wrapped in SELECT * FROM (...)
|
|
241
|
+
# because SQLite requires parentheses for ORDER BY+LIMIT in compound SELECTs.
|
|
242
|
+
lim = self._config.max_neighbors_per_node
|
|
237
243
|
rows = self._db.execute(
|
|
238
244
|
"""
|
|
239
245
|
SELECT neighbor_id, weight FROM (
|
|
240
|
-
SELECT
|
|
246
|
+
SELECT * FROM (
|
|
247
|
+
SELECT target_id AS neighbor_id, weight FROM graph_edges
|
|
241
248
|
WHERE source_id = ? AND profile_id = ?
|
|
249
|
+
ORDER BY weight DESC LIMIT ?
|
|
250
|
+
)
|
|
242
251
|
UNION ALL
|
|
243
|
-
SELECT
|
|
252
|
+
SELECT * FROM (
|
|
253
|
+
SELECT target_fact_id AS neighbor_id, weight FROM association_edges
|
|
244
254
|
WHERE source_fact_id = ? AND profile_id = ?
|
|
255
|
+
ORDER BY weight DESC LIMIT ?
|
|
256
|
+
)
|
|
245
257
|
UNION ALL
|
|
246
|
-
SELECT
|
|
258
|
+
SELECT * FROM (
|
|
259
|
+
SELECT source_id AS neighbor_id, weight FROM graph_edges
|
|
247
260
|
WHERE target_id = ? AND profile_id = ?
|
|
261
|
+
ORDER BY weight DESC LIMIT ?
|
|
262
|
+
)
|
|
248
263
|
UNION ALL
|
|
249
|
-
SELECT
|
|
264
|
+
SELECT * FROM (
|
|
265
|
+
SELECT source_fact_id AS neighbor_id, weight FROM association_edges
|
|
250
266
|
WHERE target_fact_id = ? AND profile_id = ?
|
|
267
|
+
ORDER BY weight DESC LIMIT ?
|
|
268
|
+
)
|
|
251
269
|
)
|
|
252
270
|
ORDER BY weight DESC
|
|
253
271
|
LIMIT ?
|
|
254
272
|
""",
|
|
255
|
-
(node_id, profile_id,
|
|
256
|
-
node_id, profile_id,
|
|
257
|
-
|
|
273
|
+
(node_id, profile_id, lim,
|
|
274
|
+
node_id, profile_id, lim,
|
|
275
|
+
node_id, profile_id, lim,
|
|
276
|
+
node_id, profile_id, lim,
|
|
277
|
+
lim),
|
|
258
278
|
)
|
|
259
279
|
return [
|
|
260
280
|
(dict(r)["neighbor_id"], dict(r)["weight"]) for r in rows
|