superlocalmemory 3.5.5 → 3.5.6

package/CHANGELOG.md

@@ -5,6 +5,37 @@ All notable changes to SuperLocalMemory V3 will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [3.5.6] - 2026-06-03 — Isolate LightGBM training (macOS daemon SIGSEGV fix)
+
+### Fixed (CRITICAL — daemon hard-crash on macOS / Apple Silicon)
+- **`POST /api/learning/retrain`** ("Train model now" in the dashboard Brain pane)
+  and **`POST /api/v3/learning/consolidate`** no longer crash the unified daemon.
+  The daemon serves the API in-process with PyTorch's OpenMP runtime already warm
+  (reranker + embedding warm-up); importing `lightgbm` in that same process loaded
+  a second `libomp.dylib`, corrupting shared `__kmp` state and segfaulting a worker
+  thread (`SIGSEGV`, no Python traceback — the process died on a native signal and
+  auto-restarted).
+- **Fix:** all LightGBM training now runs in an isolated subprocess
+  (`learning/lightgbm_subprocess.py`) that imports `lightgbm` **before** the
+  `superlocalmemory` package, so torch's OMP pool stays dormant and only LightGBM's
+  runtime is active. The child emits a single JSON verdict; the parent never raises,
+  so a native child crash is reported as an error and the daemon stays up. Mirrors
+  the existing `embedding_worker` / `reranker_worker` isolation pattern.
+- On success the daemon invalidates the model cache so the freshly trained model is
+  reloaded. The fix applies on all platforms (subprocess spawn cost only off macOS).
+- Thanks to @barrygfox for the detailed report, repro, and fix (#27).
+
+### Added
+- **`SLM_HOST` env var** — shorter alias for `SLM_DAEMON_HOST` to set the daemon
+  bind address (issue #23). Set `SLM_HOST=0.0.0.0` (or `SLM_DAEMON_HOST=0.0.0.0`)
+  to serve one shared instance across a trusted private network; pair with
+  `SLM_MESH_HOST=0.0.0.0` + `SLM_MESH_SHARED_SECRET` for the mesh broker.
+  `SLM_DAEMON_HOST` takes precedence when both are set.
+
+### Changed
+- CI: publish workflows now fail fast if `package.json`, `pyproject.toml`, and the
+  pushed `v*` tag disagree — prevents shipping mismatched npm/PyPI versions.
+
 ## [3.5.5] - 2026-05-31 — Write-Through Remember (instant cross-session recall)
 
 ### Fixed (CRITICAL — closes the remember→recall window)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "superlocalmemory",
-  "version": "3.5.5",
+  "version": "3.5.6",
   "description": "Information-geometric agent memory with mathematical guarantees. 4-channel retrieval, Fisher-Rao similarity, zero-LLM mode, EU AI Act compliant. Works with Claude, Cursor, Windsurf, and 17+ AI tools.",
   "keywords": [
     "ai-memory",

package/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "superlocalmemory"
-version = "3.5.5"
+version = "3.5.6"
 description = "Information-geometric agent memory with mathematical guarantees"
 readme = "README.md"
 license = {text = "AGPL-3.0-or-later"}

package/src/superlocalmemory/learning/lightgbm_subprocess.py

@@ -0,0 +1,236 @@
+# Copyright (c) 2026 Varun Pratap Bhardwaj / Qualixar
+# Licensed under AGPL-3.0-or-later - see LICENSE file
+# Part of SuperLocalMemory V3 | https://qualixar.com | https://varunpratap.com
+
+"""Isolated LightGBM work (retrain + consolidation) — OpenMP-crash guard.
+
+**Why this exists** — the unified daemon serves the HTTP API in-process and,
+by then, has already loaded PyTorch's OpenMP runtime (``torch/lib/libomp.dylib``)
+with *warm* worker threads (the cross-encoder reranker + embedding warm-up at
+startup). Any code path that imports ``lightgbm`` in that same process loads a
+*second* OpenMP runtime — the Homebrew ``libomp.dylib`` that
+``lib_lightgbm.dylib`` links against — which corrupts the shared ``__kmp``
+global state and SIGSEGVs a pre-existing torch OMP worker thread. That
+hard-crashes the whole daemon (observed on macOS / Apple Silicon;
+``DiagnosticReports/Python-*.ips`` shows ``libomp.dylib __kmp_launch_worker``).
+
+Two in-daemon entry points reach LightGBM training:
+  * ``POST /api/learning/retrain``  → legacy ``_retrain_ranker_impl``;
+  * ``POST /api/v3/learning/consolidate`` → ``ConsolidationWorker.run`` whose
+    step 5 trains via the online ``_run_shadow_cycle`` (active model) or the
+    legacy cold-start path.
+
+Both are funnelled through this module so LightGBM only ever trains in a
+**fresh subprocess** that never runs a torch tensor op — torch's OMP pool stays
+dormant there and lightgbm's runtime is the only active one. The mechanism
+mirrors the existing ``embedding_worker`` / ``reranker_worker`` isolation.
+
+Spawned via :func:`run_retrain_isolated` / :func:`run_consolidation_isolated`,
+which use a ``python -c`` bootstrap that imports lightgbm BEFORE the
+``superlocalmemory`` package. That ordering is load-bearing: importing the
+package transitively pulls in torch, and torch-OMP-first-then-lightgbm is
+exactly the sequence that segfaults. A plain ``python -m
+superlocalmemory.learning.lightgbm_subprocess`` would run the package
+``__init__`` (torch) first and crash — do not invoke it that way.
+
+The child emits a single JSON line on stdout — ``{"error": str|null, ...}`` —
+plus a task payload (``trained`` for retrain, ``stats`` for consolidate).
+Exit 0 = ran to completion; non-zero = handled failure. Either way the parent
+(the daemon) keeps running.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import logging
+import subprocess
+import sys
+from pathlib import Path
+
+logger = logging.getLogger(__name__)
+
+# Wall-clock ceilings. A retrain is 50 boosting rounds on ≤2000 rows (seconds);
+# a full consolidation cycle also decays/dedups/mines/graph-analyses, so it
+# gets more headroom.
+RETRAIN_TIMEOUT_SEC = 300
+CONSOLIDATE_TIMEOUT_SEC = 600
+
+
+# ---------------------------------------------------------------------------
+# Child-side task implementations (run in the isolated subprocess).
+# ---------------------------------------------------------------------------
+
+def _retrain(learning_db: str, profile_id: str, *, include_synthetic: bool) -> bool:
+    """Legacy ranker retrain in this (lightgbm-first) process. Returns trained."""
+    import lightgbm  # noqa: F401  (import-order side effect is intentional)
+
+    from superlocalmemory.learning.ranker_retrain_legacy import (
+        _retrain_ranker_impl,
+    )
+
+    return bool(
+        _retrain_ranker_impl(
+            learning_db, profile_id, include_synthetic=include_synthetic,
+        )
+    )
+
+
+def _consolidate(
+    memory_db: str, learning_db: str, profile_id: str, *, dry_run: bool,
+) -> dict:
+    """Full consolidation cycle (incl. step-5 LightGBM training). Returns stats."""
+    import lightgbm  # noqa: F401  (import-order side effect is intentional)
+
+    from superlocalmemory.learning.consolidation_worker import (
+        ConsolidationWorker,
+    )
+
+    worker = ConsolidationWorker(memory_db, learning_db)
+    return worker.run(profile_id, dry_run=dry_run)
+
+
+def _emit(obj: dict) -> None:
+    """Write the verdict as a single JSON line (default=str for safety)."""
+    sys.stdout.write(json.dumps(obj, default=str) + "\n")
+    sys.stdout.flush()
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(prog="slm-lightgbm-subprocess")
+    parser.add_argument("--task", choices=["retrain", "consolidate"], default="retrain")
+    parser.add_argument("--learning-db", required=True)
+    parser.add_argument("--memory-db")
+    parser.add_argument("--profile", default="default")
+    parser.add_argument("--include-synthetic", action="store_true")
+    parser.add_argument("--dry-run", action="store_true")
+    args = parser.parse_args(argv)
+
+    result: dict = {"error": None}
+    try:
+        if args.task == "retrain":
+            result["trained"] = _retrain(
+                args.learning_db, args.profile,
+                include_synthetic=args.include_synthetic,
+            )
+        else:
+            if not args.memory_db:
+                raise ValueError("--memory-db is required for the consolidate task")
+            result["stats"] = _consolidate(
+                args.memory_db, args.learning_db, args.profile,
+                dry_run=args.dry_run,
+            )
+    except Exception as exc:  # noqa: BLE001
+        result["error"] = f"{type(exc).__name__}: {exc}"
+        _emit(result)
+        return 1
+
+    _emit(result)
+    return 0
+
+
+# ---------------------------------------------------------------------------
+# Caller-side helpers — spawn the child in an isolated process.
+# ---------------------------------------------------------------------------
+
+def _run_isolated(task_args: list[str], *, timeout_sec: int) -> dict:
+    """Spawn the child with lightgbm imported first; return its JSON verdict.
+
+    Never raises for the expected failure modes (non-zero exit, timeout,
+    native crash with no JSON) — encodes them in an ``error`` key so the
+    calling daemon stays up no matter what.
+    """
+    # The leading ``import lightgbm`` is load-bearing: it must run BEFORE the
+    # superlocalmemory package (which transitively imports torch). ``-c`` is
+    # the only way to guarantee that ordering — ``-m superlocalmemory.…`` runs
+    # the package __init__ (torch) first and would reintroduce the crash.
+    bootstrap = (
+        "import lightgbm; "
+        "from superlocalmemory.learning.lightgbm_subprocess import main; "
+        "raise SystemExit(main())"
+    )
+    cmd = [sys.executable, "-c", bootstrap, *task_args]
+
+    try:
+        proc = subprocess.run(
+            cmd,
+            capture_output=True,
+            text=True,
+            timeout=timeout_sec,
+            check=False,
+        )
+    except subprocess.TimeoutExpired:
+        return {"error": f"timed out after {timeout_sec}s"}
+    except Exception as exc:  # noqa: BLE001
+        return {"error": f"spawn failed: {exc}"}
+
+    # The verdict is the last JSON line on stdout. Parse defensively — a native
+    # crash in the child would leave no JSON, which we surface plainly.
+    for line in reversed((proc.stdout or "").splitlines()):
+        line = line.strip()
+        if not line:
+            continue
+        try:
+            return json.loads(line)
+        except ValueError:
+            continue
+
+    tail = (proc.stderr or "").strip()[-500:]
+    return {
+        "error": (
+            f"subprocess produced no verdict (exit={proc.returncode}). "
+            f"stderr tail: {tail}"
+        ),
+    }
+
+
+def run_retrain_isolated(
+    learning_db: str | Path,
+    profile_id: str,
+    *,
+    include_synthetic: bool = False,
+    timeout_sec: int = RETRAIN_TIMEOUT_SEC,
+) -> dict:
+    """Train the ranker in a subprocess. Returns ``{"trained": bool, "error": ...}``."""
+    args = [
+        "--task", "retrain",
+        "--learning-db", str(learning_db),
+        "--profile", profile_id,
+    ]
+    if include_synthetic:
+        args.append("--include-synthetic")
+    verdict = _run_isolated(args, timeout_sec=timeout_sec)
+    verdict.setdefault("trained", False)
+    verdict.setdefault("error", None)
+    return verdict
+
+
+def run_consolidation_isolated(
+    memory_db: str | Path,
+    learning_db: str | Path,
+    profile_id: str,
+    *,
+    dry_run: bool = False,
+    timeout_sec: int = CONSOLIDATE_TIMEOUT_SEC,
+) -> dict:
+    """Run the full consolidation cycle in a subprocess (its step-5 training
+    would otherwise crash the torch-warm daemon).
+
+    Returns ``{"stats": dict|None, "error": str|None}``.
+    """
+    args = [
+        "--task", "consolidate",
+        "--memory-db", str(memory_db),
+        "--learning-db", str(learning_db),
+        "--profile", profile_id,
+    ]
+    if dry_run:
+        args.append("--dry-run")
+    verdict = _run_isolated(args, timeout_sec=timeout_sec)
+    verdict.setdefault("stats", None)
+    verdict.setdefault("error", None)
+    return verdict
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

package/src/superlocalmemory/server/routes/learning.py

@@ -15,6 +15,7 @@ from datetime import datetime, timezone
 from pathlib import Path
 
 from fastapi import APIRouter
+from fastapi.concurrency import run_in_threadpool
 
 from .helpers import get_active_profile, MEMORY_DIR
 
@@ -597,16 +598,31 @@ async def learning_retrain(data: dict | None = None):
         data and data.get("include_synthetic")
     ) if isinstance(data, dict) else False
     try:
-        from superlocalmemory.learning.consolidation_worker import (
-            _retrain_ranker_impl,
+        # Train OUT OF PROCESS. The daemon already has torch's OpenMP runtime
+        # loaded with warm worker threads; importing lightgbm in-process loads
+        # a second libomp and SIGSEGVs the whole daemon (see
+        # retrain_subprocess module docstring). Isolation is the fix.
+        from superlocalmemory.learning.lightgbm_subprocess import (
+            run_retrain_isolated,
         )
         profile_id = get_active_profile() or "default"
-        trained = _retrain_ranker_impl(
+        result = await run_in_threadpool(
+            run_retrain_isolated,
             LEARNING_DB,
             profile_id,
             include_synthetic=include_synthetic,
         )
-        if trained:
+        if result.get("error"):
+            logger.error("learning_retrain failed: %s", result["error"])
+            return {"success": False, "error": result["error"]}
+        if result.get("trained"):
+            # Drop the cached model so the next recall reloads the freshly
+            # trained one the subprocess just persisted to learning.db.
+            try:
+                from superlocalmemory.learning.model_cache import invalidate
+                invalidate(profile_id)
+            except Exception as exc:  # pragma: no cover — defensive
+                logger.debug("model cache invalidate failed: %s", exc)
             return {
                 "success": True,
                 "trained": True,

package/src/superlocalmemory/server/routes/v3_api.py

@@ -806,15 +806,37 @@ async def run_consolidation(request: Request):
     try:
         body = await request.json()
         dry_run = body.get("dry_run", False)
-        from superlocalmemory.learning.consolidation_worker import ConsolidationWorker
+        # Run the full cycle OUT OF PROCESS. Step 5 trains the LightGBM ranker
+        # (online _run_shadow_cycle or legacy cold-start); importing lightgbm
+        # in the torch-warm daemon loads a second libomp and SIGSEGVs it (see
+        # lightgbm_subprocess module docstring). Isolation is the fix.
+        from superlocalmemory.learning.lightgbm_subprocess import (
+            run_consolidation_isolated,
+        )
         from superlocalmemory.core.config import SLMConfig
         from superlocalmemory.server.routes.helpers import DB_PATH
-        worker = ConsolidationWorker(
-            memory_db=str(DB_PATH),
-            learning_db=str(DB_PATH.parent / "learning.db"),
-        )
+        from fastapi.concurrency import run_in_threadpool
+
         config = SLMConfig.load()
-        stats = worker.run(config.active_profile, dry_run=dry_run)
+        learning_db = DB_PATH.parent / "learning.db"
+        result = await run_in_threadpool(
+            run_consolidation_isolated,
+            str(DB_PATH),
+            str(learning_db),
+            config.active_profile,
+            dry_run=dry_run,
+        )
+        if result.get("error"):
+            return {"success": False, "error": result["error"]}
+        # Step-5 training may have promoted a new model — drop the daemon's
+        # cached model so the next recall reloads it from learning.db.
+        if not dry_run:
+            try:
+                from superlocalmemory.learning.model_cache import invalidate
+                invalidate(config.active_profile)
+            except Exception:
+                pass
+        stats = result.get("stats") or {}
         return {"success": True, **stats}
     except Exception as exc:
         return {"success": False, "error": str(exc)}

package/src/superlocalmemory/server/unified_daemon.py

@@ -1950,10 +1950,19 @@ def start_server(port: int = _DEFAULT_PORT) -> None:
     log_dir = Path.home() / ".superlocalmemory" / "logs"
     log_dir.mkdir(parents=True, exist_ok=True)
 
+    # Bind address. `SLM_DAEMON_HOST` is the canonical name; `SLM_HOST` is
+    # accepted as a shorter alias (issue #23). Set either to 0.0.0.0 to serve
+    # a shared instance over a trusted private network (e.g. WireGuard mesh).
+    bind_host = (
+        os.environ.get("SLM_DAEMON_HOST")
+        or os.environ.get("SLM_HOST")
+        or "127.0.0.1"
+    )
+
     config = uvicorn.Config(
         app="superlocalmemory.server.unified_daemon:create_app",
         factory=True,
-        host=os.environ.get("SLM_DAEMON_HOST", "127.0.0.1"),
+        host=bind_host,
         port=port,
         log_level="warning",
         timeout_graceful_shutdown=10,

package/src/superlocalmemory.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: superlocalmemory
-Version: 3.5.5
+Version: 3.5.6
 Summary: Information-geometric agent memory with mathematical guarantees
 Author-email: Varun Pratap Bhardwaj <admin@superlocalmemory.com>
 License: AGPL-3.0-or-later

package/src/superlocalmemory.egg-info/SOURCES.txt

@@ -221,6 +221,7 @@ src/superlocalmemory/learning/forgetting_scheduler.py
 src/superlocalmemory/learning/hnsw_dedup.py
 src/superlocalmemory/learning/labeler.py
 src/superlocalmemory/learning/legacy_migration.py
+src/superlocalmemory/learning/lightgbm_subprocess.py
 src/superlocalmemory/learning/memory_merge.py
 src/superlocalmemory/learning/model_cache.py
 src/superlocalmemory/learning/model_rollback.py