superlocalmemory 3.6.7 → 3.6.8

package/CHANGELOG.md

@@ -5,6 +5,21 @@ All notable changes to SuperLocalMemory V3 will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [3.6.8] - 2026-06-11 — Runtime recall-health monitor (self-healing recall)
+
+### Fixed
+- **Silent recall degradation eliminated.** On a long-running daemon the cold full-fusion recall could exceed the MCP pool's 30s timeout, so `session_init` silently fell back to FTS5/BM25 keyword-only ("DEGRADED MODE") — observed 7× in 2 days in production logs. Two root causes are now repaired in-life rather than only at boot:
+  - **Page-cache eviction:** the graph/`association_edges` table gets evicted under memory pressure, so the first recall after idle re-reads it from disk (15–24s+).
+  - **Warm-but-broken embedder:** `OllamaEmbedder.embed` returns `None` on a transient Ollama failure while the boot `_embedding_warm` flag still reports `True`; with `q_emb is None` the engine skips the semantic, hopfield and spreading_activation channels, silently degrading to keyword-only (semantic score `0.0` on every result).
+
+### Added
+- **`server/recall_health.py` — runtime recall-health monitor** (industry-standard 3-tier, validated against Ollama keep-alive, Chroma's active heartbeat, LangChain's circuit breaker and the K8s liveness/readiness split):
+  - **Tier 1 — re-warm:** fires a real `engine.recall` every 5 min to keep the graph page cache hot and nomic-embed resident.
+  - **Tier 2 — readiness probe:** asserts the semantic channel actually fired (`max semantic > 0`); rows-with-`semantic==0` everywhere is the warm-but-broken signature.
+  - **Tier 3 — circuit-breaker self-heal:** resets the embedder's cached `_available` flag (the "available once, cached forever" bug), re-exercises `embed()`, tracks consecutive failures, and logs **CRITICAL** so degradation is never silent.
+- **`/health` now reports `recall_health`** (`recall_healthy`, `consecutive_failures`, `total_heals`, `checks`, `last_semantic_score`) — degradation is visible to operators, not hidden.
+- 8 new tests in `tests/test_core/test_recall_health.py`.
+
 ## [3.6.7] - 2026-06-10 — MCP Streamable-HTTP Transport (Embedded)
 
 ### Added

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "superlocalmemory",
-  "version": "3.6.7",
+  "version": "3.6.8",
   "description": "Information-geometric agent memory with mathematical guarantees. 4-channel retrieval, Fisher-Rao similarity, zero-LLM mode, EU AI Act compliant. Works with Claude, Cursor, Windsurf, and 17+ AI tools.",
   "keywords": [
     "ai-memory",

package/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "superlocalmemory"
-version = "3.6.7"
+version = "3.6.8"
 description = "Information-geometric agent memory with mathematical guarantees"
 readme = "README.md"
 license = {text = "AGPL-3.0-or-later"}

package/src/superlocalmemory/__init__.py

@@ -28,7 +28,7 @@ if "OMP_NUM_THREADS" not in os.environ:
     os.environ["OMP_NUM_THREADS"] = "2"
 # ---------------------------------------------------------------------------
 
-__version__ = "3.6.7"
+__version__ = "3.6.8"
 
 _REQUIRED_VERSIONS = {
     "sentence_transformers": "5.3.0",

package/src/superlocalmemory/server/recall_health.py

@@ -0,0 +1,237 @@
+"""Runtime recall-health monitor — keep the 6-channel recall path warm,
+detect a "warm-but-broken" embedder at runtime, and self-heal it (v3.6.8).
+
+Why this exists
+---------------
+On a long-running daemon the cold full-fusion recall could exceed the MCP
+pool's 30s timeout, and ``session_init`` then silently fell back to FTS5/BM25
+("DEGRADED MODE") — observed 7×/2 days in production logs. Two root causes:
+
+1. **Page-cache eviction.** The ~100 MB ``association_edges`` / graph table is
+   evicted under memory pressure, so the first recall after idle re-reads it
+   from disk (15-24 s+), blowing the pool timeout.
+2. **Silent embedder death.** ``OllamaEmbedder.embed`` returns ``None`` on a
+   transient Ollama failure, while the boot-set ``_embedding_warm`` flag still
+   reports ``True``. With ``q_emb is None`` the engine skips the semantic,
+   hopfield and spreading_activation channels — recall silently degrades to
+   keyword-only (``semantic`` score ``0.0`` on every result).
+
+The boot warmup (``_warmup_recall``) runs ONCE and never again, so neither
+condition is repaired in-life.
+
+The fix — an industry-standard 3-tier monitor, validated against Ollama
+keep-alive, Chroma's active heartbeat, LangChain's circuit breaker and the
+Kubernetes liveness/readiness split:
+
+* **Tier 1 — RE-WARM.** Fire a real ``engine.recall`` every ``interval_s`` to
+  keep the graph page cache hot and nomic-embed resident.
+* **Tier 2 — READINESS PROBE.** Assert the semantic channel actually fired
+  (``max semantic > 0``). Rows returned with ``semantic == 0`` everywhere is the
+  warm-but-broken signature → the embedder is returning ``None``.
+* **Tier 3 — CIRCUIT-BREAKER SELF-HEAL.** Reset the embedder's cached
+  ``_available`` flag (the "available once, cached forever" bug), re-exercise
+  ``embed()``, track consecutive failures, and log **CRITICAL** so the
+  degradation is never silent.
+
+All logic is fail-soft: a crash in the monitor never takes down the daemon.
+"""
+from __future__ import annotations
+
+import logging
+import threading
+from dataclasses import dataclass
+
+logger = logging.getLogger(__name__)
+
+# A stable probe query. Content-agnostic: we only assert that *some* result
+# carries a non-zero semantic score, which proves the embedder + semantic
+# channel are alive end-to-end.
+DEFAULT_PROBE = "memory recall health probe"
+# A distinct string for the heal embed() call so it is obvious in thread dumps
+# / Ollama logs and never collides with a real cached query embedding.
+HEAL_PROBE = "__recall_health_rewarm__"
+
+# Default cadence. 5 min keeps nomic-embed resident (Ollama default unload is
+# 5 min) and the page cache warm without meaningful load.
+DEFAULT_INTERVAL_S = 300
+
+
+@dataclass
+class RecallHealth:
+    """Mutable health state for the recall path."""
+
+    healthy: bool = True
+    consecutive_failures: int = 0
+    total_heals: int = 0
+    checks: int = 0
+    last_semantic_score: float = 0.0
+    last_error: str = ""
+
+
+def _max_semantic(results) -> float:
+    """Largest semantic channel score across results (0.0 if none)."""
+    best = 0.0
+    for r in results:
+        cs = getattr(r, "channel_scores", None) or {}
+        try:
+            best = max(best, float(cs.get("semantic", 0.0) or 0.0))
+        except (TypeError, ValueError):
+            continue
+    return best
+
+
+def _get_embedder(engine):
+    """Locate the engine's embedder (full mode: engine._embedder; some paths
+    hang it off the retrieval engine)."""
+    emb = getattr(engine, "_embedder", None)
+    if emb is None:
+        re_eng = getattr(engine, "_retrieval_engine", None)
+        emb = getattr(re_eng, "_embedder", None) if re_eng is not None else None
+    return emb
+
+
+def _heal_embedder(engine, *, log) -> bool:
+    """Tier 3: reset the cached availability flag and re-exercise the embedder.
+
+    ``OllamaEmbedder.is_available`` caches its first result forever, so once
+    Ollama blips the flag can stay stale. Clearing ``_available`` forces a
+    re-probe; ``embed()`` itself always re-attempts the HTTP call, so a single
+    successful embed proves recovery. Returns True iff the embedder produced a
+    vector.
+    """
+    emb = _get_embedder(engine)
+    if emb is None:
+        log.critical("recall-health: no embedder on engine — cannot self-heal")
+        return False
+    # Reset the "available once, cached forever" flag if present.
+    if hasattr(emb, "_available"):
+        try:
+            emb._available = None
+        except Exception:  # pragma: no cover - defensive
+            pass
+    try:
+        vec = emb.embed(HEAL_PROBE)
+    except Exception as exc:
+        log.warning("recall-health: heal embed() raised: %s", exc)
+        return False
+    return vec is not None and bool(vec)
+
+
+def run_health_tick(engine, state: RecallHealth, *, probe: str = DEFAULT_PROBE,
+                    log=logger) -> RecallHealth:
+    """One monitor tick: re-warm (Tier 1), probe (Tier 2), self-heal (Tier 3).
+
+    Mutates and returns ``state``. Never raises — a timed-out / failing recall
+    marks the path unhealthy instead of propagating.
+    """
+    state.checks += 1
+
+    # Tier 1: re-warm. A real full-fusion recall keeps the graph page cache hot
+    # and the embedder resident.
+    try:
+        resp = engine.recall(probe, limit=3, fast=False)
+    except Exception as exc:
+        state.healthy = False
+        state.consecutive_failures += 1
+        state.last_error = f"recall raised: {exc}"
+        log.critical(
+            "recall-health: re-warm recall FAILED (%s) — recall path unhealthy",
+            exc,
+        )
+        return state
+
+    results = list(getattr(resp, "results", []) or [])
+    sem = _max_semantic(results)
+    state.last_semantic_score = sem
+
+    # Tier 2: readiness. Rows present but semantic never fired == warm-but-broken.
+    # Zero results is NOT this signature (could be an empty/filtered corpus).
+    broken = bool(results) and sem <= 0.0
+    if not broken:
+        if not state.healthy:
+            log.warning(
+                "recall-health: RECOVERED (semantic=%.3f, %d results)",
+                sem, len(results),
+            )
+        state.healthy = True
+        state.consecutive_failures = 0
+        state.last_error = ""
+        return state
+
+    # Tier 3: self-heal.
+    log.critical(
+        "recall-health: semantic channel DEAD (%d results, max semantic=0.0) "
+        "— embedder returning None; attempting self-heal",
+        len(results),
+    )
+    if _heal_embedder(engine, log=log):
+        state.total_heals += 1
+        state.healthy = True
+        state.consecutive_failures = 0
+        state.last_error = ""
+        log.warning("recall-health: embedder self-heal SUCCEEDED (re-warmed)")
+    else:
+        state.healthy = False
+        state.consecutive_failures += 1
+        state.last_error = "semantic channel dead; embedder heal failed"
+        log.critical(
+            "recall-health: self-heal FAILED — recall DEGRADED to keyword-only "
+            "(consecutive_failures=%d)", state.consecutive_failures,
+        )
+    return state
+
+
+def health_monitor_loop(engine, *, interval_s: int, stop_event: threading.Event,
+                        state: RecallHealth, probe: str = DEFAULT_PROBE,
+                        log=logger) -> None:
+    """Background loop. Sleeps ``interval_s`` between ticks; exits promptly when
+    ``stop_event`` is set. An initial short delay avoids racing boot warmup."""
+    # Initial delay (bounded) so we don't pile onto the boot warmup threads.
+    if stop_event.wait(min(interval_s, 60)):
+        return
+    while not stop_event.is_set():
+        try:
+            run_health_tick(engine, state, probe=probe, log=log)
+        except Exception as exc:  # pragma: no cover - belt & suspenders
+            log.warning("recall-health: tick crashed (non-fatal): %s", exc)
+        if stop_event.wait(interval_s):
+            break
+
+
+# Module-level state so /health can surface the latest verdict without holding
+# a reference to the thread.
+_GLOBAL_STATE = RecallHealth()
+
+
+def start_recall_health_monitor(engine, *, interval_s: int = DEFAULT_INTERVAL_S,
+                                probe: str = DEFAULT_PROBE, log=None):
+    """Start the monitor as a daemon thread. Returns ``(thread, stop_event,
+    state)``. The state is the shared module-level state read by
+    :func:`get_recall_health`."""
+    log = log or logger
+    state = _GLOBAL_STATE
+    stop = threading.Event()
+    t = threading.Thread(
+        target=health_monitor_loop,
+        kwargs=dict(
+            engine=engine, interval_s=interval_s, stop_event=stop,
+            state=state, probe=probe, log=log,
+        ),
+        daemon=True,
+        name="recall-health",
+    )
+    t.start()
+    return t, stop, state
+
+
+def get_recall_health() -> dict:
+    """Snapshot for /health surfacing (visibility — never silent degradation)."""
+    s = _GLOBAL_STATE
+    return {
+        "recall_healthy": s.healthy,
+        "consecutive_failures": s.consecutive_failures,
+        "total_heals": s.total_heals,
+        "checks": s.checks,
+        "last_semantic_score": round(s.last_semantic_score, 4),
+        "last_error": s.last_error,
+    }

package/src/superlocalmemory/server/unified_daemon.py

@@ -683,6 +683,25 @@ async def lifespan(application: FastAPI):
         threading.Thread(target=_warmup_recall, daemon=True, name="recall-warmup").start()
         threading.Thread(target=_backfill_vector_store, daemon=True, name="vs-backfill").start()
 
+        # v3.6.8: Runtime recall-health monitor. The three warmups above run
+        # ONCE at boot; on a long-running daemon the graph page cache gets
+        # evicted and the embedder can start returning None while _embedding_warm
+        # still claims True — both silently degrade recall to keyword-only BM25
+        # (observed: session_init "pool.recall timed out → DEGRADED MODE", 7×/2d).
+        # This monitor re-warms + actively probes the semantic channel + self-heals
+        # a dead embedder for the daemon's whole life. See server/recall_health.py.
+        try:
+            from superlocalmemory.server.recall_health import (
+                start_recall_health_monitor,
+            )
+            _rh_thread, _rh_stop, _ = start_recall_health_monitor(engine)
+            application.state.recall_health_stop = _rh_stop
+        except Exception as _rh_exc:
+            logger.warning(
+                "recall-health monitor start failed (non-fatal): %s", _rh_exc,
+            )
+            application.state.recall_health_stop = None
+
         # v3.4.37: QueueConsumer uses daemon's engine directly via adapter.
         # Previously routed through WorkerPool → recall_worker subprocess,
         # which loaded a duplicate MemoryEngine (~800 MB waste).
@@ -968,6 +987,14 @@ async def lifespan(application: FastAPI):
         except Exception as exc:  # pragma: no cover — defensive
             logger.warning("recall_queue close failed: %s", exc)
 
+    # v3.6.8: Stop recall-health monitor (owns a daemon thread).
+    _rh_stop = getattr(application.state, "recall_health_stop", None)
+    if _rh_stop is not None:
+        try:
+            _rh_stop.set()
+        except Exception as exc:  # pragma: no cover — defensive
+            logger.warning("recall_health monitor stop failed: %s", exc)
+
     # Stop HealthMonitor (health_monitor.py owns a daemon thread).
     _health = getattr(application.state, "health_monitor", None)
     if _health is not None:
@@ -1579,6 +1606,13 @@ def _register_daemon_routes(application: FastAPI) -> None:
         _update_activity()
         # Non-blocking peek: report status without forcing a re-init.
         engine = getattr(application.state, "engine", None)
+        # v3.6.8: surface the recall-health verdict so a silently-degraded
+        # recall path (warm-but-broken embedder) is VISIBLE, never silent.
+        try:
+            from superlocalmemory.server.recall_health import get_recall_health
+            _recall_health = get_recall_health()
+        except Exception:
+            _recall_health = {"recall_healthy": None}
         return {
             "status": "ok",
             "pid": os.getpid(),
@@ -1587,6 +1621,9 @@ def _register_daemon_routes(application: FastAPI) -> None:
             # v3.4.52: clients can poll this to wait for embedding model
             # readiness before issuing recall calls.
             "embedding_warm": _embedding_warm,
+            # v3.6.8: True iff the semantic channel actually fired on the last
+            # health probe; includes self-heal counters.
+            "recall_health": _recall_health,
         }
 
     @application.get("/recall")

package/src/superlocalmemory.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: superlocalmemory
-Version: 3.6.7
+Version: 3.6.8
 Summary: Information-geometric agent memory with mathematical guarantees
 Author-email: Varun Pratap Bhardwaj <admin@superlocalmemory.com>
 License: AGPL-3.0-or-later

package/src/superlocalmemory.egg-info/SOURCES.txt

@@ -360,6 +360,7 @@ src/superlocalmemory/retrieval/vector_store.py
 src/superlocalmemory/server/__init__.py
 src/superlocalmemory/server/api.py
 src/superlocalmemory/server/bandit_loops.py
+src/superlocalmemory/server/recall_health.py
 src/superlocalmemory/server/recall_serializer.py
 src/superlocalmemory/server/security_middleware.py
 src/superlocalmemory/server/ui.py