superlocalmemory 3.4.48 → 3.4.51
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +46 -0
- package/README.md +43 -1
- package/package.json +1 -1
- package/pyproject.toml +1 -1
- package/src/superlocalmemory/__init__.py +1 -1
- package/src/superlocalmemory/cli/commands.py +7 -2
- package/src/superlocalmemory/cli/main.py +10 -0
- package/src/superlocalmemory/core/backend_orchestrator.py +365 -0
- package/src/superlocalmemory/core/pruning_engine.py +216 -0
- package/src/superlocalmemory/core/recall_pipeline.py +26 -2
- package/src/superlocalmemory/core/store_pipeline.py +21 -0
- package/src/superlocalmemory/core/tier_manager.py +124 -0
- package/src/superlocalmemory/graph/__init__.py +9 -0
- package/src/superlocalmemory/graph/cozo_backend.py +527 -0
- package/src/superlocalmemory/llm/backbone.py +1 -12
- package/src/superlocalmemory/mcp/_pool_adapter.py +2 -0
- package/src/superlocalmemory/mcp/tools_active.py +41 -1
- package/src/superlocalmemory/retrieval/engine.py +15 -3
- package/src/superlocalmemory/retrieval/entity_channel.py +50 -1
- package/src/superlocalmemory/retrieval/reranker.py +15 -0
- package/src/superlocalmemory/server/unified_daemon.py +3 -1
- package/src/superlocalmemory/storage/migration_runner.py +3 -0
- package/src/superlocalmemory/storage/migrations/M014_v345_scale_ready.py +45 -0
- package/src/superlocalmemory/storage/schema_v345.py +109 -0
- package/src/superlocalmemory/vector/__init__.py +9 -0
- package/src/superlocalmemory/vector/lancedb_backend.py +299 -0
- package/src/superlocalmemory.egg-info/PKG-INFO +44 -2
- package/src/superlocalmemory.egg-info/SOURCES.txt +8 -0
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,52 @@ All notable changes to SuperLocalMemory V3 will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [3.4.51] - 2026-05-28 — Recency Intelligence
|
|
9
|
+
|
|
10
|
+
**Session context is now time-aware.** Stale memories from completed projects and old debugging sessions no longer surface at session start. Frequently-recalled architectural decisions resist decay automatically.
|
|
11
|
+
|
|
12
|
+
### Fixed
|
|
13
|
+
- **Exponential recency decay + FSRS stability strengthening** (`retrieval/engine.py`) — Replaced the nearly-flat linear formula (range `[0.92×, 1.1×]`, 2.3% spread) with an Ebbinghaus exponential decay enhanced by FSRS v5 access-count stabilization. Formula: `boost = 0.8 + 0.3 × e^(-(ln2/S) × age_days)` where `S = 30d × min(2.0, 1 + 0.1 × access_count)`. Effect: a 45-day-old session handoff recalled 0 times → 0.91× (was 1.075×). Same memory recalled 10 times → 0.95× — frequently-used architectural decisions naturally resist decay without any category labeling. Reference: Dae & Jarrett (2024) FSRS v5; Ebbinghaus (1885) retention curve.
|
|
14
|
+
- **`age_days` hardcoded to 0 in adaptive ranker** (`core/recall_pipeline.py`) — Both `apply_adaptive_ranking` and `apply_v2_adaptive_ranking` were passing `"age_days": 0` to the LightGBM ranker, making it permanently blind to memory age. Now computes real age from `fact.created_at`. The ranker can now learn age-preference signals.
|
|
15
|
+
- **`created_at` missing from pool recall protocol** (`server/unified_daemon.py`, `mcp/_pool_adapter.py`) — The daemon's `/recall` response omitted `created_at`. Added to both recall response serialisation paths and to `PoolFact` dataclass. All MCP-layer tools now receive real memory timestamps.
|
|
16
|
+
- **`session_init` age gate** (`mcp/tools_active.py`) — Added `max_age_days: int = 30` parameter. Memories older than 30 days are suppressed unless their relevance score ≥ 0.70 (architectural decisions always surface). `max_age_days=0` disables the gate. Removed `fast=True` — session context deserves full 6-channel recall.
|
|
17
|
+
- **`slm session-context` age gate** (`cli/commands.py`) — Fast-path SQLite query now respects `--max-age-days` (default: 30). Previously hardcoded to 7 days. CLI and MCP now apply identical age semantics.
|
|
18
|
+
|
|
19
|
+
### Added
|
|
20
|
+
- **`slm session-context --max-age-days N`** (`cli/main.py`) — New flag to control how far back session context reaches. Default 30. Set to 0 to disable. Consistent with MCP `session_init(max_age_days=N)`.
|
|
21
|
+
- **`slm session-context --full`** — Explicit flag to use the full engine path (was implicit via code). Documented.
|
|
22
|
+
- **`slm session-context --json`** — Agent-native JSON output, consistent with all other commands.
|
|
23
|
+
|
|
24
|
+
### Changed
|
|
25
|
+
- `session_init` MCP tool schema gains optional `max_age_days` parameter (default: 30). Backward-compatible.
|
|
26
|
+
- `PoolFact` gains `created_at: str = ""` field. Backward-compatible — defaults to empty string.
|
|
27
|
+
- `slm session-context` fast path changed from hardcoded 7-day window to `--max-age-days` controlled window (default 30).
|
|
28
|
+
|
|
29
|
+
## [3.4.50] - 2026-05-25 — Scale-Ready
|
|
30
|
+
|
|
31
|
+
**1 million memories. Zero slowdown.** Tiered storage, graph pruning, and optional acceleration backends for infinite scale.
|
|
32
|
+
|
|
33
|
+
### Added
|
|
34
|
+
- **Tiered Storage (Hot/Warm/Cold/Archive)** — Facts auto-classified by age + access patterns. Hot facts prioritized in graph/vector search. Cold facts archived but never deleted. Nightly rebalance with misfire-safe cron. `slm pin <fact_id>` to keep any fact hot forever.
|
|
35
|
+
- **Graph Pruning Engine** — Chain collapse, garbage entity removal, low-activity edge decay. Reduced edge count while preserving semantic connections. Safe to run repeatedly — idempotent.
|
|
36
|
+
- **access_count_30d** — Rolling 30-day access window with batch-flush recording. Replaces lifetime counter for accurate tier assignment (F-14).
|
|
37
|
+
- **Optional Graph Acceleration** — `pip install superlocalmemory[cozo]` for CozoDB embedded graph backend. Replaces NetworkX for spreading activation at 1M+ edges. Zero-config: daemon auto-migrates on restart.
|
|
38
|
+
- **Optional Vector Acceleration** — `pip install superlocalmemory[lancedb]` for LanceDB embedded vector backend. Cosine similarity with IVF+PQ indexing. Auto-migration from sqlite-vec.
|
|
39
|
+
- **Backend Status Dashboard** — `slm doctor` shows CozoDB/LanceDB migration status, tier distribution, and health warnings.
|
|
40
|
+
- **M014_v345_scale_ready migration** — Automatic on daemon restart. Adds `access_count_30d` column and graph edge indexes.
|
|
41
|
+
|
|
42
|
+
### Changed
|
|
43
|
+
- **Migration is fully automatic.** Upgrade package → restart daemon → done. No `slm migrate` needed. Schema applied silently via the migration runner. Idempotent — safe to restart multiple times.
|
|
44
|
+
|
|
45
|
+
### Fixed
|
|
46
|
+
- **WAL lock contention** — Multiple stale `slm mcp` processes causing 16-second health checks. Process reaper enhanced with orphan detection.
|
|
47
|
+
- **Graph edge index performance** — Added `idx_graph_edges_source_id` and `idx_graph_edges_target_id` for bulk import at 1M+ scale (F-20).
|
|
48
|
+
|
|
49
|
+
## [3.4.49] - 2026-05-22
|
|
50
|
+
|
|
51
|
+
### Added
|
|
52
|
+
- **`SLM_DAEMON_HOST` env var** — Configurable host binding for the unified daemon. Previously hardcoded to `127.0.0.1`; now reads `SLM_DAEMON_HOST` (default `127.0.0.1`). Set to `0.0.0.0` to expose the SLM API on all LAN interfaces for cross-machine mesh use.
|
|
53
|
+
|
|
8
54
|
## [3.4.48] - 2026-05-21
|
|
9
55
|
|
|
10
56
|
**Multi-Machine Mesh Coordination — M4 & M5 now work as one.**
|
package/README.md
CHANGED
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
|
|
5
5
|
<h1 align="center">SuperLocalMemory V3.4</h1>
|
|
6
6
|
<p align="center"><strong>Every other AI forgets. Yours won't.</strong><br/><em>Infinite memory for Claude Code, Cursor, Windsurf, and any MCP-compatible AI client.</em></p>
|
|
7
|
-
<p align="center"><code>v3.4.
|
|
7
|
+
<p align="center"><code>v3.4.51 "Recency Intelligence"</code> — <strong>Session context is now time-aware.</strong><br>Stale memories from old projects no longer surface. Ebbinghaus decay + FSRS stability. One command: <code>pip install -U superlocalmemory && slm restart</code></p>
|
|
8
8
|
<p align="center"><strong>Backed by 3 published research papers</strong> (arXiv preprints + Zenodo-archived) · <a href="https://arxiv.org/abs/2603.02240">arXiv:2603.02240</a> · <a href="https://arxiv.org/abs/2603.14588">arXiv:2603.14588</a> · <a href="https://arxiv.org/abs/2604.04514">arXiv:2604.04514</a></p>
|
|
9
9
|
|
|
10
10
|
<p align="center">
|
|
@@ -183,6 +183,30 @@ slm warmup # Pre-download embedding model (~500MB, optional)
|
|
|
183
183
|
pip install superlocalmemory
|
|
184
184
|
```
|
|
185
185
|
|
|
186
|
+
### Upgrading to v3.4.5 "Scale-Ready"
|
|
187
|
+
|
|
188
|
+
**Migration is automatic.** Upgrade your package, restart the daemon, and your database migrates silently.
|
|
189
|
+
|
|
190
|
+
```bash
|
|
191
|
+
# pip users
|
|
192
|
+
pip install -U superlocalmemory
|
|
193
|
+
slm restart
|
|
194
|
+
|
|
195
|
+
# npm users
|
|
196
|
+
npm update -g superlocalmemory
|
|
197
|
+
slm restart
|
|
198
|
+
|
|
199
|
+
# Verify migration
|
|
200
|
+
slm doctor
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
No manual commands. No data loss. Your database upgrades in-place with zero downtime. The daemon auto-detects the old version and applies the migration on first start.
|
|
204
|
+
|
|
205
|
+
**New capabilities after upgrade:**
|
|
206
|
+
- Tiered storage: memories auto-classified as active/warm/cold/archived
|
|
207
|
+
- Graph pruning: redundant edges removed, queries stay fast at 1M+ connections
|
|
208
|
+
- Optional: `pip install superlocalmemory[cozo,lancedb]` for graph + vector acceleration
|
|
209
|
+
|
|
186
210
|
### First Use
|
|
187
211
|
|
|
188
212
|
```bash
|
|
@@ -410,6 +434,23 @@ Every recall generates learning signals. Over time, the system adapts to your pa
|
|
|
410
434
|
|
|
411
435
|
Auto-capture hooks: `slm hooks install` + `slm observe` + `slm session-context`. MCP tools: `session_init`, `observe`, `report_feedback`.
|
|
412
436
|
|
|
437
|
+
**`session_init` MCP parameters:**
|
|
438
|
+
| Parameter | Type | Default | Description |
|
|
439
|
+
|---|---|---|---|
|
|
440
|
+
| `project_path` | string | `""` | Working directory — used to derive search query |
|
|
441
|
+
| `query` | string | `""` | Override search query |
|
|
442
|
+
| `max_results` | int | `10` | Max memories to return |
|
|
443
|
+
| `max_age_days` | int | `30` | Suppress memories older than N days (0 = disabled). Memories with score ≥ 0.70 always surface regardless of age. |
|
|
444
|
+
|
|
445
|
+
**`slm session-context` CLI flags** (consistent with MCP):
|
|
446
|
+
```bash
|
|
447
|
+
slm session-context # fast path, 30-day window (default)
|
|
448
|
+
slm session-context --max-age-days 7 # only last 7 days
|
|
449
|
+
slm session-context --max-age-days 0 # no age filter
|
|
450
|
+
slm session-context "my query" --full # full engine path (slow, requires Ollama)
|
|
451
|
+
slm session-context --json # agent-native JSON output
|
|
452
|
+
```
|
|
453
|
+
|
|
413
454
|
**No competitor learns at zero token cost.**
|
|
414
455
|
|
|
415
456
|
</details>
|
|
@@ -548,6 +589,7 @@ All 8 mesh tools work seamlessly across machines:
|
|
|
548
589
|
| `slm hooks install` | Wire auto-memory into Claude Code hooks |
|
|
549
590
|
| `slm profile list/create/switch` | Profile management |
|
|
550
591
|
| `slm decay` | Run memory lifecycle review |
|
|
592
|
+
| `slm session-context [query]` | Print session context (for hooks). Flags: `--max-age-days N` (default 30), `--full`, `--json` |
|
|
551
593
|
| `slm quantize` | Run smart compression cycle |
|
|
552
594
|
| `slm consolidate --cognitive` | Extract patterns from memory clusters |
|
|
553
595
|
| `slm soft-prompts` | View auto-learned patterns |
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "superlocalmemory",
|
|
3
|
-
"version": "3.4.
|
|
3
|
+
"version": "3.4.51",
|
|
4
4
|
"description": "Information-geometric agent memory with mathematical guarantees. 4-channel retrieval, Fisher-Rao similarity, zero-LLM mode, EU AI Act compliant. Works with Claude, Cursor, Windsurf, and 17+ AI tools.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"ai-memory",
|
package/pyproject.toml
CHANGED
|
@@ -2147,12 +2147,17 @@ def cmd_session_context(args: Namespace) -> None:
|
|
|
2147
2147
|
except sqlite3.OperationalError:
|
|
2148
2148
|
pass
|
|
2149
2149
|
|
|
2150
|
-
# 2. Recent important memories
|
|
2150
|
+
# 2. Recent important memories — age gate from --max-age-days (default 30)
|
|
2151
|
+
max_age = getattr(args, "max_age_days", 30)
|
|
2152
|
+
age_clause = (
|
|
2153
|
+
f"AND created_at >= datetime('now', '-{int(max_age)} days') "
|
|
2154
|
+
if max_age > 0 else ""
|
|
2155
|
+
)
|
|
2151
2156
|
try:
|
|
2152
2157
|
rows = conn.execute(
|
|
2153
2158
|
"SELECT content, fact_type, created_at FROM atomic_facts "
|
|
2154
2159
|
"WHERE profile_id = ? "
|
|
2155
|
-
"
|
|
2160
|
+
f"{age_clause}"
|
|
2156
2161
|
"AND lifecycle = 'active' "
|
|
2157
2162
|
"ORDER BY importance DESC, created_at DESC LIMIT 10",
|
|
2158
2163
|
(pid,),
|
|
@@ -289,6 +289,16 @@ def main() -> None:
|
|
|
289
289
|
|
|
290
290
|
ctx_p = sub.add_parser("session-context", help="Print session context (for hooks)")
|
|
291
291
|
ctx_p.add_argument("query", nargs="?", default="", help="Optional context query")
|
|
292
|
+
ctx_p.add_argument(
|
|
293
|
+
"--max-age-days", type=int, default=30,
|
|
294
|
+
help="Suppress memories older than N days unless score ≥ 0.7 (default: 30). "
|
|
295
|
+
"Set 0 to disable age filter.",
|
|
296
|
+
)
|
|
297
|
+
ctx_p.add_argument(
|
|
298
|
+
"--full", action="store_true",
|
|
299
|
+
help="Use full engine path (slower, requires Ollama). Default is fast SQLite path.",
|
|
300
|
+
)
|
|
301
|
+
ctx_p.add_argument("--json", action="store_true", help="Output structured JSON (agent-native)")
|
|
292
302
|
|
|
293
303
|
obs_p = sub.add_parser("observe", help="Auto-capture content (pipe or argument)")
|
|
294
304
|
obs_p.add_argument("content", nargs="?", default="", help="Content to evaluate")
|
|
@@ -0,0 +1,365 @@
|
|
|
1
|
+
# Copyright (c) 2026 Varun Pratap Bhardwaj / Qualixar
|
|
2
|
+
# Licensed under AGPL-3.0-or-later - see LICENSE file
|
|
3
|
+
# Part of SuperLocalMemory V3 | https://qualixar.com | https://varunpratap.com
|
|
4
|
+
|
|
5
|
+
"""SuperLocalMemory v3.4.5 — Backend Orchestrator.
|
|
6
|
+
|
|
7
|
+
Central coordinator for multi-backend architecture.
|
|
8
|
+
Manages CozoDB, LanceDB, and TierManager lifecycle.
|
|
9
|
+
Handles auto-migration, fallback, and incremental sync.
|
|
10
|
+
|
|
11
|
+
This is the ONLY module that imports all three backends.
|
|
12
|
+
Other modules call BackendOrchestrator methods.
|
|
13
|
+
|
|
14
|
+
Part of Qualixar | Author: Varun Pratap Bhardwaj
|
|
15
|
+
"""
|
|
16
|
+
|
|
17
|
+
from __future__ import annotations
|
|
18
|
+
|
|
19
|
+
import logging
|
|
20
|
+
import sqlite3
|
|
21
|
+
import threading
|
|
22
|
+
from pathlib import Path
|
|
23
|
+
from typing import TYPE_CHECKING, Any
|
|
24
|
+
|
|
25
|
+
if TYPE_CHECKING:
|
|
26
|
+
from superlocalmemory.core.config import SLMConfig
|
|
27
|
+
from superlocalmemory.storage.database import DatabaseManager
|
|
28
|
+
|
|
29
|
+
logger = logging.getLogger(__name__)
|
|
30
|
+
|
|
31
|
+
# ---------------------------------------------------------------------------
|
|
32
|
+
# Global singleton (set by daemon, read by store_pipeline)
|
|
33
|
+
# ---------------------------------------------------------------------------
|
|
34
|
+
|
|
35
|
+
_orchestrator: BackendOrchestrator | None = None
|
|
36
|
+
|
|
37
|
+
|
|
38
|
+
def get_orchestrator() -> BackendOrchestrator | None:
|
|
39
|
+
"""Return the global BackendOrchestrator singleton."""
|
|
40
|
+
return _orchestrator
|
|
41
|
+
|
|
42
|
+
|
|
43
|
+
def set_orchestrator(orch: BackendOrchestrator) -> None:
|
|
44
|
+
"""Set the global BackendOrchestrator singleton."""
|
|
45
|
+
global _orchestrator
|
|
46
|
+
_orchestrator = orch
|
|
47
|
+
|
|
48
|
+
|
|
49
|
+
# ---------------------------------------------------------------------------
|
|
50
|
+
# BackendOrchestrator
|
|
51
|
+
# ---------------------------------------------------------------------------
|
|
52
|
+
|
|
53
|
+
class BackendOrchestrator:
|
|
54
|
+
"""Central coordinator for multi-backend architecture.
|
|
55
|
+
|
|
56
|
+
Lifecycle:
|
|
57
|
+
on_daemon_start() → migrate backends → ready
|
|
58
|
+
sync_new_fact() → called from store_pipeline after SQLite write
|
|
59
|
+
health_check() → returns status of all backends
|
|
60
|
+
"""
|
|
61
|
+
|
|
62
|
+
def __init__(self, config: SLMConfig, db: DatabaseManager) -> None:
|
|
63
|
+
self._config = config
|
|
64
|
+
self._db = db
|
|
65
|
+
self._data_dir = Path(config.data_dir)
|
|
66
|
+
self._cozo: Any = None
|
|
67
|
+
self._lancedb: Any = None
|
|
68
|
+
self._tiers: Any = None
|
|
69
|
+
self._backend_cache: dict[str, str] = {}
|
|
70
|
+
|
|
71
|
+
# ------------------------------------------------------------------
|
|
72
|
+
# Daemon Startup
|
|
73
|
+
# ------------------------------------------------------------------
|
|
74
|
+
|
|
75
|
+
def on_daemon_start(self) -> None:
|
|
76
|
+
"""Called once on daemon startup. Order matters (F-11: rebalance before migration)."""
|
|
77
|
+
logger.info("BackendOrchestrator: daemon starting")
|
|
78
|
+
|
|
79
|
+
# 1. Apply schema (if not already applied)
|
|
80
|
+
self._apply_schema_v345()
|
|
81
|
+
|
|
82
|
+
# 2. Initialize TierManager (always)
|
|
83
|
+
try:
|
|
84
|
+
from superlocalmemory.core.tier_manager import evaluate_tiers, set_backends
|
|
85
|
+
self._tiers = evaluate_tiers
|
|
86
|
+
set_backends(cozo=self._cozo, lancedb=self._lancedb)
|
|
87
|
+
logger.info("BackendOrchestrator: TierManager initialized")
|
|
88
|
+
except Exception as exc:
|
|
89
|
+
logger.warning("TierManager init failed (non-fatal): %s", exc)
|
|
90
|
+
|
|
91
|
+
# 3. Run initial tier rebalance FIRST (F-11: before migration)
|
|
92
|
+
try:
|
|
93
|
+
from superlocalmemory.core.tier_manager import evaluate_tiers as rebalance
|
|
94
|
+
result = rebalance(self._db)
|
|
95
|
+
logger.info("BackendOrchestrator: initial rebalance — %s",
|
|
96
|
+
result.get("total_evaluated", "?"))
|
|
97
|
+
except Exception as exc:
|
|
98
|
+
logger.warning("Initial rebalance failed (non-fatal): %s", exc)
|
|
99
|
+
|
|
100
|
+
# 4. Initialize CozoDB if available
|
|
101
|
+
cozo_available = self._detect_cozo()
|
|
102
|
+
if cozo_available:
|
|
103
|
+
self._init_cozo()
|
|
104
|
+
|
|
105
|
+
# 5. Initialize LanceDB if available
|
|
106
|
+
lancedb_available = self._detect_lancedb()
|
|
107
|
+
if lancedb_available:
|
|
108
|
+
self._init_lancedb()
|
|
109
|
+
|
|
110
|
+
# 6. Auto-migrate
|
|
111
|
+
if self._cozo:
|
|
112
|
+
status = self._cozo_status()
|
|
113
|
+
if status in ("not_initialized", "migrating"):
|
|
114
|
+
if status == "migrating":
|
|
115
|
+
logger.warning("CozoDB migration interrupted — rebuilding")
|
|
116
|
+
self._migrate_cozo()
|
|
117
|
+
|
|
118
|
+
if self._lancedb:
|
|
119
|
+
status = self._lancedb_status()
|
|
120
|
+
if status in ("not_initialized", "migrating"):
|
|
121
|
+
if status == "migrating":
|
|
122
|
+
logger.warning("LanceDB migration interrupted — rebuilding")
|
|
123
|
+
self._migrate_lancedb()
|
|
124
|
+
|
|
125
|
+
logger.info("BackendOrchestrator: daemon ready (cozo=%s, lancedb=%s)",
|
|
126
|
+
"active" if self._cozo and self._cozo_status() == "active" else "off",
|
|
127
|
+
"active" if self._lancedb and self._lancedb_status() == "active" else "off")
|
|
128
|
+
|
|
129
|
+
# ------------------------------------------------------------------
|
|
130
|
+
# Incremental Sync (F-04: called from store_pipeline)
|
|
131
|
+
# ------------------------------------------------------------------
|
|
132
|
+
|
|
133
|
+
def sync_new_fact(self, fact: Any) -> None:
|
|
134
|
+
"""Sync a newly stored fact to CozoDB and LanceDB.
|
|
135
|
+
|
|
136
|
+
Called AFTER SQLite write in store_pipeline.
|
|
137
|
+
Non-blocking, best-effort. Failures are logged, not raised.
|
|
138
|
+
"""
|
|
139
|
+
try:
|
|
140
|
+
tier = getattr(fact, "lifecycle", "active")
|
|
141
|
+
except Exception:
|
|
142
|
+
tier = "active"
|
|
143
|
+
|
|
144
|
+
if tier in ("active", "warm"):
|
|
145
|
+
if self._cozo and self._cozo_status() == "active":
|
|
146
|
+
self._sync_fact_entities(fact)
|
|
147
|
+
|
|
148
|
+
if self._lancedb and self._lancedb_status() == "active":
|
|
149
|
+
self._sync_fact_embedding(fact)
|
|
150
|
+
|
|
151
|
+
def _sync_fact_entities(self, fact: Any) -> None:
|
|
152
|
+
"""Sync fact's entities and edges to CozoDB."""
|
|
153
|
+
try:
|
|
154
|
+
entities = getattr(fact, "canonical_entities", []) or []
|
|
155
|
+
for eid in entities:
|
|
156
|
+
self._cozo.add_entity(eid, eid, "concept", {})
|
|
157
|
+
# Add edges from this entity to existing ones
|
|
158
|
+
for other in entities:
|
|
159
|
+
if other != eid:
|
|
160
|
+
self._cozo.add_edge(eid, other, "co_occurs", 1.0)
|
|
161
|
+
except Exception as exc:
|
|
162
|
+
logger.debug("CozoDB incremental sync skipped: %s", exc)
|
|
163
|
+
|
|
164
|
+
def _sync_fact_embedding(self, fact: Any) -> None:
|
|
165
|
+
"""Sync fact's embedding to LanceDB."""
|
|
166
|
+
try:
|
|
167
|
+
embedding = getattr(fact, "embedding", None)
|
|
168
|
+
if embedding:
|
|
169
|
+
tier = getattr(fact, "lifecycle", "active")
|
|
170
|
+
self._lancedb.add_vectors(
|
|
171
|
+
[fact.fact_id], [embedding], [tier],
|
|
172
|
+
)
|
|
173
|
+
except Exception as exc:
|
|
174
|
+
logger.debug("LanceDB incremental sync skipped: %s", exc)
|
|
175
|
+
|
|
176
|
+
# ------------------------------------------------------------------
|
|
177
|
+
# Backend Access
|
|
178
|
+
# ------------------------------------------------------------------
|
|
179
|
+
|
|
180
|
+
def get_graph_backend(self) -> Any:
|
|
181
|
+
"""Return active graph backend or None (caller falls back to NetworkX)."""
|
|
182
|
+
if self._cozo and self._cozo_status() == "active":
|
|
183
|
+
return self._cozo
|
|
184
|
+
return None
|
|
185
|
+
|
|
186
|
+
def get_vector_backend(self) -> Any:
|
|
187
|
+
"""Return active vector backend or None."""
|
|
188
|
+
if self._lancedb and self._lancedb_status() == "active":
|
|
189
|
+
return self._lancedb
|
|
190
|
+
return None
|
|
191
|
+
|
|
192
|
+
# ------------------------------------------------------------------
|
|
193
|
+
# Health Check
|
|
194
|
+
# ------------------------------------------------------------------
|
|
195
|
+
|
|
196
|
+
def health_check(self) -> dict[str, Any]:
|
|
197
|
+
"""Comprehensive health status for dashboard + CLI."""
|
|
198
|
+
result: dict[str, Any] = {
|
|
199
|
+
"sqlite": {"status": "active"},
|
|
200
|
+
"cozo": {"status": "not_available"},
|
|
201
|
+
"lancedb": {"status": "not_available"},
|
|
202
|
+
"tiers": {},
|
|
203
|
+
"warnings": [],
|
|
204
|
+
}
|
|
205
|
+
|
|
206
|
+
try:
|
|
207
|
+
from superlocalmemory.core.tier_manager import get_tier_stats
|
|
208
|
+
result["tiers"] = get_tier_stats(self._db)
|
|
209
|
+
except Exception:
|
|
210
|
+
pass
|
|
211
|
+
|
|
212
|
+
if self._cozo:
|
|
213
|
+
try:
|
|
214
|
+
result["cozo"] = self._cozo.health_check()
|
|
215
|
+
except Exception as exc:
|
|
216
|
+
result["cozo"] = {"status": "error", "error": str(exc)}
|
|
217
|
+
else:
|
|
218
|
+
result["warnings"].append(
|
|
219
|
+
"CozoDB not active. Install: pip install superlocalmemory[cozo]"
|
|
220
|
+
)
|
|
221
|
+
|
|
222
|
+
if self._lancedb:
|
|
223
|
+
try:
|
|
224
|
+
result["lancedb"] = self._lancedb.health_check()
|
|
225
|
+
except Exception as exc:
|
|
226
|
+
result["lancedb"] = {"status": "error", "error": str(exc)}
|
|
227
|
+
else:
|
|
228
|
+
result["warnings"].append(
|
|
229
|
+
"LanceDB not active. Install: pip install superlocalmemory[lancedb]"
|
|
230
|
+
)
|
|
231
|
+
|
|
232
|
+
return result
|
|
233
|
+
|
|
234
|
+
# ------------------------------------------------------------------
|
|
235
|
+
# Internal: Detection
|
|
236
|
+
# ------------------------------------------------------------------
|
|
237
|
+
|
|
238
|
+
def _detect_cozo(self) -> bool:
|
|
239
|
+
if self._config.get("graph_backend") == "sqlite":
|
|
240
|
+
return False
|
|
241
|
+
try:
|
|
242
|
+
import pycozo # noqa: F401
|
|
243
|
+
return True
|
|
244
|
+
except ImportError:
|
|
245
|
+
return False
|
|
246
|
+
|
|
247
|
+
def _detect_lancedb(self) -> bool:
|
|
248
|
+
if self._config.get("vector_backend") == "sqlite-vec":
|
|
249
|
+
return False
|
|
250
|
+
try:
|
|
251
|
+
import lancedb # noqa: F401
|
|
252
|
+
return True
|
|
253
|
+
except ImportError:
|
|
254
|
+
return False
|
|
255
|
+
|
|
256
|
+
# ------------------------------------------------------------------
|
|
257
|
+
# Internal: Init
|
|
258
|
+
# ------------------------------------------------------------------
|
|
259
|
+
|
|
260
|
+
def _init_cozo(self) -> None:
|
|
261
|
+
try:
|
|
262
|
+
from superlocalmemory.graph.cozo_backend import CozoDBGraphBackend
|
|
263
|
+
cozo_path = self._data_dir / "cozo"
|
|
264
|
+
cozo_path.mkdir(parents=True, exist_ok=True)
|
|
265
|
+
self._cozo = CozoDBGraphBackend(str(cozo_path / "graph"))
|
|
266
|
+
self._update_status("cozo", "not_initialized")
|
|
267
|
+
logger.info("CozoDB initialized at %s", cozo_path)
|
|
268
|
+
except Exception as exc:
|
|
269
|
+
logger.warning("CozoDB init failed: %s", exc)
|
|
270
|
+
self._cozo = None
|
|
271
|
+
|
|
272
|
+
def _init_lancedb(self) -> None:
|
|
273
|
+
try:
|
|
274
|
+
from superlocalmemory.vector.lancedb_backend import LanceDBVectorBackend
|
|
275
|
+
lance_path = self._data_dir / "lance"
|
|
276
|
+
self._lancedb = LanceDBVectorBackend(str(lance_path))
|
|
277
|
+
self._update_status("lancedb", "not_initialized")
|
|
278
|
+
logger.info("LanceDB initialized at %s", lance_path)
|
|
279
|
+
except Exception as exc:
|
|
280
|
+
logger.warning("LanceDB init failed: %s", exc)
|
|
281
|
+
self._lancedb = None
|
|
282
|
+
|
|
283
|
+
# ------------------------------------------------------------------
|
|
284
|
+
# Internal: Migration
|
|
285
|
+
# ------------------------------------------------------------------
|
|
286
|
+
|
|
287
|
+
def _migrate_cozo(self) -> None:
|
|
288
|
+
self._update_status("cozo", "migrating")
|
|
289
|
+
|
|
290
|
+
def _run():
|
|
291
|
+
conn = sqlite3.connect(str(self._data_dir / "memory.db"))
|
|
292
|
+
conn.execute("PRAGMA journal_mode=WAL")
|
|
293
|
+
conn.execute("PRAGMA query_only=ON") # F-07: read-only in migration thread
|
|
294
|
+
try:
|
|
295
|
+
count = self._cozo.bulk_import_from_sqlite(conn)
|
|
296
|
+
self._update_status("cozo", "active", count)
|
|
297
|
+
logger.info("CozoDB migration complete: %d edges", count)
|
|
298
|
+
except Exception as exc:
|
|
299
|
+
logger.error("CozoDB migration failed: %s", exc)
|
|
300
|
+
self._update_status("cozo", "failed", error=str(exc))
|
|
301
|
+
finally:
|
|
302
|
+
conn.close()
|
|
303
|
+
|
|
304
|
+
threading.Thread(target=_run, daemon=True).start()
|
|
305
|
+
|
|
306
|
+
def _migrate_lancedb(self) -> None:
|
|
307
|
+
self._update_status("lancedb", "migrating")
|
|
308
|
+
|
|
309
|
+
def _run():
|
|
310
|
+
conn = sqlite3.connect(str(self._data_dir / "memory.db"))
|
|
311
|
+
conn.execute("PRAGMA journal_mode=WAL")
|
|
312
|
+
conn.execute("PRAGMA query_only=ON")
|
|
313
|
+
try:
|
|
314
|
+
count = self._lancedb.bulk_import_from_sqlite(conn)
|
|
315
|
+
self._update_status("lancedb", "active", count)
|
|
316
|
+
logger.info("LanceDB migration complete: %d vectors", count)
|
|
317
|
+
except Exception as exc:
|
|
318
|
+
logger.error("LanceDB migration failed: %s", exc)
|
|
319
|
+
self._update_status("lancedb", "failed", error=str(exc))
|
|
320
|
+
finally:
|
|
321
|
+
conn.close()
|
|
322
|
+
|
|
323
|
+
threading.Thread(target=_run, daemon=True).start()
|
|
324
|
+
|
|
325
|
+
# ------------------------------------------------------------------
|
|
326
|
+
# Internal: Status
|
|
327
|
+
# ------------------------------------------------------------------
|
|
328
|
+
|
|
329
|
+
def _cozo_status(self) -> str:
|
|
330
|
+
return self._backend_cache.get("cozo", "not_initialized")
|
|
331
|
+
|
|
332
|
+
def _lancedb_status(self) -> str:
|
|
333
|
+
return self._backend_cache.get("lancedb", "not_initialized")
|
|
334
|
+
|
|
335
|
+
def _update_status(self, name: str, status: str,
|
|
336
|
+
count: int = 0, error: str = "") -> None:
|
|
337
|
+
self._backend_cache[name] = status
|
|
338
|
+
try:
|
|
339
|
+
self._db.conn.execute(
|
|
340
|
+
"INSERT OR REPLACE INTO backend_status "
|
|
341
|
+
"(backend_name, status, record_count, error_message, last_sync_at) "
|
|
342
|
+
"VALUES (?, ?, ?, ?, datetime('now'))",
|
|
343
|
+
(name, status, count, error),
|
|
344
|
+
)
|
|
345
|
+
self._db.conn.commit()
|
|
346
|
+
except Exception:
|
|
347
|
+
pass
|
|
348
|
+
|
|
349
|
+
# ------------------------------------------------------------------
|
|
350
|
+
# Internal: Schema
|
|
351
|
+
# ------------------------------------------------------------------
|
|
352
|
+
|
|
353
|
+
def _apply_schema_v345(self) -> None:
|
|
354
|
+
try:
|
|
355
|
+
from superlocalmemory.storage.schema_v345 import (
|
|
356
|
+
apply_migration, schema_version_applied,
|
|
357
|
+
)
|
|
358
|
+
if not schema_version_applied(self._db.conn):
|
|
359
|
+
result = apply_migration(self._db.conn)
|
|
360
|
+
if result.get("errors"):
|
|
361
|
+
logger.warning("Schema v3.4.5 had errors: %s", result["errors"])
|
|
362
|
+
except ImportError:
|
|
363
|
+
logger.debug("schema_v345 not found — skipping")
|
|
364
|
+
except Exception as exc:
|
|
365
|
+
logger.warning("Schema v3.4.5 apply failed (non-fatal): %s", exc)
|