superlocalmemory 3.7.2 → 3.7.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -5,6 +5,15 @@ All notable changes to SuperLocalMemory V3 will be documented in this file.
5
5
  The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
6
6
  and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
7
 
8
+ ## [3.7.3] - 2026-07-18 — Scale Engine integrity release
9
+
10
+ ### Fixed
11
+
12
+ - Added explicit, structurally verified adoption for pre-v3.7 CozoDB/LanceDB projections.
13
+ - Added durable promotion recovery, lifecycle serialization, and a final canonical-source consistency fence.
14
+ - Made Scale Engine status distinguish installed projection paths, lifecycle state, and live daemon backend health.
15
+ - Unified shipped runtime identity metadata at the release version.
16
+
8
17
  ## [3.7.2] - 2026-07-16 — Reliability release
9
18
 
10
19
  ### Fixed
package/README.md CHANGED
@@ -1,11 +1,14 @@
1
1
  <p align="center">
2
- <img src="https://superlocalmemory.com/assets/logo-mark.png" alt="SuperLocalMemory" width="200"/>
2
+ <picture>
3
+ <source media="(prefers-color-scheme: dark)" srcset="assets/branding/slm-wordmark-dark.svg">
4
+ <img src="assets/branding/slm-wordmark-light.svg" alt="SuperLocalMemory" width="390">
5
+ </picture>
3
6
  </p>
4
7
 
5
- <h1 align="center">SuperLocalMemory V3.7.2</h1>
8
+ <h1 align="center">SuperLocalMemory V3.7.3</h1>
6
9
  <p align="center"><strong>Cache. Compress. Remember. Three surfaces — proxy, MCP tools, or skill. Every setup covered.</strong><br/>
7
10
  <em>Local-first agent memory with explicit operating modes, auditable retrieval, and optional Optimize tools.</em></p>
8
- <p align="center"><code>v3.7.2</code> — <strong>Reliability release: durable local writes, cross-platform installation, and bounded enrichment recovery.</strong><br/>
11
+ <p align="center"><code>v3.7.3</code> — <strong>Scale Engine integrity release: explicit legacy adoption, durable promotion recovery, and honest runtime status.</strong><br/>
9
12
  Proxy: <code>slm wrap claude</code> &nbsp;·&nbsp; MCP: add <code>slm_compress</code> to your config &nbsp;·&nbsp; Skill: zero-config</p>
10
13
  <p align="center"><strong>3 public research preprints</strong> (arXiv + Zenodo archives) · <a href="https://arxiv.org/abs/2603.02240">arXiv:2603.02240</a> · <a href="https://arxiv.org/abs/2603.14588">arXiv:2603.14588</a> · <a href="https://arxiv.org/abs/2604.04514">arXiv:2604.04514</a></p>
11
14
 
@@ -108,7 +111,7 @@ health surfaces expose the stages actually completed by the installed runtime.
108
111
  | **Retrieval and recall** | Semantic, lexical, temporal, Hopfield and spreading-activation candidate channels; RRF fusion, optional reranking and graph score enhancement | Healthy channels participate; response provenance states the evidence used. |
109
112
  | **Brain and learning** | Behavioral patterns, feedback/outcome records, rewards, consolidation, LightGBM-related ranking components, soft prompts, and guarded skill-evolution workflows | Learning is evidence-driven; it does not claim autonomous correctness or guaranteed improvement. |
110
113
  | **Knowledge graph and entities** | Canonical entities, aliases, entity profiles, graph edges, scenes, timelines, explorer and graph APIs | Stored/derived graph data is evidence, not an instruction authority. |
111
- | **Scale Engine** | SQLite + sqlite-vec are canonical. CozoDB graph and LanceDB vector projections are packaged and managed with prepare → verify → promote → rollback | Promotion is explicit and parity-gated; do not advertise an unverified projection as the source of truth. |
114
+ | **Scale Engine** | SQLite + sqlite-vec are canonical. CozoDB graph and LanceDB vector projections are managed with prepare → verify → promote → rollback; a structurally detected pre-v3.7 projection can be explicitly adopted. | Promotion is parity-gated and crash-recoverable. Legacy adoption preserves the prior projection as a rollback backup; if proof fails, routing remains on canonical SQLite and status reports repair required. |
112
115
  | **Optimize** | Exact cache, tagged invalidation, safe compression, opt-in aggressive prose compression, CCR originals, proxy/MCP/skill surfaces | Only proxy intercepts a primary provider turn. MCP/skill cache results explicitly routed through SLM. |
113
116
  | **Mesh** | Authenticated peer messages, inbox/outbox, locks, offline queue, optional discovery and mesh MCP tools | Mesh is coordination, not automatic replicated memory or conflict resolution. |
114
117
  | **Governance and operations** | Provenance, audit/retention/policy surfaces, export/erasure controls, diagnostics, health, backups and daemon lifecycle | These are engineering controls, not a legal certification. |
@@ -227,10 +230,15 @@ optional reranking, and entity-graph score enhancement. The entity graph does
227
230
  not create an independent candidate in the current implementation. Core memory
228
231
  is SQLite-backed. SQLite and sqlite-vec remain the canonical source of truth.
229
232
  The packaged Scale Engine can maintain CozoDB graph and LanceDB vector
230
- projections, and it remains outside active retrieval paths until
231
- `slm db scale prepare`, `verify`, and `promote` prove parity against the
232
- canonical store. This makes the capability available on a fresh installation
233
- without silently migrating an existing user's data.
233
+ projections, and it remains outside active retrieval paths until a staged
234
+ parity witness proves it matches the canonical store. New installations remain
235
+ on Local Core. During upgrade, `slm db scale status` can identify a positive
236
+ pre-v3.7 layout candidate; the operator confirms it with `slm db scale adopt`.
237
+ SLM then rebuilds from canonical SQLite, verifies it, and promotes it with a
238
+ durable recovery journal while retaining the prior directories as a rollback
239
+ backup. `adopt` reports `restart_required: true`; run `slm restart` before
240
+ checking daemon health. If proof fails, recall remains on SQLite and status
241
+ reports repair required.
234
242
 
235
243
  Canonical ingestion is a durable state machine: `raw → queryable → enriching →
236
244
  complete`, with `failed` retaining raw evidence, error details, attempt count,
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "superlocalmemory",
3
- "version": "3.7.2",
3
+ "version": "3.7.3",
4
4
  "description": "Local-first agent memory with MCP and an agent-native CLI. Documented clients include Claude Code, Cursor, and Windsurf.",
5
5
  "keywords": [
6
6
  "ai-memory",
@@ -16,5 +16,5 @@
16
16
  "mcpServers": "./.mcp.json",
17
17
  "name": "superlocalmemory",
18
18
  "repository": "https://github.com/qualixar/superlocalmemory",
19
- "version": "3.7.2"
19
+ "version": "3.7.3"
20
20
  }
@@ -1 +1 @@
1
- superlocalmemory==3.7.2
1
+ superlocalmemory==3.7.3
@@ -1,5 +1,5 @@
1
1
  {
2
- "version": "3.7.2",
2
+ "version": "3.7.3",
3
3
  "pluginName": "superlocalmemory",
4
4
  "displayName": "SuperLocalMemory",
5
5
  "repository": "https://github.com/qualixar/superlocalmemory",
@@ -1 +1 @@
1
- superlocalmemory==3.7.2
1
+ superlocalmemory==3.7.3
package/pyproject.toml CHANGED
@@ -1,6 +1,6 @@
1
1
  [project]
2
2
  name = "superlocalmemory"
3
- version = "3.7.2"
3
+ version = "3.7.3"
4
4
  description = "Local-first agent memory with auditable hybrid retrieval"
5
5
  readme = "README.md"
6
6
  license = "AGPL-3.0-or-later"
@@ -32,7 +32,7 @@ if "OMP_NUM_THREADS" not in os.environ:
32
32
  os.environ["OMP_NUM_THREADS"] = "2"
33
33
  # ---------------------------------------------------------------------------
34
34
 
35
- __version__ = "3.7.2"
35
+ __version__ = "3.7.3"
36
36
 
37
37
  _REQUIRED_VERSIONS = {
38
38
  "sentence_transformers": "5.3.0",
@@ -270,8 +270,8 @@ def main() -> None:
270
270
  )
271
271
  db_scale_p.add_argument(
272
272
  "scale_action",
273
- choices=("status", "prepare", "verify", "promote", "rollback"),
274
- help="Lifecycle action. Prepare and verify never mutate active projection paths.",
273
+ choices=("status", "adopt", "prepare", "verify", "promote", "rollback"),
274
+ help="Lifecycle action. Adopt confirms a detected legacy projection; prepare and verify never mutate active paths.",
275
275
  )
276
276
  db_scale_p.add_argument("--stage-id", help="Stage identifier required by verify/promote")
277
277
  db_scale_p.add_argument("--backup-id", help="Backup identifier required by rollback")
@@ -9,11 +9,23 @@ def cmd_db_scale(args: Namespace) -> int:
9
9
  from superlocalmemory.core.config import SLMConfig
10
10
  from superlocalmemory.core.scale_engine import ScaleEngineError, ScaleEngineManager
11
11
 
12
- manager = ScaleEngineManager(SLMConfig.load())
13
12
  action = args.scale_action
13
+ # Scale Engine projections are currently canonical-default-profile data.
14
+ # Adoption must not fail merely because the operator is working in another
15
+ # profile, and it must not rewrite that selected profile.
16
+ manager = ScaleEngineManager(
17
+ SLMConfig.load(), profile_id="default" if action == "adopt" else None
18
+ )
14
19
  try:
15
20
  if action == "status":
16
21
  result = manager.status()
22
+ elif action == "adopt":
23
+ result = manager.adopt_legacy_projection()
24
+ if result is None:
25
+ raise ScaleEngineError(
26
+ "no confirmed legacy projection candidate; see `slm db scale status`"
27
+ )
28
+ result = {**result, "restart_required": True}
17
29
  elif action == "prepare":
18
30
  result = manager.prepare()
19
31
  elif action == "verify":
@@ -97,6 +97,8 @@ class BackendOrchestrator:
97
97
  except Exception as exc:
98
98
  logger.warning("Initial rebalance failed (non-fatal): %s", exc)
99
99
 
100
+ self._recover_interrupted_scale_promotion()
101
+
100
102
  # Backends may be installed with the product, but installing a wheel
101
103
  # is not authorization to mutate an existing data root. Only a
102
104
  # verified, explicit promotion may initialize and migrate projections.
@@ -134,6 +136,19 @@ class BackendOrchestrator:
134
136
  "active" if self._cozo and self._cozo_status() == "active" else "off",
135
137
  "active" if self._lancedb and self._lancedb_status() == "active" else "off")
136
138
 
139
+ def _recover_interrupted_scale_promotion(self) -> None:
140
+ """Repair an interrupted promotion; never auto-mutate a legacy root."""
141
+ try:
142
+ from superlocalmemory.core.scale_engine import ScaleEngineManager
143
+
144
+ result = ScaleEngineManager(self._config, profile_id="default").recover_interrupted_promotion()
145
+ if result:
146
+ logger.warning("Scale Engine promotion recovery: %s", result)
147
+ except Exception as exc:
148
+ # A scale projection is derived data. Startup must keep serving
149
+ # canonical SQLite even if optional recovery itself is unhealthy.
150
+ logger.error("Scale Engine recovery requires repair; Local Core remains active: %s", exc)
151
+
137
152
  # ------------------------------------------------------------------
138
153
  # Incremental Sync (F-04: called from store_pipeline)
139
154
  # ------------------------------------------------------------------
@@ -41,17 +41,20 @@ class ScaleEngineManager:
41
41
 
42
42
  MANIFEST_NAME = "scale-engine.json"
43
43
  SCHEMA_VERSION = 1
44
+ LIFECYCLE_LOCK = "scale-engine.lifecycle.lock"
45
+ PROMOTION_JOURNAL = "scale-engine.promotion.json"
44
46
 
45
47
  def __init__(
46
48
  self,
47
49
  config: Any,
48
50
  *,
49
51
  backend_factory: Callable[[Path, Path], tuple[Any, Any]] | None = None,
52
+ profile_id: str | None = None,
50
53
  ) -> None:
51
54
  self.config = config
52
55
  self.data_dir = Path(getattr(config, "data_dir", None) or config.base_dir)
53
56
  self.db_path = Path(getattr(config, "db_path", None) or self.data_dir / "memory.db")
54
- self.profile_id = getattr(config, "active_profile", "default")
57
+ self.profile_id = profile_id or getattr(config, "active_profile", "default")
55
58
  self._backend_factory = backend_factory or self._real_backend_factory
56
59
 
57
60
  @property
@@ -66,6 +69,14 @@ class ScaleEngineManager:
66
69
  def active_paths(self) -> tuple[Path, Path]:
67
70
  return self.data_dir / "cozo", self.data_dir / "lance"
68
71
 
72
+ @property
73
+ def lifecycle_lock_path(self) -> Path:
74
+ return self.data_dir / self.LIFECYCLE_LOCK
75
+
76
+ @property
77
+ def promotion_journal_path(self) -> Path:
78
+ return self.data_dir / self.PROMOTION_JOURNAL
79
+
69
80
  def status(self) -> dict[str, Any]:
70
81
  manifests: list[dict[str, Any]] = []
71
82
  if self.staging_root.exists():
@@ -74,21 +85,93 @@ class ScaleEngineManager:
74
85
  manifests.append(json.loads(path.read_text()))
75
86
  except (OSError, json.JSONDecodeError):
76
87
  manifests.append({"stage_id": path.parent.name, "state": "corrupt"})
88
+ backups = (
89
+ sorted(p.name for p in self.backup_root.glob("*") if p.is_dir())
90
+ if self.backup_root.exists()
91
+ else []
92
+ )
93
+ paths_present = {
94
+ "cozo": self.active_paths[0].exists(),
95
+ "lance": self.active_paths[1].exists(),
96
+ }
97
+ state = getattr(self.config, "scale_engine_state", "local_core")
98
+ legacy_projection_candidate = (
99
+ state == "local_core"
100
+ and all(paths_present.values())
101
+ and not manifests
102
+ and not backups
103
+ and self._has_legacy_projection_layout()
104
+ )
105
+ runtime = self._runtime_backend_status()
77
106
  return {
78
- "state": getattr(self.config, "scale_engine_state", "local_core"),
79
- "active": {"cozo": self.active_paths[0].exists(), "lance": self.active_paths[1].exists()},
107
+ "state": state,
108
+ # This command reads persisted state, not the live daemon. Never
109
+ # turn a last-known backend row into a present-tense routing claim.
110
+ "active": {"cozo": False, "lance": False},
111
+ "last_daemon_observation": runtime,
112
+ "paths_present": paths_present,
113
+ "retrieval_routing": (
114
+ "daemon_runtime_check_required" if state == "promoted" else "canonical_sqlite"
115
+ ),
116
+ "legacy_projection_candidate": legacy_projection_candidate,
117
+ "legacy_candidate_requires_confirmation": legacy_projection_candidate,
118
+ "migration_repair_required": (
119
+ self.promotion_journal_path.exists()
120
+ or (state == "local_core" and bool(manifests))
121
+ ),
80
122
  "stages": manifests,
81
- "backups": sorted(p.name for p in self.backup_root.glob("*") if p.is_dir()) if self.backup_root.exists() else [],
123
+ "backups": backups,
82
124
  }
83
125
 
126
+ def adopt_legacy_projection(self) -> dict[str, Any] | None:
127
+ """Safely adopt a v3.5-era projection into the staged lifecycle.
128
+
129
+ A legacy projection proves only that an older runtime created files.
130
+ It cannot establish parity with today's canonical SQLite database.
131
+ Rebuild a fresh stage, verify it while the canonical database is
132
+ stable, and promote it atomically; the legacy directories become the
133
+ explicit rollback copy.
134
+ """
135
+ if not self.status()["legacy_projection_candidate"]:
136
+ return None
137
+ lock_path = self._acquire_lifecycle_lock()
138
+ try:
139
+ # Re-check inside the lock. A concurrent command may have
140
+ # completed promotion while this caller waited to acquire it.
141
+ if not self.status()["legacy_projection_candidate"]:
142
+ return None
143
+ prepared = self._prepare()
144
+ self._verify(prepared["stage_id"])
145
+ return self._promote(prepared["stage_id"])
146
+ except Exception:
147
+ # A failed adoption must leave the canonical path selected. The
148
+ # stage is retained for operator inspection instead of deleting
149
+ # evidence about why an upgrade could not be completed.
150
+ self.config.scale_engine_state = "local_core"
151
+ self.config.graph_backend = "auto"
152
+ self.config.vector_backend = "auto"
153
+ self._save_config()
154
+ raise
155
+ finally:
156
+ self._release_lifecycle_lock(lock_path)
157
+
84
158
  def prepare(self) -> dict[str, Any]:
85
159
  """Build a new projection in a private staging directory."""
160
+ lock_path = self._acquire_lifecycle_lock()
161
+ try:
162
+ self._recover_interrupted_promotion()
163
+ return self._prepare()
164
+ finally:
165
+ self._release_lifecycle_lock(lock_path)
166
+
167
+ def _prepare(self) -> dict[str, Any]:
168
+ """Build a new projection while the caller owns the lifecycle lock."""
86
169
  self._require_default_profile()
87
170
  self._require_canonical_db()
88
171
  stage_id = f"{datetime.now(UTC).strftime('%Y%m%dT%H%M%SZ')}-{uuid.uuid4().hex[:8]}"
89
172
  stage_dir = self.staging_root / stage_id
90
173
  cozo_dir, lance_dir = stage_dir / "cozo", stage_dir / "lance"
91
- stage_dir.mkdir(parents=True, exist_ok=False)
174
+ self._mkdir_durable(stage_dir, exist_ok=False)
92
175
  cozo = lance = None
93
176
  try:
94
177
  cozo, lance = self._backend_factory(cozo_dir, lance_dir)
@@ -96,6 +179,7 @@ class ScaleEngineManager:
96
179
  cozo.bulk_import_from_sqlite(conn, self.profile_id)
97
180
  lance.bulk_import_from_sqlite(conn, self.profile_id)
98
181
  canonical = self._canonical_counts(conn)
182
+ source_fingerprint = self._projection_fingerprint(conn, canonical)
99
183
  observed = self._observed_counts(cozo, lance)
100
184
  manifest = {
101
185
  "schema_version": self.SCHEMA_VERSION,
@@ -105,7 +189,7 @@ class ScaleEngineManager:
105
189
  "profile_id": self.profile_id,
106
190
  "canonical": canonical,
107
191
  "observed": observed,
108
- "source_fingerprint": self._source_fingerprint(canonical),
192
+ "source_fingerprint": source_fingerprint,
109
193
  }
110
194
  self._write_manifest(stage_dir, manifest)
111
195
  self.config.scale_engine_state = "prepared"
@@ -120,6 +204,15 @@ class ScaleEngineManager:
120
204
 
121
205
  def verify(self, stage_id: str) -> dict[str, Any]:
122
206
  """Prove a staged projection matches the current canonical SQLite data."""
207
+ lock_path = self._acquire_lifecycle_lock()
208
+ try:
209
+ self._recover_interrupted_promotion()
210
+ return self._verify(stage_id)
211
+ finally:
212
+ self._release_lifecycle_lock(lock_path)
213
+
214
+ def _verify(self, stage_id: str) -> dict[str, Any]:
215
+ """Verify while the caller owns the lifecycle lock."""
123
216
  stage_dir, manifest = self._load_stage(stage_id)
124
217
  self._validate_manifest(manifest, state="prepared")
125
218
  self._require_default_profile()
@@ -128,8 +221,9 @@ class ScaleEngineManager:
128
221
  cozo, lance = self._backend_factory(stage_dir / "cozo", stage_dir / "lance")
129
222
  with self._readonly_connection() as conn:
130
223
  canonical = self._canonical_counts(conn)
224
+ source_fingerprint = self._projection_fingerprint(conn, canonical)
131
225
  observed = self._observed_counts(cozo, lance)
132
- if manifest["source_fingerprint"] != self._source_fingerprint(canonical):
226
+ if manifest["source_fingerprint"] != source_fingerprint:
133
227
  raise ScaleEngineError("canonical SQLite changed after preparation; prepare a new stage")
134
228
  if canonical != manifest["canonical"] or observed != canonical:
135
229
  raise ScaleEngineError(
@@ -146,6 +240,15 @@ class ScaleEngineManager:
146
240
 
147
241
  def promote(self, stage_id: str) -> dict[str, Any]:
148
242
  """Move a verified stage into active paths, preserving a rollback copy."""
243
+ lock_path = self._acquire_lifecycle_lock()
244
+ try:
245
+ self._recover_interrupted_promotion()
246
+ return self._promote(stage_id)
247
+ finally:
248
+ self._release_lifecycle_lock(lock_path)
249
+
250
+ def _promote(self, stage_id: str) -> dict[str, Any]:
251
+ """Promote while the caller owns the lifecycle lock."""
149
252
  stage_dir, manifest = self._load_stage(stage_id)
150
253
  self._validate_manifest(manifest, state="verified")
151
254
  staged = (stage_dir / "cozo", stage_dir / "lance")
@@ -153,39 +256,78 @@ class ScaleEngineManager:
153
256
  raise ScaleEngineError("verified stage is incomplete; prepare a new stage")
154
257
  backup_dir = self.backup_root / f"{stage_id}-{uuid.uuid4().hex[:6]}"
155
258
  active = self.active_paths
156
- self.backup_root.mkdir(parents=True, exist_ok=True)
157
- # Keep an explicit empty rollback point as well: a first promotion has
158
- # no former projection directories, but rollback must still be able to
159
- # return the installation to Local Core without deleting anything.
160
- backup_dir.mkdir(parents=True, exist_ok=False)
161
- moved_active: list[tuple[Path, Path]] = []
162
- moved_stage: list[tuple[Path, Path]] = []
259
+ gate = sqlite3.connect(self.db_path, timeout=30)
163
260
  try:
261
+ # The stage was built from a point-in-time SQLite snapshot. Hold a
262
+ # short writer fence for the final fingerprint check and directory
263
+ # swap so no successful promotion can trail a canonical write.
264
+ gate.execute("BEGIN IMMEDIATE")
265
+ canonical = self._canonical_counts(gate)
266
+ if manifest["source_fingerprint"] != self._projection_fingerprint(gate, canonical):
267
+ raise ScaleEngineError("canonical SQLite changed after verification; prepare a new stage")
268
+ self._mkdir_durable(self.backup_root)
269
+ journal = {
270
+ "schema_version": self.SCHEMA_VERSION,
271
+ "operation": "promotion",
272
+ "state": "intent",
273
+ "stage_id": stage_id,
274
+ "backup_id": backup_dir.name,
275
+ "moves": [],
276
+ }
277
+ self._write_promotion_journal(journal)
278
+ self._mkdir_durable(backup_dir, exist_ok=False)
164
279
  for name, source, destination in zip(("cozo", "lance"), active, staged):
165
280
  if source.exists():
166
281
  target = backup_dir / name
167
- os.replace(source, target)
168
- moved_active.append((source, target))
169
- os.replace(destination, source)
170
- moved_stage.append((destination, source))
282
+ move = {"name": name, "kind": "active_to_backup", "state": "intent"}
283
+ journal["moves"].append(move)
284
+ self._write_promotion_journal(journal)
285
+ self._replace_durable(source, target)
286
+ move["state"] = "complete"
287
+ self._write_promotion_journal(journal)
288
+ move = {"name": name, "kind": "stage_to_active", "state": "intent"}
289
+ journal["moves"].append(move)
290
+ self._write_promotion_journal(journal)
291
+ self._replace_durable(destination, source)
292
+ move["state"] = "complete"
293
+ self._write_promotion_journal(journal)
171
294
  manifest.update({"state": "promoted", "promoted_at": _utc_now(), "backup_id": backup_dir.name})
172
295
  self._write_manifest(stage_dir, manifest)
173
296
  self.config.scale_engine_state = "promoted"
174
297
  self.config.graph_backend = "cozo"
175
298
  self.config.vector_backend = "lancedb"
176
299
  self._save_config()
300
+ journal["state"] = "committed"
301
+ self._write_promotion_journal(journal)
302
+ self.promotion_journal_path.unlink(missing_ok=True)
303
+ gate.rollback()
177
304
  return manifest
178
305
  except Exception as exc:
179
- for staged_path, active_path in reversed(moved_stage):
180
- if active_path.exists():
181
- os.replace(active_path, staged_path)
182
- for active_path, backup_path in reversed(moved_active):
183
- if backup_path.exists():
184
- os.replace(backup_path, active_path)
306
+ try:
307
+ gate.rollback()
308
+ except sqlite3.Error:
309
+ pass
310
+ try:
311
+ self._recover_interrupted_promotion()
312
+ except ScaleEngineError as recovery_error:
313
+ raise ScaleEngineError(
314
+ f"promotion interrupted; automatic recovery needs repair: {recovery_error}"
315
+ ) from exc
185
316
  raise ScaleEngineError(f"promotion rolled back: {exc}") from exc
317
+ finally:
318
+ gate.close()
186
319
 
187
320
  def rollback(self, backup_id: str) -> dict[str, Any]:
188
321
  """Restore an explicitly named pre-promotion backup."""
322
+ lock_path = self._acquire_lifecycle_lock()
323
+ try:
324
+ self._recover_interrupted_promotion()
325
+ return self._rollback(backup_id)
326
+ finally:
327
+ self._release_lifecycle_lock(lock_path)
328
+
329
+ def _rollback(self, backup_id: str) -> dict[str, Any]:
330
+ """Roll back while the caller owns the lifecycle lock."""
189
331
  backup_dir = self.backup_root / backup_id
190
332
  if not backup_dir.is_dir():
191
333
  raise ScaleEngineError(f"backup does not exist: {backup_id}")
@@ -193,19 +335,47 @@ class ScaleEngineManager:
193
335
  active = self.active_paths
194
336
  displaced = self.backup_root / f"rollback-displaced-{uuid.uuid4().hex[:8]}"
195
337
  try:
338
+ journal = {
339
+ "schema_version": self.SCHEMA_VERSION,
340
+ "operation": "rollback",
341
+ "state": "intent",
342
+ "backup_id": backup_id,
343
+ "displaced_id": displaced.name,
344
+ "moves": [],
345
+ }
346
+ self._write_promotion_journal(journal)
196
347
  for name, source, target in zip(("cozo", "lance"), active, backup_paths):
197
348
  if source.exists():
198
- displaced.mkdir(parents=True, exist_ok=True)
199
- os.replace(source, displaced / name)
349
+ self._mkdir_durable(displaced)
350
+ move = {"name": name, "kind": "active_to_displaced", "state": "intent"}
351
+ journal["moves"].append(move)
352
+ self._write_promotion_journal(journal)
353
+ self._replace_durable(source, displaced / name)
354
+ move["state"] = "complete"
355
+ self._write_promotion_journal(journal)
200
356
  if target.exists():
201
- os.replace(target, source)
357
+ move = {"name": name, "kind": "backup_to_active", "state": "intent"}
358
+ journal["moves"].append(move)
359
+ self._write_promotion_journal(journal)
360
+ self._replace_durable(target, source)
361
+ move["state"] = "complete"
362
+ self._write_promotion_journal(journal)
202
363
  self.config.scale_engine_state = "local_core"
203
364
  self.config.graph_backend = "auto"
204
365
  self.config.vector_backend = "auto"
205
366
  self._save_config()
367
+ journal["state"] = "committed"
368
+ self._write_promotion_journal(journal)
369
+ self.promotion_journal_path.unlink(missing_ok=True)
206
370
  return {"state": "local_core", "restored_backup": backup_id, "displaced": displaced.name}
207
371
  except Exception as exc:
208
- raise ScaleEngineError(f"rollback failed; inspect {backup_dir}: {exc}") from exc
372
+ try:
373
+ self._recover_interrupted_promotion()
374
+ except ScaleEngineError as recovery_error:
375
+ raise ScaleEngineError(
376
+ f"rollback interrupted; automatic recovery needs repair: {recovery_error}"
377
+ ) from exc
378
+ raise ScaleEngineError(f"rollback recovered: {exc}") from exc
209
379
 
210
380
  def _real_backend_factory(self, cozo_dir: Path, lance_dir: Path) -> tuple[Any, Any]:
211
381
  from superlocalmemory.graph.cozo_backend import CozoDBGraphBackend
@@ -243,17 +413,67 @@ class ScaleEngineManager:
243
413
  raise ScaleEngineError(f"projection health failed: cozo={graph}, lancedb={vector}")
244
414
  return {"entities": int(graph["entities"]), "edges": int(graph["edges"]), "vectors": int(vector["vectors"])}
245
415
 
246
- def _source_fingerprint(self, counts: dict[str, int]) -> str:
416
+ def _projection_fingerprint(
417
+ self, conn: sqlite3.Connection, counts: dict[str, int]
418
+ ) -> str:
419
+ """Hash the exact projection source rows inside one SQLite snapshot."""
247
420
  digest = hashlib.sha256()
248
421
  digest.update(json.dumps(counts, sort_keys=True).encode())
249
- for path in (self.db_path, self.db_path.with_name(self.db_path.name + "-wal")):
250
- if path.exists():
251
- digest.update(path.name.encode())
252
- with path.open("rb") as handle:
253
- for block in iter(lambda: handle.read(1024 * 1024), b""):
254
- digest.update(block)
422
+ tables = (
423
+ ("canonical_entities", "entity_id, canonical_name, entity_type, first_seen, last_seen, fact_count, profile_id", "entity_id"),
424
+ ("atomic_facts", "fact_id, canonical_entities_json, lifecycle, profile_id", "fact_id"),
425
+ ("graph_edges", "source_id, target_id, edge_type, weight, profile_id", "source_id, target_id, edge_type"),
426
+ )
427
+ for table, columns, ordering in tables:
428
+ try:
429
+ rows = conn.execute(
430
+ f"SELECT {columns} FROM {table} WHERE profile_id=? ORDER BY {ordering}",
431
+ (self.profile_id,),
432
+ )
433
+ for row in rows:
434
+ self._digest_row(digest, table, row)
435
+ except sqlite3.OperationalError as exc:
436
+ raise ScaleEngineError(f"canonical SQLite missing required {table} table") from exc
437
+ try:
438
+ rows = conn.execute(
439
+ "SELECT fer.rowid, fer.fact_id, vec.vector FROM fact_embeddings_rowids fer "
440
+ "JOIN atomic_facts af ON af.fact_id = fer.fact_id "
441
+ "LEFT JOIN fact_embeddings_vector_chunks00 vec ON vec.rowid = fer.rowid "
442
+ "WHERE af.profile_id=? ORDER BY fer.rowid",
443
+ (self.profile_id,),
444
+ )
445
+ for row in rows:
446
+ self._digest_row(digest, "fact_embeddings_rowids", row)
447
+ except sqlite3.OperationalError:
448
+ pass
255
449
  return digest.hexdigest()
256
450
 
451
+ @staticmethod
452
+ def _digest_row(digest: Any, table: str, row: Any) -> None:
453
+ digest.update(table.encode())
454
+ digest.update(json.dumps(list(row), default=str, separators=(",", ":")).encode())
455
+
456
+ def _has_legacy_projection_layout(self) -> bool:
457
+ cozo, lance = self.active_paths
458
+ return (cozo / "graph").is_dir() and (lance / "embeddings.lance").exists()
459
+
460
+ def _runtime_backend_status(self) -> dict[str, str]:
461
+ status = {"cozo": "unknown", "lance": "unknown"}
462
+ try:
463
+ with self._readonly_connection() as conn:
464
+ rows = conn.execute(
465
+ "SELECT backend_name, status FROM backend_status "
466
+ "WHERE backend_name IN ('cozo', 'lancedb')"
467
+ )
468
+ for name, value in rows:
469
+ if name == "cozo":
470
+ status["cozo"] = str(value)
471
+ elif name == "lancedb":
472
+ status["lance"] = str(value)
473
+ except sqlite3.OperationalError:
474
+ pass
475
+ return status
476
+
257
477
  def _load_stage(self, stage_id: str) -> tuple[Path, dict[str, Any]]:
258
478
  stage_dir = self.staging_root / stage_id
259
479
  try:
@@ -280,6 +500,201 @@ class ScaleEngineManager:
280
500
  if not self.db_path.exists():
281
501
  raise ScaleEngineError(f"canonical SQLite database not found: {self.db_path}")
282
502
 
503
+ def _acquire_lifecycle_lock(self) -> Path:
504
+ """Serialize every mutating lifecycle command across processes."""
505
+ lock_path = self.lifecycle_lock_path
506
+ descriptor = None
507
+ for attempt in range(2):
508
+ try:
509
+ descriptor = os.open(lock_path, os.O_CREAT | os.O_EXCL | os.O_WRONLY)
510
+ break
511
+ except FileExistsError as exc:
512
+ if attempt == 0 and self._clear_dead_legacy_adoption_lock(lock_path):
513
+ continue
514
+ raise ScaleEngineError(
515
+ "Scale Engine lifecycle operation already in progress; retry after it completes"
516
+ ) from exc
517
+ if descriptor is None:
518
+ raise ScaleEngineError("could not acquire Scale Engine lifecycle lock")
519
+ try:
520
+ with os.fdopen(descriptor, "w") as lock_file:
521
+ json.dump({"pid": os.getpid(), "started_at": _utc_now()}, lock_file)
522
+ except Exception:
523
+ lock_path.unlink(missing_ok=True)
524
+ raise
525
+ return lock_path
526
+
527
+ @staticmethod
528
+ def _release_lifecycle_lock(lock_path: Path) -> None:
529
+ lock_path.unlink(missing_ok=True)
530
+
531
+ @staticmethod
532
+ def _clear_dead_legacy_adoption_lock(lock_path: Path) -> bool:
533
+ """Recover only a lock whose recorded process no longer exists."""
534
+ try:
535
+ owner = json.loads(lock_path.read_text())
536
+ pid = owner.get("pid")
537
+ if not isinstance(pid, int) or pid <= 0:
538
+ return False
539
+ os.kill(pid, 0)
540
+ except ProcessLookupError:
541
+ lock_path.unlink(missing_ok=True)
542
+ return True
543
+ except (OSError, ValueError, json.JSONDecodeError):
544
+ return False
545
+ return False
546
+
547
+ def recover_interrupted_promotion(self) -> str | None:
548
+ """Recover a durable promotion journal before opening projection paths."""
549
+ lock_path = self._acquire_lifecycle_lock()
550
+ try:
551
+ return self._recover_interrupted_promotion()
552
+ finally:
553
+ self._release_lifecycle_lock(lock_path)
554
+
555
+ def _recover_interrupted_promotion(self) -> str | None:
556
+ """Finalize or reverse an interrupted directory swap under lifecycle lock."""
557
+ if not self.promotion_journal_path.exists():
558
+ return None
559
+ try:
560
+ journal = json.loads(self.promotion_journal_path.read_text())
561
+ backup_id = str(journal["backup_id"])
562
+ state = str(journal["state"])
563
+ except (OSError, KeyError, TypeError, json.JSONDecodeError) as exc:
564
+ raise ScaleEngineError("invalid promotion journal; manual repair required") from exc
565
+ operation = str(journal.get("operation", "promotion"))
566
+ if state == "committed":
567
+ if operation == "promotion":
568
+ self.config.scale_engine_state = "promoted"
569
+ self.config.graph_backend = "cozo"
570
+ self.config.vector_backend = "lancedb"
571
+ elif operation == "rollback":
572
+ self.config.scale_engine_state = "local_core"
573
+ self.config.graph_backend = "auto"
574
+ self.config.vector_backend = "auto"
575
+ else:
576
+ raise ScaleEngineError(f"unknown journal operation: {operation!r}")
577
+ self._save_config()
578
+ self.promotion_journal_path.unlink(missing_ok=True)
579
+ return f"finalized_committed_{operation}"
580
+ if state != "intent":
581
+ raise ScaleEngineError(f"unknown promotion journal state: {state!r}")
582
+ backup_dir = self.backup_root / backup_id
583
+ active = dict(zip(("cozo", "lance"), self.active_paths))
584
+ stage_dir = self.staging_root / str(journal.get("stage_id", ""))
585
+ displaced_dir = self.backup_root / str(journal.get("displaced_id", ""))
586
+ moves = journal.get("moves")
587
+ if not isinstance(moves, list):
588
+ # Compatibility for journals written by the initial v3.7.3
589
+ # candidate before per-rename intents existed.
590
+ moves = [
591
+ {"name": name, "kind": "active_to_backup", "state": "complete"}
592
+ for name in journal.get("moved_active", [])
593
+ ] + [
594
+ {"name": name, "kind": "stage_to_active", "state": "complete"}
595
+ for name in journal.get("moved_stage", [])
596
+ ]
597
+ for move in reversed(moves):
598
+ self._reverse_journal_move(move, active, stage_dir, backup_dir, displaced_dir)
599
+ if operation == "promotion":
600
+ if backup_dir.exists() and not any(backup_dir.iterdir()):
601
+ backup_dir.rmdir()
602
+ self.config.scale_engine_state = "local_core"
603
+ self.config.graph_backend = "auto"
604
+ self.config.vector_backend = "auto"
605
+ elif operation == "rollback":
606
+ if displaced_dir.exists() and not any(displaced_dir.iterdir()):
607
+ displaced_dir.rmdir()
608
+ self.config.scale_engine_state = "promoted"
609
+ self.config.graph_backend = "cozo"
610
+ self.config.vector_backend = "lancedb"
611
+ else:
612
+ raise ScaleEngineError(f"unknown journal operation: {operation!r}")
613
+ self._save_config()
614
+ self.promotion_journal_path.unlink(missing_ok=True)
615
+ return f"reversed_interrupted_{operation}"
616
+
617
+ @staticmethod
618
+ def _reverse_journal_move(
619
+ move: Any,
620
+ active: dict[str, Path],
621
+ stage_dir: Path,
622
+ backup_dir: Path,
623
+ displaced_dir: Path,
624
+ ) -> None:
625
+ """Reverse one planned rename based on actual paths, not journal timing."""
626
+ if not isinstance(move, dict):
627
+ raise ScaleEngineError("invalid promotion journal move")
628
+ name = move.get("name")
629
+ kind = move.get("kind")
630
+ active_path = active.get(name)
631
+ if active_path is None:
632
+ raise ScaleEngineError(f"invalid promotion journal backend: {name!r}")
633
+ if kind == "stage_to_active":
634
+ source, target = active_path, stage_dir / name
635
+ elif kind == "active_to_backup":
636
+ source, target = backup_dir / name, active_path
637
+ elif kind == "active_to_displaced":
638
+ source, target = displaced_dir / name, active_path
639
+ elif kind == "backup_to_active":
640
+ source, target = active_path, backup_dir / name
641
+ else:
642
+ raise ScaleEngineError(f"invalid promotion journal move kind: {kind!r}")
643
+ if source.exists() and not target.exists():
644
+ ScaleEngineManager._replace_durable(source, target)
645
+ elif target.exists() and not source.exists():
646
+ return
647
+ else:
648
+ raise ScaleEngineError(f"cannot safely reconcile {kind} for {name}")
649
+
650
+ def _write_promotion_journal(self, journal: dict[str, Any]) -> None:
651
+ self._write_json_durable(self.promotion_journal_path, journal)
652
+
653
+ @staticmethod
654
+ def _write_json_durable(target: Path, payload: dict[str, Any]) -> None:
655
+ temporary = target.with_suffix(target.suffix + ".tmp")
656
+ with temporary.open("w") as handle:
657
+ json.dump(payload, handle, indent=2, sort_keys=True)
658
+ handle.write("\n")
659
+ handle.flush()
660
+ os.fsync(handle.fileno())
661
+ os.replace(temporary, target)
662
+ ScaleEngineManager._fsync_directory(target.parent)
663
+
664
+ @staticmethod
665
+ def _replace_durable(source: Path, target: Path) -> None:
666
+ """Rename a projection path and persist both directory entries."""
667
+ os.replace(source, target)
668
+ ScaleEngineManager._fsync_directory(source.parent)
669
+ if target.parent != source.parent:
670
+ ScaleEngineManager._fsync_directory(target.parent)
671
+
672
+ @staticmethod
673
+ def _mkdir_durable(path: Path, *, exist_ok: bool = True) -> None:
674
+ """Create a directory and persist every new parent entry before rename."""
675
+ missing: list[Path] = []
676
+ ancestor = path
677
+ while not ancestor.exists():
678
+ missing.append(ancestor)
679
+ ancestor = ancestor.parent
680
+ path.mkdir(parents=True, exist_ok=exist_ok)
681
+ for created in reversed(missing):
682
+ ScaleEngineManager._fsync_directory(created.parent)
683
+
684
+ @staticmethod
685
+ def _fsync_directory(directory_path: Path) -> None:
686
+ """Best-effort directory-entry durability across local filesystems."""
687
+ try:
688
+ directory = os.open(directory_path, os.O_RDONLY)
689
+ try:
690
+ os.fsync(directory)
691
+ finally:
692
+ os.close(directory)
693
+ except OSError:
694
+ # The file itself is already durable where directory fsync is not
695
+ # supported by the local filesystem (notably some Windows setups).
696
+ pass
697
+
283
698
  def _save_config(self) -> None:
284
699
  save = getattr(self.config, "save", None)
285
700
  if callable(save):
@@ -28,6 +28,7 @@ import sys
28
28
  import tempfile
29
29
  from pathlib import Path
30
30
 
31
+ from superlocalmemory import __version__
31
32
  from superlocalmemory.infra.data_root import canonical_data_root
32
33
  from superlocalmemory.infra.data_root import state_path as runtime_state_path
33
34
 
@@ -40,7 +41,7 @@ _DEFAULT_DISABLED_FILE = _DEFAULT_VERSION_DIR / ".hooks-disabled"
40
41
  VERSION_DIR = _DEFAULT_VERSION_DIR
41
42
  VERSION_FILE = _DEFAULT_VERSION_FILE
42
43
  DISABLED_FILE = _DEFAULT_DISABLED_FILE
43
- HOOKS_VERSION = "3.7.1"
44
+ HOOKS_VERSION = __version__
44
45
 
45
46
  # Cross-platform temp dir and backwards-compatible marker overrides. Runtime
46
47
  # defaults are root-namespaced and resolved when hook definitions are built.
@@ -19,10 +19,11 @@ from datetime import datetime, timezone
19
19
  from pathlib import Path
20
20
  from typing import Callable, Iterable
21
21
 
22
+ from superlocalmemory import __version__
22
23
  from superlocalmemory.core.security_primitives import redact_secrets
23
24
 
24
25
 
25
- VERSION = "3.7.1"
26
+ VERSION = __version__
26
27
  DEFAULT_TOP_K = 10
27
28
  DEFAULT_DECISIONS_K = 5
28
29
  DEFAULT_MEMORIES_K = 10
@@ -16,6 +16,7 @@ from __future__ import annotations
16
16
  from mcp.server.fastmcp import FastMCP
17
17
  from sse_starlette.sse import EventSourceResponse
18
18
  from starlette.types import Receive, Scope, Send
19
+ from superlocalmemory import __version__
19
20
 
20
21
 
21
22
  class ClosingEventSourceResponse(EventSourceResponse):
@@ -40,7 +41,7 @@ def install_streamable_http_resource_guard() -> None:
40
41
  class SLMFastMCP(FastMCP):
41
42
  """FastMCP with SLM release identity and deterministic SSE cleanup."""
42
43
 
43
- def __init__(self, *args, product_version: str = "3.7.1", **kwargs) -> None:
44
+ def __init__(self, *args, product_version: str = __version__, **kwargs) -> None:
44
45
  super().__init__(*args, **kwargs)
45
46
  # FastMCP delegates the initialize response to the low-level MCP
46
47
  # server. Without an explicit value it reports the installed ``mcp``
@@ -12,12 +12,13 @@ from fastapi import APIRouter
12
12
  from fastapi.requests import Request
13
13
  from fastapi.responses import Response
14
14
 
15
+ from superlocalmemory import __version__
15
16
  from superlocalmemory.optimize.config.schema import OptimizeConfig
16
17
  from superlocalmemory.optimize.proxy.lifecycle import HookChain
17
18
 
18
19
  logger = logging.getLogger("slm.optimize.proxy")
19
20
 
20
- _PROXY_VERSION = "3.7.1"
21
+ _PROXY_VERSION = __version__
21
22
  _REQUEST_TIMEOUT_S = 300.0
22
23
  _CONNECT_TIMEOUT_S = 10.0
23
24
  _MAX_CONNECTIONS = 100
@@ -43,6 +43,7 @@ from pathlib import Path
43
43
  from typing import Any
44
44
 
45
45
  from fastapi import APIRouter, Depends, HTTPException, Request
46
+ from superlocalmemory import __version__
46
47
 
47
48
  from superlocalmemory.core.security_primitives import (
48
49
  redact_secrets,
@@ -64,7 +65,7 @@ router = APIRouter(prefix="/api/v3", tags=["brain"])
64
65
  # LLD-03 v2 stratum space = 4 query types × 3 entity bins × 4 time buckets.
65
66
  _STRATA_TOTAL: int = 48
66
67
 
67
- _VERSION: str = "3.7.1"
68
+ _VERSION: str = __version__
68
69
 
69
70
  # Banned metric names (LLD-04 U4). Kept as a tuple for grep visibility;
70
71
  # the source-level test asserts we don't accidentally reintroduce them.