superlocalmemory 3.7.3 → 3.7.5

package/CHANGELOG.md

@@ -5,6 +5,23 @@ All notable changes to SuperLocalMemory V3 will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [3.7.5] - 2026-07-18 — Complete Scale Engine projection parity
+
+### Improved
+
+- Scale Engine vector preparation now reads the supported sqlite-vec virtual-table contract and joins vectors through canonical embedding metadata.
+- LanceDB projection imports are profile-scoped, lifecycle-aware, and processed in bounded batches to keep migration memory stable.
+- Scale verification and fingerprints now cover the same canonical vector rows that are written to LanceDB, so a partial vector projection cannot pass parity.
+
+## [3.7.4] - 2026-07-18 — Scale Engine projection-parity release
+
+### Fixed
+
+- Aligned CozoDB projection parity with SLM's logical graph-edge identity while retaining the strongest relationship weight.
+- Preserved canonical SQLite history when normalizing legacy repeated graph rows into derived scale projections.
+- Kept rejected projection manifests inspectable, retired replaceable stage payloads, and allowed a corrected, explicitly confirmed adoption retry.
+- Removed stale CozoDB graph edges when a fact is deleted and kept shadow-error telemetry side-effect free.
+
 ## [3.7.3] - 2026-07-18 — Scale Engine integrity release
 
 ### Fixed

package/README.md

@@ -5,10 +5,10 @@
   </picture>
 </p>
 
-<h1 align="center">SuperLocalMemory V3.7.3</h1>
+<h1 align="center">SuperLocalMemory V3.7.5</h1>
 <p align="center"><strong>Cache. Compress. Remember. Three surfaces — proxy, MCP tools, or skill. Every setup covered.</strong><br/>
 <em>Local-first agent memory with explicit operating modes, auditable retrieval, and optional Optimize tools.</em></p>
-<p align="center"><code>v3.7.3</code> — <strong>Scale Engine integrity release: explicit legacy adoption, durable promotion recovery, and honest runtime status.</strong><br/>
+<p align="center"><code>v3.7.5</code> — <strong>Scale Engine graph and vector projection parity, durable promotion recovery, and honest runtime status.</strong><br/>
 Proxy: <code>slm wrap claude</code> &nbsp;·&nbsp; MCP: add <code>slm_compress</code> to your config &nbsp;·&nbsp; Skill: zero-config</p>
 <p align="center"><strong>3 public research preprints</strong> (arXiv + Zenodo archives) · <a href="https://arxiv.org/abs/2603.02240">arXiv:2603.02240</a> · <a href="https://arxiv.org/abs/2603.14588">arXiv:2603.14588</a> · <a href="https://arxiv.org/abs/2604.04514">arXiv:2604.04514</a></p>
 
@@ -111,7 +111,7 @@ health surfaces expose the stages actually completed by the installed runtime.
 | **Retrieval and recall** | Semantic, lexical, temporal, Hopfield and spreading-activation candidate channels; RRF fusion, optional reranking and graph score enhancement | Healthy channels participate; response provenance states the evidence used. |
 | **Brain and learning** | Behavioral patterns, feedback/outcome records, rewards, consolidation, LightGBM-related ranking components, soft prompts, and guarded skill-evolution workflows | Learning is evidence-driven; it does not claim autonomous correctness or guaranteed improvement. |
 | **Knowledge graph and entities** | Canonical entities, aliases, entity profiles, graph edges, scenes, timelines, explorer and graph APIs | Stored/derived graph data is evidence, not an instruction authority. |
-| **Scale Engine** | SQLite + sqlite-vec are canonical. CozoDB graph and LanceDB vector projections are managed with prepare → verify → promote → rollback; a structurally detected pre-v3.7 projection can be explicitly adopted. | Promotion is parity-gated and crash-recoverable. Legacy adoption preserves the prior projection as a rollback backup; if proof fails, routing remains on canonical SQLite and status reports repair required. |
+| **Scale Engine** | SQLite + sqlite-vec are canonical. CozoDB graph and LanceDB vector projections are managed with prepare → verify → promote → rollback; a structurally detected pre-v3.7 projection can be explicitly adopted. | Promotion is parity-gated and crash-recoverable. Legacy adoption preserves the prior projection as a rollback backup; repeated physical edge rows normalize to one logical edge with the strongest weight. |
 | **Optimize** | Exact cache, tagged invalidation, safe compression, opt-in aggressive prose compression, CCR originals, proxy/MCP/skill surfaces | Only proxy intercepts a primary provider turn. MCP/skill cache results explicitly routed through SLM. |
 | **Mesh** | Authenticated peer messages, inbox/outbox, locks, offline queue, optional discovery and mesh MCP tools | Mesh is coordination, not automatic replicated memory or conflict resolution. |
 | **Governance and operations** | Provenance, audit/retention/policy surfaces, export/erasure controls, diagnostics, health, backups and daemon lifecycle | These are engineering controls, not a legal certification. |
@@ -238,7 +238,8 @@ SLM then rebuilds from canonical SQLite, verifies it, and promotes it with a
 durable recovery journal while retaining the prior directories as a rollback
 backup. `adopt` reports `restart_required: true`; run `slm restart` before
 checking daemon health. If proof fails, recall remains on SQLite and status
-reports repair required.
+retains the rejected manifest for inspection, retires its replaceable derived
+payload, and allows a corrected retry.
 
 Canonical ingestion is a durable state machine: `raw → queryable → enriching →
 complete`, with `failed` retaining raw evidence, error details, attempt count,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "superlocalmemory",
-  "version": "3.7.3",
+  "version": "3.7.5",
   "description": "Local-first agent memory with MCP and an agent-native CLI. Documented clients include Claude Code, Cursor, and Windsurf.",
   "keywords": [
     "ai-memory",

package/plugin/.claude-plugin/plugin.json

@@ -16,5 +16,5 @@
   "mcpServers": "./.mcp.json",
   "name": "superlocalmemory",
   "repository": "https://github.com/qualixar/superlocalmemory",
-  "version": "3.7.3"
+  "version": "3.7.5"
 }

package/plugin/requirements.txt

@@ -1 +1 @@
-superlocalmemory==3.7.3
+superlocalmemory==3.7.5

package/plugin-src/manifest.json

@@ -1,5 +1,5 @@
 {
-  "version": "3.7.3",
+  "version": "3.7.5",
   "pluginName": "superlocalmemory",
   "displayName": "SuperLocalMemory",
   "repository": "https://github.com/qualixar/superlocalmemory",

package/plugin-src/requirements.txt

@@ -1 +1 @@
-superlocalmemory==3.7.3
+superlocalmemory==3.7.5

package/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "superlocalmemory"
-version = "3.7.3"
+version = "3.7.5"
 description = "Local-first agent memory with auditable hybrid retrieval"
 readme = "README.md"
 license = "AGPL-3.0-or-later"

package/src/superlocalmemory/__init__.py

@@ -32,7 +32,7 @@ if "OMP_NUM_THREADS" not in os.environ:
     os.environ["OMP_NUM_THREADS"] = "2"
 # ---------------------------------------------------------------------------
 
-__version__ = "3.7.3"
+__version__ = "3.7.5"
 
 _REQUIRED_VERSIONS = {
     "sentence_transformers": "5.3.0",

package/src/superlocalmemory/cli/scale_engine_cmd.py

@@ -8,6 +8,7 @@ from argparse import Namespace
 def cmd_db_scale(args: Namespace) -> int:
     from superlocalmemory.core.config import SLMConfig
     from superlocalmemory.core.scale_engine import ScaleEngineError, ScaleEngineManager
+    from superlocalmemory.storage.sqlite_vectors import CanonicalVectorError
 
     action = args.scale_action
     # Scale Engine projections are currently canonical-default-profile data.
@@ -42,7 +43,7 @@ def cmd_db_scale(args: Namespace) -> int:
             result = manager.rollback(args.backup_id)
         else:
             raise ScaleEngineError(f"unknown Scale Engine action: {action}")
-    except ScaleEngineError as exc:
+    except (ScaleEngineError, CanonicalVectorError) as exc:
         print(f"Scale Engine: {exc}")
         return 1
     print(json.dumps(result, indent=2, sort_keys=True))

package/src/superlocalmemory/core/scale_engine.py

@@ -22,6 +22,14 @@ from datetime import UTC, datetime
 from pathlib import Path
 from typing import Any, Callable
 
+from superlocalmemory.storage.logical_edges import count_logical_edges, iter_logical_edges
+from superlocalmemory.storage.sqlite_vectors import (
+    CanonicalVectorError,
+    count_canonical_vectors,
+    iter_canonical_vectors,
+    load_sqlite_vec_extension,
+)
+
 
 class ScaleEngineError(RuntimeError):
     """A scale projection cannot safely advance to its next lifecycle state."""
@@ -98,8 +106,8 @@ class ScaleEngineManager:
         legacy_projection_candidate = (
             state == "local_core"
             and all(paths_present.values())
-            and not manifests
             and not backups
+            and not self.promotion_journal_path.exists()
             and self._has_legacy_projection_layout()
         )
         runtime = self._runtime_backend_status()
@@ -117,7 +125,17 @@ class ScaleEngineManager:
             "legacy_candidate_requires_confirmation": legacy_projection_candidate,
             "migration_repair_required": (
                 self.promotion_journal_path.exists()
-                or (state == "local_core" and bool(manifests))
+                or any(
+                    manifest.get("state") == "promoted" and state != "promoted"
+                    for manifest in manifests
+                )
+                or (
+                    state == "promoted"
+                    and any(
+                        manifest.get("state") in {"prepared", "verified"}
+                        for manifest in manifests
+                    )
+                )
             ),
             "stages": manifests,
             "backups": backups,
@@ -135,26 +153,91 @@ class ScaleEngineManager:
         if not self.status()["legacy_projection_candidate"]:
             return None
         lock_path = self._acquire_lifecycle_lock()
+        prepared: dict[str, Any] | None = None
         try:
             # Re-check inside the lock. A concurrent command may have
             # completed promotion while this caller waited to acquire it.
-            if not self.status()["legacy_projection_candidate"]:
+            current_status = self.status()
+            if not current_status["legacy_projection_candidate"]:
                 return None
+            retry_payloads = [
+                manifest["stage_id"]
+                for manifest in current_status["stages"]
+                if manifest.get("state") in {"prepared", "verified"}
+                and manifest.get("stage_id")
+            ]
             prepared = self._prepare()
             self._verify(prepared["stage_id"])
-            return self._promote(prepared["stage_id"])
-        except Exception:
+            promoted = self._promote(prepared["stage_id"])
+            retired: list[str] = []
+            retirement_failures: dict[str, str] = {}
+            for stage_id in retry_payloads:
+                try:
+                    self._retire_superseded_stage(stage_id)
+                    retired.append(stage_id)
+                except Exception as cleanup_exc:
+                    retirement_failures[stage_id] = str(cleanup_exc)
+            return {
+                **promoted,
+                "retired_stages": retired,
+                "retirement_failures": retirement_failures,
+            }
+        except Exception as exc:
             # A failed adoption must leave the canonical path selected.  The
-            # stage is retained for operator inspection instead of deleting
-            # evidence about why an upgrade could not be completed.
-            self.config.scale_engine_state = "local_core"
+            # manifest is retained for inspection, but its replaceable Cozo
+            # and Lance payloads are retired so repeated retries cannot grow
+            # the data root without bound.
+            retirement_error: Exception | None = None
+            unresolved_promotion = self.promotion_journal_path.exists()
+            if prepared is not None and not unresolved_promotion:
+                try:
+                    self._retire_rejected_stage(prepared["stage_id"], exc)
+                except Exception as cleanup_exc:
+                    retirement_error = cleanup_exc
+            self.config.scale_engine_state = (
+                "verified" if unresolved_promotion else "local_core"
+            )
             self.config.graph_backend = "auto"
             self.config.vector_backend = "auto"
             self._save_config()
+            if retirement_error is not None:
+                raise ScaleEngineError(
+                    f"{exc}; rejected stage retirement failed: {retirement_error}"
+                ) from exc
             raise
         finally:
             self._release_lifecycle_lock(lock_path)
 
+    def _retire_rejected_stage(self, stage_id: str, error: Exception) -> None:
+        """Keep rejection evidence while removing replaceable projection data."""
+        self._retire_stage_payload(
+            stage_id,
+            {
+                "state": "rejected",
+                "rejected_at": _utc_now(),
+                "failure": f"{type(error).__name__}: {error}",
+            },
+        )
+
+    def _retire_superseded_stage(self, stage_id: str) -> None:
+        """Retain an old retry manifest after a newer projection is promoted."""
+        self._retire_stage_payload(
+            stage_id,
+            {"state": "superseded", "superseded_at": _utc_now()},
+        )
+
+    def _retire_stage_payload(
+        self, stage_id: str, manifest_updates: dict[str, Any]
+    ) -> None:
+        """Remove derived stage bytes while retaining its durable manifest."""
+        stage_dir, manifest = self._load_stage(stage_id)
+        for payload in (stage_dir / "cozo", stage_dir / "lance"):
+            if payload.exists():
+                shutil.rmtree(payload)
+        self._fsync_directory(stage_dir)
+        manifest.update(manifest_updates)
+        self._write_manifest(stage_dir, manifest)
+
     def prepare(self) -> dict[str, Any]:
         """Build a new projection in a private staging directory."""
         lock_path = self._acquire_lifecycle_lock()
@@ -195,6 +278,9 @@ class ScaleEngineManager:
             self.config.scale_engine_state = "prepared"
             self._save_config()
             return manifest
+        except CanonicalVectorError as exc:
+            shutil.rmtree(stage_dir, ignore_errors=True)
+            raise ScaleEngineError(f"canonical vector projection failed: {exc}") from exc
         except Exception:
             shutil.rmtree(stage_dir, ignore_errors=True)
             raise
@@ -208,6 +294,8 @@ class ScaleEngineManager:
         try:
             self._recover_interrupted_promotion()
             return self._verify(stage_id)
+        except CanonicalVectorError as exc:
+            raise ScaleEngineError(f"canonical vector projection failed: {exc}") from exc
         finally:
             self._release_lifecycle_lock(lock_path)
 
@@ -224,7 +312,9 @@ class ScaleEngineManager:
                 source_fingerprint = self._projection_fingerprint(conn, canonical)
             observed = self._observed_counts(cozo, lance)
             if manifest["source_fingerprint"] != source_fingerprint:
-                raise ScaleEngineError("canonical SQLite changed after preparation; prepare a new stage")
+                raise ScaleEngineError(
+                    "canonical SQLite changed after preparation; prepare a new stage"
+                )
             if canonical != manifest["canonical"] or observed != canonical:
                 raise ScaleEngineError(
                     f"projection parity failed: canonical={canonical}, observed={observed}"
@@ -244,6 +334,8 @@ class ScaleEngineManager:
         try:
             self._recover_interrupted_promotion()
             return self._promote(stage_id)
+        except CanonicalVectorError as exc:
+            raise ScaleEngineError(f"canonical vector projection failed: {exc}") from exc
         finally:
             self._release_lifecycle_lock(lock_path)
 
@@ -264,7 +356,9 @@ class ScaleEngineManager:
             gate.execute("BEGIN IMMEDIATE")
             canonical = self._canonical_counts(gate)
             if manifest["source_fingerprint"] != self._projection_fingerprint(gate, canonical):
-                raise ScaleEngineError("canonical SQLite changed after verification; prepare a new stage")
+                raise ScaleEngineError(
+                    "canonical SQLite changed after verification; prepare a new stage"
+                )
             self._mkdir_durable(self.backup_root)
             journal = {
                 "schema_version": self.SCHEMA_VERSION,
@@ -291,7 +385,13 @@ class ScaleEngineManager:
                 self._replace_durable(destination, source)
                 move["state"] = "complete"
                 self._write_promotion_journal(journal)
-            manifest.update({"state": "promoted", "promoted_at": _utc_now(), "backup_id": backup_dir.name})
+            manifest.update(
+                {
+                    "state": "promoted",
+                    "promoted_at": _utc_now(),
+                    "backup_id": backup_dir.name,
+                }
+            )
             self._write_manifest(stage_dir, manifest)
             self.config.scale_engine_state = "promoted"
             self.config.graph_backend = "cozo"
@@ -308,11 +408,14 @@ class ScaleEngineManager:
             except sqlite3.Error:
                 pass
             try:
-                self._recover_interrupted_promotion()
-            except ScaleEngineError as recovery_error:
+                recovery = self._recover_interrupted_promotion()
+            except Exception as recovery_error:
                 raise ScaleEngineError(
                     f"promotion interrupted; automatic recovery needs repair: {recovery_error}"
                 ) from exc
+            if recovery == "finalized_committed_promotion":
+                _, recovered_manifest = self._load_stage(stage_id)
+                return recovered_manifest
             raise ScaleEngineError(f"promotion rolled back: {exc}") from exc
         finally:
             gate.close()
@@ -367,7 +470,11 @@ class ScaleEngineManager:
             journal["state"] = "committed"
             self._write_promotion_journal(journal)
             self.promotion_journal_path.unlink(missing_ok=True)
-            return {"state": "local_core", "restored_backup": backup_id, "displaced": displaced.name}
+            return {
+                "state": "local_core",
+                "restored_backup": backup_id,
+                "displaced": displaced.name,
+            }
         except Exception as exc:
             try:
                 self._recover_interrupted_promotion()
@@ -385,6 +492,7 @@ class ScaleEngineManager:
     def _readonly_connection(self) -> sqlite3.Connection:
         conn = sqlite3.connect(f"file:{self.db_path}?mode=ro", uri=True)
         conn.execute("PRAGMA query_only=ON")
+        load_sqlite_vec_extension(conn)
         return conn
 
     def _canonical_counts(self, conn: sqlite3.Connection) -> dict[str, int]:
@@ -392,18 +500,8 @@ class ScaleEngineManager:
             "SELECT COUNT(*) FROM canonical_entities WHERE profile_id=?",
             (self.profile_id,),
         ).fetchone()[0]
-        edges = conn.execute(
-            "SELECT COUNT(*) FROM graph_edges WHERE profile_id=?", (self.profile_id,)
-        ).fetchone()[0]
-        try:
-            vectors = conn.execute(
-                "SELECT COUNT(*) FROM fact_embeddings_rowids fer "
-                "JOIN atomic_facts af ON af.fact_id = fer.fact_id "
-                "WHERE af.profile_id = ?",
-                (self.profile_id,),
-            ).fetchone()[0]
-        except sqlite3.OperationalError:
-            vectors = 0
+        edges = count_logical_edges(conn, self.profile_id)
+        vectors = count_canonical_vectors(conn, self.profile_id)
         return {"entities": int(nodes), "edges": int(edges), "vectors": int(vectors)}
 
     def _observed_counts(self, cozo: Any, lance: Any) -> dict[str, int]:
@@ -411,7 +509,11 @@ class ScaleEngineManager:
         vector = lance.health_check()
         if graph.get("status") != "active" or vector.get("status") != "active":
             raise ScaleEngineError(f"projection health failed: cozo={graph}, lancedb={vector}")
-        return {"entities": int(graph["entities"]), "edges": int(graph["edges"]), "vectors": int(vector["vectors"])}
+        return {
+            "entities": int(graph["entities"]),
+            "edges": int(graph["edges"]),
+            "vectors": int(vector["vectors"]),
+        }
 
     def _projection_fingerprint(
         self, conn: sqlite3.Connection, counts: dict[str, int]
@@ -420,9 +522,13 @@ class ScaleEngineManager:
         digest = hashlib.sha256()
         digest.update(json.dumps(counts, sort_keys=True).encode())
         tables = (
-            ("canonical_entities", "entity_id, canonical_name, entity_type, first_seen, last_seen, fact_count, profile_id", "entity_id"),
+            (
+                "canonical_entities",
+                "entity_id, canonical_name, entity_type, first_seen, last_seen, "
+                "fact_count, profile_id",
+                "entity_id",
+            ),
             ("atomic_facts", "fact_id, canonical_entities_json, lifecycle, profile_id", "fact_id"),
-            ("graph_edges", "source_id, target_id, edge_type, weight, profile_id", "source_id, target_id, edge_type"),
         )
         for table, columns, ordering in tables:
             try:
@@ -435,17 +541,12 @@ class ScaleEngineManager:
             except sqlite3.OperationalError as exc:
                 raise ScaleEngineError(f"canonical SQLite missing required {table} table") from exc
         try:
-            rows = conn.execute(
-                "SELECT fer.rowid, fer.fact_id, vec.vector FROM fact_embeddings_rowids fer "
-                "JOIN atomic_facts af ON af.fact_id = fer.fact_id "
-                "LEFT JOIN fact_embeddings_vector_chunks00 vec ON vec.rowid = fer.rowid "
-                "WHERE af.profile_id=? ORDER BY fer.rowid",
-                (self.profile_id,),
-            )
-            for row in rows:
-                self._digest_row(digest, "fact_embeddings_rowids", row)
-        except sqlite3.OperationalError:
-            pass
+            for row in iter_logical_edges(conn, self.profile_id):
+                self._digest_row(digest, "graph_edges", row)
+        except sqlite3.OperationalError as exc:
+            raise ScaleEngineError("canonical SQLite missing required graph_edges table") from exc
+        for row in iter_canonical_vectors(conn, self.profile_id):
+            self._digest_row(digest, "fact_embeddings", row)
         return digest.hexdigest()
 
     @staticmethod
@@ -494,7 +595,9 @@ class ScaleEngineManager:
 
     def _require_default_profile(self) -> None:
         if self.profile_id != "default":
-            raise ScaleEngineError("Scale Engine promotion currently supports the default profile only")
+            raise ScaleEngineError(
+                "Scale Engine promotion currently supports the default profile only"
+            )
 
     def _require_canonical_db(self) -> None:
         if not self.db_path.exists():

package/src/superlocalmemory/graph/cozo_backend.py

@@ -24,6 +24,8 @@ from datetime import datetime
 from pathlib import Path
 from typing import Any
 
+from superlocalmemory.storage.logical_edges import iter_logical_edges
+
 logger = logging.getLogger(__name__)
 
 # Optional import — CozoDB is an optional dependency
@@ -260,6 +262,12 @@ class CozoDBGraphBackend:
                 *fact_entity{fact_id, entity_id, profile_id}, fact_id = $fact_id
             :rm fact_entity {fact_id, entity_id => profile_id}
         """, {"fact_id": fact_id})
+        self._db.run("""
+            ?[from_id, to_id, edge_type, weight, metadata, profile_id, created_at] :=
+                *edge{from_id, to_id, edge_type, weight, metadata, profile_id, created_at},
+                (from_id = $fact_id or to_id = $fact_id)
+            :rm edge {from_id, to_id, edge_type => weight, metadata, profile_id, created_at}
+        """, {"fact_id": fact_id})
 
     def record_shadow_comparison(
         self,
@@ -280,12 +288,6 @@ class CozoDBGraphBackend:
     def record_shadow_error(self, error: str) -> None:
         self._shadow_errors += 1
         logger.warning("Cozo entity recall failed closed to SQLite: %s", error)
-        self._db.run("""
-            ?[from_id, to_id, edge_type, weight, metadata, profile_id, created_at] :=
-                *edge{from_id, to_id, edge_type, weight, metadata, profile_id, created_at},
-                (from_id = $fact_id or to_id = $fact_id)
-            :rm edge {from_id, to_id, edge_type => weight, metadata, profile_id, created_at}
-        """, {"fact_id": fact_id})
 
     # ------------------------------------------------------------------
     # Bulk Import (SQLite → CozoDB)
@@ -358,22 +360,15 @@ class CozoDBGraphBackend:
 
         # Step 3: Export fact graph edges directly.  Fact graph traversal is
         # intentionally kept in its native fact-ID namespace.
-        edges_sql = """
-            SELECT source_id, target_id, edge_type, weight
-            FROM graph_edges WHERE profile_id = ?
-        """
-        edge_rows = conn.execute(edges_sql, (profile_id,)).fetchall()
-
         edge_dicts = []
-        for row in edge_rows:
-            ea, eb, etype, weight = row
+        for ea, eb, etype, weight, edge_profile in iter_logical_edges(conn, profile_id):
             edge_dicts.append({
                 "from_id": ea,
                 "to_id": eb,
-                "edge_type": etype or "related",
-                "weight": float(weight or 1.0),
+                "edge_type": etype,
+                "weight": float(weight),
                 "metadata": "{}",
-                "profile_id": profile_id,
+                "profile_id": edge_profile,
                 "created_at": now,
             })
 

package/src/superlocalmemory/storage/logical_edges.py

@@ -0,0 +1,45 @@
+"""Canonical logical-edge projection shared by scale backends.
+
+Legacy databases can contain multiple physical ``graph_edges`` rows for one
+logical relationship. Current writes define identity as profile, source,
+target, and edge type, retaining the strongest weight. Derived projections
+must use that same contract without rewriting canonical SQLite history.
+"""
+from __future__ import annotations
+
+import sqlite3
+from collections.abc import Iterator
+from typing import Any
+
+_LOGICAL_EDGE_SELECT = """
+    SELECT
+        source_id,
+        target_id,
+        COALESCE(edge_type, 'related') AS edge_type,
+        MAX(COALESCE(weight, 1.0)) AS weight,
+        profile_id
+    FROM graph_edges
+    WHERE profile_id = ?
+    GROUP BY profile_id, source_id, target_id, COALESCE(edge_type, 'related')
+"""
+
+
+def iter_logical_edges(
+    conn: sqlite3.Connection, profile_id: str
+) -> Iterator[tuple[Any, ...]]:
+    """Yield normalized graph edges in deterministic fingerprint order."""
+    return iter(
+        conn.execute(
+            _LOGICAL_EDGE_SELECT + " ORDER BY source_id, target_id, edge_type",
+            (profile_id,),
+        )
+    )
+
+
+def count_logical_edges(conn: sqlite3.Connection, profile_id: str) -> int:
+    """Count relationships using the canonical logical identity."""
+    row = conn.execute(
+        "SELECT COUNT(*) FROM (" + _LOGICAL_EDGE_SELECT + ")",
+        (profile_id,),
+    ).fetchone()
+    return int(row[0] if row else 0)

package/src/superlocalmemory/storage/sqlite_vectors.py

@@ -0,0 +1,169 @@
+# Copyright (c) 2026 Varun Pratap Bhardwaj / Qualixar
+# Licensed under AGPL-3.0-or-later - see LICENSE file
+
+"""Canonical sqlite-vec export contract for derived vector projections."""
+
+from __future__ import annotations
+
+import sqlite3
+from collections.abc import Iterator
+from typing import TypeAlias
+
+
+class CanonicalVectorError(RuntimeError):
+    """Canonical sqlite-vec data cannot be read without risking data loss."""
+
+
+CanonicalVector: TypeAlias = tuple[int, str, str, str, bytes]
+
+
+def load_sqlite_vec_extension(conn: sqlite3.Connection) -> None:
+    """Load sqlite-vec for this connection or fail the migration explicitly."""
+    try:
+        import sqlite_vec
+
+        conn.enable_load_extension(True)
+        sqlite_vec.load(conn)
+    except Exception as exc:
+        raise CanonicalVectorError(
+            "sqlite-vec could not be loaded; refusing to treat canonical vectors as empty"
+        ) from exc
+    finally:
+        try:
+            conn.enable_load_extension(False)
+        except (AttributeError, sqlite3.Error):
+            pass
+
+
+def count_canonical_vectors(conn: sqlite3.Connection, profile_id: str) -> int:
+    """Count profile-scoped vectors that have a canonical fact identity."""
+    if not _canonical_vector_table_exists(conn):
+        return 0
+    load_sqlite_vec_extension(conn)
+    return _validate_canonical_vector_contract(conn, profile_id)
+
+
+def iter_canonical_vectors(
+    conn: sqlite3.Connection, profile_id: str
+) -> Iterator[CanonicalVector]:
+    """Yield supported vec0 rows joined to canonical fact identity and lifecycle."""
+    if not _canonical_vector_table_exists(conn):
+        return
+    load_sqlite_vec_extension(conn)
+    _validate_canonical_vector_contract(conn, profile_id)
+    try:
+        rows = conn.execute(
+            "SELECT fe.rowid, em.fact_id, COALESCE(af.lifecycle, 'active'), "
+            "af.profile_id, fe.embedding "
+            "FROM fact_embeddings fe "
+            "JOIN embedding_metadata em ON em.vec_rowid = fe.rowid "
+            "JOIN atomic_facts af ON af.fact_id = em.fact_id "
+            "AND af.profile_id = em.profile_id "
+            "WHERE af.profile_id = ? AND fe.profile_id = af.profile_id "
+            "ORDER BY fe.rowid",
+            (profile_id,),
+        )
+        for rowid, fact_id, lifecycle, row_profile_id, blob in rows:
+            if not isinstance(blob, bytes):
+                raise CanonicalVectorError(
+                    f"canonical vector {rowid} is not a float32 blob"
+                )
+            yield int(rowid), str(fact_id), str(lifecycle), str(row_profile_id), blob
+    except sqlite3.Error as exc:
+        raise CanonicalVectorError(
+            "canonical sqlite-vec rows are unreadable; refusing a partial projection"
+        ) from exc
+
+
+def _validate_canonical_vector_contract(
+    conn: sqlite3.Connection, profile_id: str
+) -> int:
+    """Prove every profile-owned metadata row maps to one vec0 partition row."""
+    try:
+        misowned = int(
+            conn.execute(
+                "SELECT COUNT(*) FROM embedding_metadata em "
+                "LEFT JOIN atomic_facts af ON af.fact_id = em.fact_id "
+                "WHERE (em.profile_id = ? OR af.profile_id = ?) "
+                "AND (af.fact_id IS NULL OR af.profile_id <> em.profile_id)",
+                (profile_id, profile_id),
+            ).fetchone()[0]
+        )
+        expected = int(
+            conn.execute(
+                "SELECT COUNT(*) FROM embedding_metadata em "
+                "JOIN atomic_facts af ON af.fact_id = em.fact_id "
+                "AND af.profile_id = em.profile_id "
+                "WHERE af.profile_id = ?",
+                (profile_id,),
+            ).fetchone()[0]
+        )
+        mapped = int(
+            conn.execute(
+                "SELECT COUNT(*) FROM embedding_metadata em "
+                "JOIN atomic_facts af ON af.fact_id = em.fact_id "
+                "AND af.profile_id = em.profile_id "
+                "JOIN fact_embeddings fe ON fe.rowid = em.vec_rowid "
+                "AND fe.profile_id = af.profile_id "
+                "WHERE af.profile_id = ?",
+                (profile_id,),
+            ).fetchone()[0]
+        )
+    except sqlite3.Error as exc:
+        raise CanonicalVectorError(
+            "canonical vector contract is unreadable; refusing a partial projection"
+        ) from exc
+    if misowned:
+        raise CanonicalVectorError(
+            f"canonical vector ownership mismatch for {misowned} metadata row(s)"
+        )
+    if mapped != expected:
+        raise CanonicalVectorError(
+            "canonical vector mapping is incomplete: "
+            f"metadata={expected}, mapped_vec0={mapped}"
+        )
+    return expected
+
+
+def _canonical_vector_table_exists(conn: sqlite3.Connection) -> bool:
+    row = conn.execute(
+        "SELECT 1 FROM sqlite_master WHERE name='fact_embeddings' "
+        "AND type='table'"
+    ).fetchone()
+    if row is not None:
+        return True
+    metadata = conn.execute(
+        "SELECT 1 FROM sqlite_master WHERE name='embedding_metadata' "
+        "AND type='table'"
+    ).fetchone()
+    if metadata is not None:
+        try:
+            metadata_count = int(
+                conn.execute("SELECT COUNT(*) FROM embedding_metadata").fetchone()[0]
+            )
+        except sqlite3.Error as exc:
+            raise CanonicalVectorError("embedding metadata is unreadable") from exc
+        if metadata_count:
+            raise CanonicalVectorError(
+                "embedding metadata exists but fact_embeddings is missing"
+            )
+    shadow_payload = 0
+    for table in ("fact_embeddings_rowids", "fact_embeddings_vector_chunks00"):
+        exists = conn.execute(
+            "SELECT 1 FROM sqlite_master WHERE name=? AND type='table'",
+            (table,),
+        ).fetchone()
+        if exists is not None:
+            try:
+                shadow_payload += int(
+                    conn.execute(f'SELECT COUNT(*) FROM "{table}"').fetchone()[0]
+                )
+            except sqlite3.Error as exc:
+                raise CanonicalVectorError(
+                    "sqlite-vec shadow payload is unreadable"
+                ) from exc
+    if shadow_payload:
+        raise CanonicalVectorError(
+            "sqlite-vec shadow payload exists without the fact_embeddings virtual table"
+        )
+    return False

package/src/superlocalmemory/vector/lancedb_backend.py

@@ -20,6 +20,8 @@ import struct
 from pathlib import Path
 from typing import Any
 
+from superlocalmemory.storage.sqlite_vectors import iter_canonical_vectors
+
 logger = logging.getLogger(__name__)
 
 # Optional import
@@ -178,80 +180,49 @@ class LanceDBVectorBackend:
     ) -> int:
         """Export embeddings from sqlite-vec → LanceDB.
 
-        sqlite-vec stores vectors as raw float32 little-endian blobs
-        in fact_embeddings_vector_chunks00, with rowid mapping in
-        fact_embeddings_rowids.
+        Reads the supported vec0 virtual table and joins row IDs through
+        ``embedding_metadata``. Shadow-table layouts are sqlite-vec internals
+        and must not be treated as a stable migration API.
 
         Returns number of vectors imported.
         """
-        # Get rowid → fact_id mapping
-        row_map: dict[int, str] = {}
-        try:
-            for row in conn.execute("SELECT rowid, fact_id FROM fact_embeddings_rowids"):
-                row_map[row[0]] = row[1]
-        except sqlite3.OperationalError:
-            logger.warning("fact_embeddings_rowids not found — no vectors to import")
-            return 0
-
-        # Get tiers
-        tier_map: dict[str, str] = {}
-        profile_map: dict[str, str] = {}
-        try:
-            for row in conn.execute(
-                "SELECT fact_id, COALESCE(lifecycle, 'active'), profile_id "
-                "FROM atomic_facts WHERE profile_id = ?",
-                (profile_id,),
-            ):
-                tier_map[row[0]] = row[1]
-                profile_map[row[0]] = row[2] or "default"
-        except sqlite3.OperationalError:
-            pass
-
-        # Read vectors from sqlite-vec
-        try:
-            rows = conn.execute(
-                "SELECT rowid, vector FROM fact_embeddings_vector_chunks00"
-            ).fetchall()
-        except sqlite3.OperationalError:
-            logger.warning("fact_embeddings_vector_chunks00 not found")
-            return 0
-
-        # Reconstruct and batch import
-        data = []
-        for rowid, blob in rows:
-            fact_id = row_map.get(rowid)
-            # The rowid mapping is global, but a staged Scale Engine is
-            # explicitly profile-scoped.  Do not import a foreign profile by
-            # giving it a default tier/profile below.
-            if fact_id is None or fact_id not in profile_map:
-                continue
+        imported = 0
+        batch: list[tuple[str, list[float], str, str]] = []
+        for rowid, fact_id, tier, row_profile_id, blob in iter_canonical_vectors(
+            conn, profile_id
+        ):
             try:
                 vector = self._decode_vector_blob(blob)
             except Exception as exc:
-                logger.warning("Failed to decode vector for rowid %d: %s", rowid, exc)
-                continue
-            tier = tier_map.get(fact_id, "active")
-            data.append({
-                "fact_id": fact_id,
-                "vector": vector,
-                "tier": tier,
-                "profile_id": profile_map.get(fact_id, profile_id),
-            })
-
-        if data:
-            by_profile: dict[str, list[dict[str, Any]]] = {}
-            for item in data:
-                by_profile.setdefault(item["profile_id"], []).append(item)
-            for record_profile_id, records in by_profile.items():
-                self.add_vectors(
-                    [item["fact_id"] for item in records],
-                    [item["vector"] for item in records],
-                    [item["tier"] for item in records],
-                    record_profile_id,
-                )
-
-        logger.info("LanceDB: imported %d vectors from sqlite-vec", len(data))
-        return len(data)
+                raise LanceDBError(
+                    f"Invalid canonical vector for rowid {rowid}: {exc}"
+                ) from exc
+            batch.append((fact_id, vector, tier, row_profile_id))
+            if len(batch) >= 256:
+                imported += self._flush_import_batch(batch)
+                batch.clear()
+        if batch:
+            imported += self._flush_import_batch(batch)
+
+        logger.info("LanceDB: imported %d vectors from sqlite-vec", imported)
+        return imported
+
+    def _flush_import_batch(
+        self, batch: list[tuple[str, list[float], str, str]]
+    ) -> int:
+        """Write one bounded-memory, single-profile canonical vector batch."""
+        by_profile: dict[str, list[tuple[str, list[float], str]]] = {}
+        for fact_id, vector, tier, profile_id in batch:
+            by_profile.setdefault(profile_id, []).append((fact_id, vector, tier))
+        imported = 0
+        for profile_id, records in by_profile.items():
+            imported += self.add_vectors(
+                [item[0] for item in records],
+                [item[1] for item in records],
+                [item[2] for item in records],
+                profile_id,
+            )
+        return imported
 
     def _decode_vector_blob(self, blob: bytes) -> list[float]:
         """Decode sqlite-vec BLOB to list of floats.