superlocalmemory 4.0.10 → 4.1.2

package/src/superlocalmemory/core/graph_metrics.py

@@ -0,0 +1,597 @@
+# Copyright (c) 2026 Varun Pratap Bhardwaj / Qualixar
+# Licensed under AGPL-3.0-or-later - see LICENSE file
+
+"""Whole-graph structural metrics for ``fact_importance``.
+
+WHAT THIS IS FOR
+----------------
+Recall multiplies a candidate's activation by ``min(1 + pagerank * 2, 2)`` at
+every hop and biases it toward the communities its query seeds belong to. Both
+numbers come from ``fact_importance``. A memory missing from that table is found
+by the graph walk and then ranked as though it had no position in the graph at
+all -- so the table's coverage is a recall-quality property, not a reporting
+nicety.
+
+WHY ONE WRITER
+--------------
+Two writers were filling this table with numbers that do not mean the same
+thing. The whole-graph pass (``core/graph_analyzer.py``) computes PageRank over
+every fact and edge. A second path compiled an entity's facts and, on finding no
+PageRank for them, computed its own over a ``LIMIT 50`` slice of facts sharing
+that one entity -- a near-clique of at most 50 nodes, which hands every member
+roughly ``1/n``. Measured on the author's store: the whole-graph pass produced a
+maximum of 0.008744 and a median of 0.000214, while the local pass wrote 0.1 --
+**eleven times the largest real score, and roughly 470x the median**. Those facts
+took the maximum hop boost the formula allows (1.2 against everyone else's
+1.0004) for no reason other than having shared an entity with a few others, and
+they were written with no community, so the community bias could not see them
+either. The signal was not stale, it was wrong.
+
+So: one function computes this table, and it computes it over the whole graph.
+
+WHICH ENGINE, AND WHY IT IS NOT THE GRAPH ENGINE
+-----------------------------------------------
+Two adapters compute the same function of the same edge set. In-process wins on
+measurement, so it is the default. Timed on a copy of the author's 208,151-edge
+store, each algorithm alone, warmed:
+
+    algorithm     in process   graph engine   winner
+    PageRank          0.36 s        0.38 s    tie
+    Louvain           1.98 s        5.17 s    in process, 2.6x
+    whole pass        2.33 s        5.56 s    in process, 2.4x
+
+An earlier reading of 1.88 s for ``nx.pagerank`` against 0.13 s was a cold scipy
+import counted as compute -- the third time a first-call warm-up has been
+misread as a backend difference in this release. Both engines are deterministic
+across runs (checked, not assumed) and agree to Spearman rho 0.996, so the choice
+is a cost decision and may be revisited by measurement, not by argument.
+
+The graph engine still earns its keep, just not here: reading the adjacency for
+one recall measured 395 ms against SQLite's 2,477 ms on the same store, 6.3x, and
+the gap widens with edge count. Those two figures were measured by hand on a copy
+of a 208,151-edge store and no test reproduces them -- see the table in
+``graph/cozo_adjacency`` for what was run. That is a latency win on the recall path; this
+pass is background work where 2 s versus 5 s buys nothing.
+
+Its ``pagerank()``/``community_detect()`` helpers are NOT used and could not have
+been: they take their node set from the ``entity`` relation and their edges from
+``edge``, which hold canonical entity IDs and fact IDs respectively.
+``pagerank()`` therefore indexes an entity-keyed dict with a fact ID, raises
+KeyError, and returns ``{}`` -- verified against the real store, where it
+returned nothing and ``community_detect`` returned 1,386 singleton communities of
+entities and not one fact. The adapter here calls the native
+``PageRank``/``CommunityDetectionLouvain`` fixed rules, which operate on whatever
+relation they are handed.
+
+WHAT COUNTS AS A NODE, AND AS AN EDGE
+-------------------------------------
+Nodes are the profile's *visible* facts, isolated ones included: a fact with no
+edges still needs a row, or the ranker treats "no position in the graph" and "not
+computed yet" as the same thing. Edges come from ``iter_logical_edges``, which
+already excludes any edge with a withheld or soft-deleted endpoint -- the same
+predicate the retrieval channel prunes its adjacency with. The previous
+whole-graph pass read ``graph_edges`` and ``atomic_facts`` raw, so it ranked
+1,299 unreturnable facts alongside real memories and diluted every real score.
+
+PageRank runs on the edge-connected subgraph and the isolated facts are then
+given the uniform teleport share ``(1 - damping) / N`` over the full node set,
+with the connected scores scaled to leave room for it. That convention lives
+here, in the port, so both engines produce numbers on one scale -- the ranker
+reads absolute values, so an engine swap that shifted the scale would silently
+re-tune every boost in the system.
+"""
+
+from __future__ import annotations
+
+import logging
+import sqlite3
+import time
+from dataclasses import dataclass, field
+from typing import Any
+
+from contextlib import contextmanager
+
+from superlocalmemory.storage.logical_edges import iter_logical_edges
+
+logger = logging.getLogger(__name__)
+
+
+@contextmanager
+def _short_connection(db: Any) -> Any:
+    """A connection held for one read or one write, never across the compute.
+
+    ``raw_connection`` takes the manager's write lock, so holding it around a
+    two-to-ten-second graph computation would stall every store for that long.
+    The pass therefore reads, releases, computes, and writes.
+
+    The consolidation cycle passes a minimal proxy that owns a bare sqlite3
+    connection and exposes ``execute`` only. Supporting it here is what keeps
+    this the single writer of the table; the alternative was a second
+    implementation for that one caller, which is how the two disagreeing
+    PageRanks happened in the first place.
+    """
+    # Resolved on the TYPE, and the fallback is checked with isinstance, for the
+    # reason written up in retrieval/scope_policy.py: a MagicMock fabricates any
+    # attribute you ask for, so ``getattr(db, "raw_connection")`` returns a
+    # callable whose context manager yields another MagicMock -- and iterating
+    # that raises, which this module's error path then reports as "could not read
+    # the graph" for a store that is perfectly readable. The suite passes exactly
+    # such a mock, and it cost a debugging round to notice the pass had not run
+    # rather than found nothing.
+    raw = getattr(type(db), "raw_connection", None)
+    if callable(raw):
+        with raw(db) as conn:
+            yield conn
+        return
+    conn = getattr(db, "_conn", None)
+    if not isinstance(conn, sqlite3.Connection):
+        raise TypeError(
+            f"{type(db).__name__} exposes no connection to compute metrics on"
+        )
+    yield conn
+    conn.commit()
+
+#: PageRank damping. Matches the previous whole-graph pass so scores stay
+#: comparable across the version that introduced this module.
+DEFAULT_DAMPING = 0.85
+
+#: Betweenness centrality is O(V*E). At 4k facts and 130k edges that is minutes,
+#: and at the 12k/208k store it is hours -- which is the likeliest reason the
+#: whole-graph pass had run exactly once in nine days. Nothing in retrieval reads
+#: ``bridge_score``; only the dashboard displays it. So it is computed under a
+#: node ceiling and reported as skipped above it, rather than being the reason
+#: PageRank never lands.
+BRIDGE_NODE_LIMIT = 1500
+
+#: Louvain returns a hierarchy of partitions per node. Level 0 is the coarsest,
+#: and on the author's store it yields 13 communities against the 11
+#: meaningfully-sized ones the previous pass found -- so the community bias keeps
+#: the granularity it was tuned against instead of being handed 210 fragments.
+LOUVAIN_LEVEL = 0
+
+#: A projection is only trustworthy if it agrees with the store. If Cozo's edge
+#: count for this profile differs from SQLite's by more than this fraction, the
+#: pass computes on SQLite instead and says so, rather than ranking the store's
+#: memories from a graph the store does not have.
+MAX_PROJECTION_DRIFT = 0.02
+
+
+@dataclass(frozen=True)
+class GraphMetricsReport:
+    """What one pass actually did. No field here is inferred from another."""
+
+    profile_id: str
+    engine: str = "none"
+    facts: int = 0
+    edges: int = 0
+    connected: int = 0
+    isolated: int = 0
+    communities: int = 0
+    written: int = 0
+    removed: int = 0
+    bridges_computed: bool = False
+    duration_ms: int = 0
+    error: str | None = None
+    notes: tuple[str, ...] = field(default_factory=tuple)
+
+    @property
+    def ok(self) -> bool:
+        return self.error is None
+
+    def summary(self) -> str:
+        if self.error:
+            return f"graph metrics for {self.profile_id} FAILED: {self.error}"
+        return (
+            f"graph metrics for {self.profile_id}: {self.written} facts "
+            f"({self.connected} connected, {self.isolated} isolated), "
+            f"{self.edges} edges, {self.communities} communities, "
+            f"engine={self.engine}, {self.duration_ms} ms"
+        )
+
+
+# ----------------------------------------------------------------------
+# Engines. Each returns metrics for the edge-connected nodes only; the
+# isolated-node convention belongs to the caller, once, for both.
+# ----------------------------------------------------------------------
+
+
+def _cozo_metrics(
+    backend: Any, profile_id: str, damping: float
+) -> tuple[dict[str, float], dict[str, int]]:
+    """PageRank and Louvain from Cozo's native fixed rules.
+
+    Both queries filter on ``profile_id``: the projection holds every profile's
+    edges in one relation, and an unfiltered rule would rank one profile's
+    memories using another's graph.
+    """
+    client = backend._db  # the module-private client wrapper; no public accessor
+    pr_rows = client.run(
+        "rel[a, b, w] := *edge{from_id: a, to_id: b, weight: w, "
+        "                      profile_id: $pid}\n"
+        "?[node, score] <~ PageRank(rel[a, b, w], theta: $theta)",
+        {"pid": profile_id, "theta": damping},
+    )
+    pagerank = {str(r[0]): float(r[1]) for r in pr_rows.values.tolist()}
+
+    comm_rows = client.run(
+        "rel[a, b, w] := *edge{from_id: a, to_id: b, weight: w, "
+        "                      profile_id: $pid}\n"
+        "?[grp, node] <~ CommunityDetectionLouvain(rel[a, b, w])",
+        {"pid": profile_id},
+    )
+    communities: dict[str, int] = {}
+    for grp, node in comm_rows.values.tolist():
+        label = grp[LOUVAIN_LEVEL] if isinstance(grp, (list, tuple)) else grp
+        communities[str(node)] = int(label)
+    return pagerank, communities
+
+
+def _networkx_metrics(
+    edges: list[tuple[str, str, float]], damping: float
+) -> tuple[dict[str, float], dict[str, int]]:
+    """The same two metrics without a graph projection to read."""
+    import networkx as nx
+    from networkx.algorithms.community import louvain_communities
+
+    digraph = nx.DiGraph()
+    for source, target, weight in edges:
+        if digraph.has_edge(source, target):
+            if weight > digraph[source][target].get("weight", 0.0):
+                digraph[source][target]["weight"] = weight
+        else:
+            digraph.add_edge(source, target, weight=weight)
+    if digraph.number_of_nodes() == 0:
+        return {}, {}
+
+    pagerank = nx.pagerank(digraph, alpha=damping, weight="weight")
+    communities: dict[str, int] = {}
+    try:
+        partitions = louvain_communities(
+            digraph.to_undirected(), weight="weight", seed=42,
+        )
+        for label, members in enumerate(partitions):
+            for node in members:
+                communities[str(node)] = label
+    except Exception as exc:  # noqa: BLE001 -- a partition is optional, a score is not
+        logger.debug("Louvain partition unavailable: %s", exc)
+    return pagerank, communities
+
+
+def _bridge_scores(
+    edges: list[tuple[str, str, float]], node_count: int
+) -> dict[str, float] | None:
+    """Sampled betweenness, or None when the graph is too big to afford it."""
+    if node_count > BRIDGE_NODE_LIMIT or node_count <= 2:
+        return None
+    try:
+        import networkx as nx
+
+        graph = nx.DiGraph()
+        for source, target, weight in edges:
+            graph.add_edge(source, target, weight=weight)
+        return nx.betweenness_centrality(graph, weight="weight", normalized=True)
+    except Exception as exc:  # noqa: BLE001
+        logger.debug("Bridge scores unavailable: %s", exc)
+        return None
+
+
+# ----------------------------------------------------------------------
+# The pass
+# ----------------------------------------------------------------------
+
+
+def _visible_fact_ids(conn: sqlite3.Connection, profile_id: str) -> list[str]:
+    from superlocalmemory.storage.database import (
+        visible_fact_clause_for_connection,
+    )
+
+    clause = visible_fact_clause_for_connection(conn)
+    rows = conn.execute(
+        f"SELECT fact_id FROM atomic_facts WHERE profile_id = ?{clause}",
+        (profile_id,),
+    ).fetchall()
+    return [str(row[0]) for row in rows]
+
+
+def _projection_usable(
+    backend: Any, profile_id: str, sqlite_edges: int
+) -> tuple[bool, str]:
+    """Whether Cozo's edge set is close enough to the store's to rank from."""
+    if backend is None:
+        return False, "no graph projection"
+    try:
+        rows = backend._db.run(
+            "?[count(a)] := *edge{from_id: a, profile_id: $pid}",
+            {"pid": profile_id},
+        )
+        projected = int(rows.values.tolist()[0][0]) if len(rows) else 0
+    except Exception as exc:  # noqa: BLE001
+        return False, f"projection unreadable: {exc}"
+    if sqlite_edges == 0:
+        return projected == 0, "both empty"
+    drift = abs(projected - sqlite_edges) / float(sqlite_edges)
+    if drift > MAX_PROJECTION_DRIFT:
+        return False, (
+            f"projection drifted {drift:.1%} "
+            f"({projected} projected vs {sqlite_edges} stored)"
+        )
+    return True, f"projection within {drift:.2%}"
+
+
+def _projection_is_current(db: Any, profile_id: str) -> bool:
+    """Whether every change SQLite has recorded has reached the projection.
+
+    Two engines ranking two different graphs is worse than one engine ranking
+    the right one, because the difference is invisible in the output: both
+    return a full table of plausible scores. The queue is the only place that
+    records a change the projection has not seen yet, so an outstanding row
+    means the projection is a graph the store no longer has.
+
+    Absent queue means an older store that never had one; there is nothing to
+    be behind on, so the projection is as current as it can be.
+    """
+    try:
+        from superlocalmemory.storage import projection_outbox
+
+        if not projection_outbox.is_available(db):
+            return True
+        with _short_connection(db) as conn:
+            row = conn.execute(
+                "SELECT COUNT(*) FROM projection_outbox WHERE profile_id = ?",
+                (profile_id,),
+            ).fetchone()
+        return int(row[0] if row else 0) == 0
+    except Exception as exc:  # noqa: BLE001 -- unreadable means do not trust it
+        logger.debug("graph metrics: cannot check projection currency: %s", exc)
+        return False
+
+
+def compute_graph_metrics(
+    db: Any,
+    profile_id: str,
+    *,
+    backend: Any = None,
+    damping: float = DEFAULT_DAMPING,
+    prefer: str = "networkx",
+) -> GraphMetricsReport:
+    """Recompute ``fact_importance`` for one profile. Returns what it did.
+
+    ``backend`` is a live graph projection to compute on and ``prefer`` selects
+    the engine; the projection is used only when both point at it AND it agrees
+    with the store. Errors are returned in the report rather than swallowed: a
+    pass that writes nothing and a store with no facts are different events, and
+    the previous implementation reported both as ``node_count: 0``.
+    """
+    started = time.monotonic()
+    notes: list[str] = []
+    try:
+        with _short_connection(db) as conn:
+            nodes = _visible_fact_ids(conn, profile_id)
+            edges = [
+                (str(source), str(target), float(weight))
+                for source, target, _etype, weight, _pid
+                in iter_logical_edges(conn, profile_id)
+            ]
+    except Exception as exc:  # noqa: BLE001
+        return GraphMetricsReport(
+            profile_id=profile_id,
+            error=f"could not read the graph: {exc}",
+            duration_ms=int((time.monotonic() - started) * 1000),
+        )
+
+    if not nodes:
+        return GraphMetricsReport(
+            profile_id=profile_id,
+            engine="none",
+            duration_ms=int((time.monotonic() - started) * 1000),
+            notes=("no visible facts",),
+        )
+
+    usable, why = _projection_usable(backend, profile_id, len(edges))
+    notes.append(why)
+    if usable and not _projection_is_current(db, profile_id):
+        usable = False
+        why = "projection has unapplied changes"
+        notes.append(why)
+    engine = "cozo" if (usable and prefer == "cozo") else "networkx"
+    try:
+        # On ``engine``, not on ``usable``. Branching on ``usable`` ran the
+        # projection whenever one was available, whatever the caller asked for,
+        # and then reported the engine the caller had asked for -- so a store
+        # with a projection was ranked by it while every log line said
+        # otherwise, and the fallback below could never be reached.
+        if engine == "cozo":
+            pagerank, communities = _cozo_metrics(backend, profile_id, damping)
+        else:
+            pagerank, communities = _networkx_metrics(edges, damping)
+    except Exception as exc:  # noqa: BLE001
+        if engine == "cozo":
+            notes.append(f"cozo failed, fell back: {exc}")
+            engine = "networkx"
+            try:
+                pagerank, communities = _networkx_metrics(edges, damping)
+            except Exception as inner:  # noqa: BLE001
+                return GraphMetricsReport(
+                    profile_id=profile_id, engine="networkx",
+                    facts=len(nodes), edges=len(edges),
+                    error=f"both engines failed: {exc} / {inner}",
+                    duration_ms=int((time.monotonic() - started) * 1000),
+                    notes=tuple(notes),
+                )
+        else:
+            return GraphMetricsReport(
+                profile_id=profile_id, engine=engine,
+                facts=len(nodes), edges=len(edges),
+                error=f"metrics failed: {exc}",
+                duration_ms=int((time.monotonic() - started) * 1000),
+                notes=tuple(notes),
+            )
+
+    # An engine only sees nodes that appear in an edge. Everything else is a
+    # visible fact with no graph position, and it gets the teleport share -- the
+    # value PageRank would give a node nothing links to.
+    node_set = set(nodes)
+    total = len(node_set)
+    base = (1.0 - damping) / float(total)
+    connected = {fid for fid in pagerank if fid in node_set}
+    isolated = node_set - connected
+    # Leave room for the isolated mass so the whole table still sums to ~1 and
+    # the ranker's absolute thresholds keep meaning what they meant.
+    headroom = max(0.0, 1.0 - base * len(isolated))
+    connected_mass = sum(pagerank[fid] for fid in connected) or 1.0
+
+    degree: dict[str, int] = {}
+    for source, target, _weight in edges:
+        degree[source] = degree.get(source, 0) + 1
+        degree[target] = degree.get(target, 0) + 1
+    divisor = float(total - 1) if total > 1 else 1.0
+
+    bridges = _bridge_scores(edges, total)
+    if bridges is None:
+        notes.append(f"bridge scores skipped above {BRIDGE_NODE_LIMIT} facts")
+
+    rows: list[tuple[Any, ...]] = []
+    for fact_id in nodes:
+        if fact_id in connected:
+            score = pagerank[fact_id] / connected_mass * headroom
+        else:
+            score = base
+        community = communities.get(fact_id)
+        rows.append((
+            fact_id,
+            profile_id,
+            round(float(score), 9),
+            int(community) if community is not None else None,
+            round(degree.get(fact_id, 0) / divisor, 6),
+            round(float((bridges or {}).get(fact_id, 0.0)), 6),
+        ))
+
+    try:
+        removed = _write(db, profile_id, rows)
+    except Exception as exc:  # noqa: BLE001
+        return GraphMetricsReport(
+            profile_id=profile_id, engine=engine, facts=total,
+            edges=len(edges), connected=len(connected),
+            isolated=len(isolated),
+            error=f"metrics computed but not stored: {exc}",
+            duration_ms=int((time.monotonic() - started) * 1000),
+            notes=tuple(notes),
+        )
+
+    return GraphMetricsReport(
+        profile_id=profile_id,
+        engine=engine,
+        facts=total,
+        edges=len(edges),
+        connected=len(connected),
+        isolated=len(isolated),
+        communities=len({c for c in communities.values()}),
+        written=len(rows),
+        removed=removed,
+        bridges_computed=bridges is not None,
+        duration_ms=int((time.monotonic() - started) * 1000),
+        notes=tuple(notes),
+    )
+
+
+def _write(db: Any, profile_id: str, rows: list[tuple[Any, ...]]) -> int:
+    """Replace this profile's rows in one transaction.
+
+    Deleting first is what makes the table a projection rather than an
+    accumulation: a fact that has since been withheld or erased must lose its
+    row, or the ranker keeps scoring something recall will never return. The
+    delete and the insert share a transaction so no recall ever sees the
+    intermediate state where the profile has no metrics at all.
+    """
+    removed = 0
+    with _short_connection(db) as conn:
+        _ensure_bridge_column(conn)
+        keep = {row[0] for row in rows}
+        existing = {
+            str(r[0]) for r in conn.execute(
+                "SELECT fact_id FROM fact_importance WHERE profile_id = ?",
+                (profile_id,),
+            ).fetchall()
+        }
+        stale = existing - keep
+        for index in range(0, len(list(stale)), 800):
+            chunk = list(stale)[index:index + 800]
+            placeholders = ",".join("?" for _ in chunk)
+            conn.execute(
+                f"DELETE FROM fact_importance WHERE profile_id = ? "
+                f"AND fact_id IN ({placeholders})",
+                (profile_id, *chunk),
+            )
+            removed += len(chunk)
+        conn.executemany(
+            "INSERT INTO fact_importance "
+            "(fact_id, profile_id, pagerank_score, community_id, "
+            " degree_centrality, bridge_score, computed_at) "
+            "VALUES (?, ?, ?, ?, ?, ?, datetime('now')) "
+            "ON CONFLICT(fact_id) DO UPDATE SET "
+            "  profile_id = excluded.profile_id, "
+            "  pagerank_score = excluded.pagerank_score, "
+            "  community_id = excluded.community_id, "
+            "  degree_centrality = excluded.degree_centrality, "
+            "  bridge_score = excluded.bridge_score, "
+            "  computed_at = excluded.computed_at",
+            rows,
+        )
+    return removed
+
+
+def _ensure_bridge_column(conn: sqlite3.Connection) -> None:
+    """Idempotent: ``bridge_score`` arrived after the table did.
+
+    A store created before it exists in the wild, so the insert below cannot
+    assume the column. Adding it here rather than refusing keeps an upgrade from
+    silently losing its metrics on first run.
+    """
+    try:
+        columns = {row[1] for row in conn.execute("PRAGMA table_info(fact_importance)")}
+        if "bridge_score" not in columns:
+            conn.execute(
+                "ALTER TABLE fact_importance ADD COLUMN bridge_score REAL DEFAULT 0.0"
+            )
+    except sqlite3.Error as exc:
+        logger.debug("bridge_score column check failed: %s", exc)
+
+
+def metrics_are_stale(db: Any, profile_id: str) -> tuple[bool, str]:
+    """Whether this profile's metrics no longer describe its graph.
+
+    Deliberately not a clock. "Recomputed 30 minutes ago" says nothing about
+    whether the store changed, and the failure this guards against is a memory
+    that has no row at all -- which the ranker cannot distinguish from a memory
+    with no graph position. So the test is coverage: is any visible fact
+    missing, or does the table describe facts that are gone.
+
+    Cheap enough to run every cycle: two counting queries against an indexed
+    column.
+    """
+    try:
+        with _short_connection(db) as conn:
+            from superlocalmemory.storage.database import (
+                visible_fact_clause_for_connection,
+            )
+
+            clause = visible_fact_clause_for_connection(conn, prefix="f")
+            missing = conn.execute(
+                "SELECT COUNT(*) FROM atomic_facts f "
+                "LEFT JOIN fact_importance fi ON fi.fact_id = f.fact_id "
+                f"WHERE f.profile_id = ?{clause} AND fi.fact_id IS NULL",
+                (profile_id,),
+            ).fetchone()[0]
+            if missing:
+                return True, f"{missing} visible fact(s) have no metrics"
+            surplus = conn.execute(
+                "SELECT COUNT(*) FROM fact_importance fi "
+                "LEFT JOIN atomic_facts f ON f.fact_id = fi.fact_id "
+                f"WHERE fi.profile_id = ? AND (f.fact_id IS NULL OR NOT (1=1{clause}))",
+                (profile_id,),
+            ).fetchone()[0]
+            if surplus:
+                return True, f"{surplus} metric row(s) describe facts recall cannot return"
+        return False, "metrics cover the visible graph"
+    except Exception as exc:  # noqa: BLE001 -- a failed check must not skip the pass
+        return True, f"staleness check failed ({exc}); recomputing"