memgres 0.6.0__tar.gz → 0.7.0__tar.gz

{memgres-0.6.0 → memgres-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: memgres
-Version: 0.6.0
+Version: 0.7.0
 Summary: Drop-in memory for AI agents: one Postgres, lexical + semantic recall, diff-versioned history, GDPR-erasable.
 Author: mozgsml
 License-Expression: MIT
@@ -99,7 +99,7 @@ optional layers on top of the same core:
   └─ MCP server           write/recall/get/blame/move/forget as MCP tools
 ```
 
-**Record model:** one memory = one mutable body (up to a configurable ceiling, default 256 KB) plus metadata — `tags` (cross-cutting labels, `text[]` + GIN), a `path` (its place in an `ltree` tree), timestamps, and per-diff provenance (`source`/`reason`, kept in history). A single write/diff is capped smaller (default 16 KB), so large bodies accrue over many authored diffs. **Organization is two orthogonal axes:** the tree is *where a memory lives* (one place, subtree-selectable); tags are *what it's about* (many, overlapping). Both filter either search — narrow a semantic query to a subtree, or list a tag across the tree.
+**Record model:** one memory = one mutable body (up to a configurable ceiling, default 256 KB) plus metadata — `tags` (cross-cutting labels, `text[]` + GIN), a `path` (its place in an `ltree` tree), timestamps, and per-diff provenance (`source`/`reason`, kept in history). A single write/diff is capped smaller (default 16 KB), so large bodies accrue over many authored diffs. **Organization is two orthogonal axes:** the tree is *where a memory lives* (one place, subtree-selectable); tags are *what it's about* (many, overlapping). Both filter either search — narrow a semantic query to a subtree, or list a tag across the tree. **Links are the third axis:** `[[path]]` in a body becomes a real edge — walkable in BOTH directions, so "what relies on this?" has an answer before you change a fact — and a move rewrites the bodies that named the old address, so a link keeps working and an address copied out of a body is still one. Each memory also records how often it surfaces in search and how often it is opened, which is how the part of a corpus nobody reads becomes visible.
 
 **Isolation:** optional multi-tenant identity keeps tenants from seeing each other's memories — users own *namespaces*, and rotatable, permission-scoped *tokens* authenticate as a user (turned on with `MEMGRES_KEY_MODE=open|managed`; see [docs/TENANCY.md](docs/TENANCY.md)). Encryption at rest is left to the deployment — Postgres/managed-PG/disk TDE stays transparent to queries, so search keeps working; memgres deliberately does **not** encrypt bodies application-side (that would make them unsearchable, which is why no comparable tool does it either). GDPR erasure is real: `forget()` hard-deletes the row, its vectors, and crypto-shreds the history chain. All limits are env-configurable, so the same code serves a single-user embed and a capped multi-tenant service.
 
@@ -116,7 +116,7 @@ docker compose up
 
 Defaults suit a single-user setup with no auth. To change limits, the embedding provider, tokens, … drop a `.env` beside it — every `MEMGRES_*` is optional (see [Configuration](#configuration) or [.env.example](.env.example)).
 
-**Give it to an LLM / agent — no code (MCP).** Point any URL-capable MCP client (Cursor, Cline, Claude Desktop, …) at the running server; the model gets `memory_write`, `memory_recall`, `memory_get`, `memory_list`, `memory_blame`, `memory_history`, `memory_move`, `memory_forget`, `memory_server_info` as tools:
+**Give it to an LLM / agent — no code (MCP).** Point any URL-capable MCP client (Cursor, Cline, Claude Desktop, …) at the running server; the model gets `memory_write`, `memory_recall`, `memory_get`, `memory_list`, `memory_tags`, `memory_links`, `memory_blame`, `memory_history`, `memory_move`, `memory_forget`, `memory_server_info` as tools:
 
 ```json
 {
@@ -191,7 +191,7 @@ Not sure which fits? Start with the decision guide: [docs/CHOOSING.md](docs/CHOO
 1. **`docker compose up`** — `pgvector` + service, nothing to configure. For a dedicated vector service instead, `docker compose --profile qdrant up` and set `MEMGRES_VECTOR_BACKEND=qdrant` (Qdrant ranks vectors; Postgres still holds bodies and does tag/subtree/TTL filtering).
 2. **Your own Postgres** — install the `[server]` extra (above), point `MEMGRES_DATABASE_URL` at it, run `memgres-server` (migrates on startup).
 3. **Embedded library** — install the core package, use `Store` directly, no HTTP at all.
-4. **Split service (many clients)** — a stateless API tier that only flags writes plus a scalable `memgres-worker` tier that embeds; see [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) and `deploy/docker-compose.yml`. Switch the embedding model later with `memgres-reembed`.
+4. **Split service (many clients)** — a stateless API tier that only flags writes plus a scalable `memgres-worker` tier that embeds; see [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) and `deploy/docker-compose.yml`. Switch the embedding model later with `memgres-reembed`; rebuild the link graph with `memgres-relink`.
 
 Semantic recall is optional: the default `MEMGRES_EMBED_PROVIDER=none` gives you lexical FTS with zero models. Turn on `local` (sentence-transformers), a cloud API (`openai`/`jina`), or any OpenAI-compatible server (LM Studio, Ollama, …) when you want meaning-based search — see [docs/EMBEDDINGS.md](docs/EMBEDDINGS.md) for choosing local vs cloud and [docs/BACKENDS.md](docs/BACKENDS.md) for copy-paste setups. The model id + dimension get stamped into the schema and a later mismatch hard-fails instead of silently returning garbage.
 
@@ -206,12 +206,16 @@ Everything is env, all optional (defaults suit a single-user embed). Full list i
 | `MEMGRES_MAX_BODY_BYTES` | `262144` | ceiling for a whole record body (256 KB) |
 | `MEMGRES_MAX_WRITE_BYTES` | `16384` | ceiling for one write/diff payload (≤ body) |
 | `MEMGRES_MAX_SOURCE_BYTES` / `_MAX_REASON_BYTES` | `2048` / `1024` | ceilings for a write's `source` / `reason` provenance |
-| `MEMGRES_RETENTION_DAYS` | `0` | `0` = keep forever (TTL off); `>0` = expire N days after last touch |
+| `MEMGRES_RETENTION_DAYS` | `0` | how long data is kept: `0` = forever (off); `>0` = expire N days after **last touch**. Operator-only — there is no per-write TTL. Note the clock restarts on a touch, and a read counts as one unless `MEMGRES_RENEW_ON_READ=false`: the window covers data nobody uses |
 | `MEMGRES_RENEW_ON_READ` | `true` | a read pushes the expiry clock forward |
+| `MEMGRES_RETENTION_SWEEP` | `true` | this process runs the retention sweep. Every server process starts one; set `false` where a dedicated sweeper already runs |
+| `MEMGRES_RETENTION_SWEEP_INTERVAL` | `3600` | seconds between sweeps that DELETE expired rows (and their vectors). Only runs when `RETENTION_DAYS > 0` |
+| `MEMGRES_USAGE_COUNTERS` | `true` | count how often each memory surfaces in search and is read in full (`memory_usage`, a separate table — never the `memory` row, never the hash chain). Off makes reads pure again, for a read-only replica |
 | `MEMGRES_KEY_MODE` | `single` | `single` (no auth, one space) · `open` (bring-your-own token, self-registers) · `managed` (admin-provisioned). See [docs/TENANCY.md](docs/TENANCY.md) |
 | `MEMGRES_ADMIN_TOKEN` | — | global admin bearer for provisioning (managed mode) |
 | `MEMGRES_TOKEN` | — | default token used when a call passes none (single-tenant endpoints) |
 | `MEMGRES_TREE` | `true` | `ltree` path column + GiST index (fast subtree select) |
+| `MEMGRES_REQUIRE_TITLE` | `true` | `true` = a write that stores content must supply `title`. Captions are what name a memory in results and what title-weighted ranking weighs; `move`/`retag` are exempt (they store no content) |
 | `MEMGRES_REQUIRE_PARENT` | `false` | `true` = a node's parent path must already exist |
 | `MEMGRES_HISTORY` | `true` | keep the hash-chained diff history (deleted with the record) |
 | `MEMGRES_FTS_LANGUAGE` | `simple` | Postgres FTS dictionary (`simple`/`english`/…) |
@@ -327,8 +331,8 @@ pip install "memgres[mcp]"
 ```
 
 Either way the model gets tools `memory_write`, `memory_recall`, `memory_get`,
-`memory_list`, `memory_blame`, `memory_history`, `memory_move`, `memory_forget`,
-`memory_server_info`. Tell it *"remember X"* / *"what do you know about Y?"* and it
+`memory_list`, `memory_tags`, `memory_links`, `memory_blame`, `memory_history`,
+`memory_move`, `memory_forget`, `memory_server_info`. Tell it *"remember X"* / *"what do you know about Y?"* and it
 calls them. (For semantic recall add the embedding env vars — see
 [docs/BACKENDS.md](docs/BACKENDS.md).)
 

{memgres-0.6.0 → memgres-0.7.0}/README.md

@@ -63,7 +63,7 @@ optional layers on top of the same core:
   └─ MCP server           write/recall/get/blame/move/forget as MCP tools
 ```
 
-**Record model:** one memory = one mutable body (up to a configurable ceiling, default 256 KB) plus metadata — `tags` (cross-cutting labels, `text[]` + GIN), a `path` (its place in an `ltree` tree), timestamps, and per-diff provenance (`source`/`reason`, kept in history). A single write/diff is capped smaller (default 16 KB), so large bodies accrue over many authored diffs. **Organization is two orthogonal axes:** the tree is *where a memory lives* (one place, subtree-selectable); tags are *what it's about* (many, overlapping). Both filter either search — narrow a semantic query to a subtree, or list a tag across the tree.
+**Record model:** one memory = one mutable body (up to a configurable ceiling, default 256 KB) plus metadata — `tags` (cross-cutting labels, `text[]` + GIN), a `path` (its place in an `ltree` tree), timestamps, and per-diff provenance (`source`/`reason`, kept in history). A single write/diff is capped smaller (default 16 KB), so large bodies accrue over many authored diffs. **Organization is two orthogonal axes:** the tree is *where a memory lives* (one place, subtree-selectable); tags are *what it's about* (many, overlapping). Both filter either search — narrow a semantic query to a subtree, or list a tag across the tree. **Links are the third axis:** `[[path]]` in a body becomes a real edge — walkable in BOTH directions, so "what relies on this?" has an answer before you change a fact — and a move rewrites the bodies that named the old address, so a link keeps working and an address copied out of a body is still one. Each memory also records how often it surfaces in search and how often it is opened, which is how the part of a corpus nobody reads becomes visible.
 
 **Isolation:** optional multi-tenant identity keeps tenants from seeing each other's memories — users own *namespaces*, and rotatable, permission-scoped *tokens* authenticate as a user (turned on with `MEMGRES_KEY_MODE=open|managed`; see [docs/TENANCY.md](docs/TENANCY.md)). Encryption at rest is left to the deployment — Postgres/managed-PG/disk TDE stays transparent to queries, so search keeps working; memgres deliberately does **not** encrypt bodies application-side (that would make them unsearchable, which is why no comparable tool does it either). GDPR erasure is real: `forget()` hard-deletes the row, its vectors, and crypto-shreds the history chain. All limits are env-configurable, so the same code serves a single-user embed and a capped multi-tenant service.
 
@@ -80,7 +80,7 @@ docker compose up
 
 Defaults suit a single-user setup with no auth. To change limits, the embedding provider, tokens, … drop a `.env` beside it — every `MEMGRES_*` is optional (see [Configuration](#configuration) or [.env.example](.env.example)).
 
-**Give it to an LLM / agent — no code (MCP).** Point any URL-capable MCP client (Cursor, Cline, Claude Desktop, …) at the running server; the model gets `memory_write`, `memory_recall`, `memory_get`, `memory_list`, `memory_blame`, `memory_history`, `memory_move`, `memory_forget`, `memory_server_info` as tools:
+**Give it to an LLM / agent — no code (MCP).** Point any URL-capable MCP client (Cursor, Cline, Claude Desktop, …) at the running server; the model gets `memory_write`, `memory_recall`, `memory_get`, `memory_list`, `memory_tags`, `memory_links`, `memory_blame`, `memory_history`, `memory_move`, `memory_forget`, `memory_server_info` as tools:
 
 ```json
 {
@@ -155,7 +155,7 @@ Not sure which fits? Start with the decision guide: [docs/CHOOSING.md](docs/CHOO
 1. **`docker compose up`** — `pgvector` + service, nothing to configure. For a dedicated vector service instead, `docker compose --profile qdrant up` and set `MEMGRES_VECTOR_BACKEND=qdrant` (Qdrant ranks vectors; Postgres still holds bodies and does tag/subtree/TTL filtering).
 2. **Your own Postgres** — install the `[server]` extra (above), point `MEMGRES_DATABASE_URL` at it, run `memgres-server` (migrates on startup).
 3. **Embedded library** — install the core package, use `Store` directly, no HTTP at all.
-4. **Split service (many clients)** — a stateless API tier that only flags writes plus a scalable `memgres-worker` tier that embeds; see [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) and `deploy/docker-compose.yml`. Switch the embedding model later with `memgres-reembed`.
+4. **Split service (many clients)** — a stateless API tier that only flags writes plus a scalable `memgres-worker` tier that embeds; see [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) and `deploy/docker-compose.yml`. Switch the embedding model later with `memgres-reembed`; rebuild the link graph with `memgres-relink`.
 
 Semantic recall is optional: the default `MEMGRES_EMBED_PROVIDER=none` gives you lexical FTS with zero models. Turn on `local` (sentence-transformers), a cloud API (`openai`/`jina`), or any OpenAI-compatible server (LM Studio, Ollama, …) when you want meaning-based search — see [docs/EMBEDDINGS.md](docs/EMBEDDINGS.md) for choosing local vs cloud and [docs/BACKENDS.md](docs/BACKENDS.md) for copy-paste setups. The model id + dimension get stamped into the schema and a later mismatch hard-fails instead of silently returning garbage.
 
@@ -170,12 +170,16 @@ Everything is env, all optional (defaults suit a single-user embed). Full list i
 | `MEMGRES_MAX_BODY_BYTES` | `262144` | ceiling for a whole record body (256 KB) |
 | `MEMGRES_MAX_WRITE_BYTES` | `16384` | ceiling for one write/diff payload (≤ body) |
 | `MEMGRES_MAX_SOURCE_BYTES` / `_MAX_REASON_BYTES` | `2048` / `1024` | ceilings for a write's `source` / `reason` provenance |
-| `MEMGRES_RETENTION_DAYS` | `0` | `0` = keep forever (TTL off); `>0` = expire N days after last touch |
+| `MEMGRES_RETENTION_DAYS` | `0` | how long data is kept: `0` = forever (off); `>0` = expire N days after **last touch**. Operator-only — there is no per-write TTL. Note the clock restarts on a touch, and a read counts as one unless `MEMGRES_RENEW_ON_READ=false`: the window covers data nobody uses |
 | `MEMGRES_RENEW_ON_READ` | `true` | a read pushes the expiry clock forward |
+| `MEMGRES_RETENTION_SWEEP` | `true` | this process runs the retention sweep. Every server process starts one; set `false` where a dedicated sweeper already runs |
+| `MEMGRES_RETENTION_SWEEP_INTERVAL` | `3600` | seconds between sweeps that DELETE expired rows (and their vectors). Only runs when `RETENTION_DAYS > 0` |
+| `MEMGRES_USAGE_COUNTERS` | `true` | count how often each memory surfaces in search and is read in full (`memory_usage`, a separate table — never the `memory` row, never the hash chain). Off makes reads pure again, for a read-only replica |
 | `MEMGRES_KEY_MODE` | `single` | `single` (no auth, one space) · `open` (bring-your-own token, self-registers) · `managed` (admin-provisioned). See [docs/TENANCY.md](docs/TENANCY.md) |
 | `MEMGRES_ADMIN_TOKEN` | — | global admin bearer for provisioning (managed mode) |
 | `MEMGRES_TOKEN` | — | default token used when a call passes none (single-tenant endpoints) |
 | `MEMGRES_TREE` | `true` | `ltree` path column + GiST index (fast subtree select) |
+| `MEMGRES_REQUIRE_TITLE` | `true` | `true` = a write that stores content must supply `title`. Captions are what name a memory in results and what title-weighted ranking weighs; `move`/`retag` are exempt (they store no content) |
 | `MEMGRES_REQUIRE_PARENT` | `false` | `true` = a node's parent path must already exist |
 | `MEMGRES_HISTORY` | `true` | keep the hash-chained diff history (deleted with the record) |
 | `MEMGRES_FTS_LANGUAGE` | `simple` | Postgres FTS dictionary (`simple`/`english`/…) |
@@ -291,8 +295,8 @@ pip install "memgres[mcp]"
 ```
 
 Either way the model gets tools `memory_write`, `memory_recall`, `memory_get`,
-`memory_list`, `memory_blame`, `memory_history`, `memory_move`, `memory_forget`,
-`memory_server_info`. Tell it *"remember X"* / *"what do you know about Y?"* and it
+`memory_list`, `memory_tags`, `memory_links`, `memory_blame`, `memory_history`,
+`memory_move`, `memory_forget`, `memory_server_info`. Tell it *"remember X"* / *"what do you know about Y?"* and it
 calls them. (For semantic recall add the embedding env vars — see
 [docs/BACKENDS.md](docs/BACKENDS.md).)
 

{memgres-0.6.0 → memgres-0.7.0}/memgres/__init__.py

@@ -22,7 +22,8 @@ from .embeddings import Embedder, get_embedder
 from .schema import migrate, SchemaMismatch, SCHEMA_VERSION
 from .search import Hit, recall
 from .blame import annotate, annotate_grouped, reconstruct, replay
-from .store import Store, Memory, Conflict, NotFound, TooLarge, NoParent
+from .store import (Store, Memory, Conflict, NotFound, TooLarge, NoParent,
+                    MissingTitle)
 from .identity import (
     Principal, AuthError, SpaceNotFound,
     resolve, resolve_space, new_token, valid_format,
@@ -36,6 +37,7 @@ __all__ = [
     "__version__",
     "Config", "load_config",
     "Store", "Memory", "Conflict", "NotFound", "TooLarge", "NoParent",
+    "MissingTitle",
     "make_diff", "apply_diff", "content_hash", "DiffConflict",
     "Embedder", "get_embedder",
     "migrate", "SchemaMismatch", "SCHEMA_VERSION",

{memgres-0.6.0 → memgres-0.7.0}/memgres/_version.py

@@ -8,4 +8,4 @@ here at release; nowhere else carries the number.
 PEP 440: a ``.devN`` suffix marks an unreleased build ahead of the last tag.
 """
 
-__version__ = "0.6.0"
+__version__ = "0.7.0"

{memgres-0.6.0 → memgres-0.7.0}/memgres/config.py

@@ -72,6 +72,7 @@ class Config:
                                  # user_manager (default) | superadmin
     # organization
     tree_enabled: bool           # ltree path column + GiST index for fast subtree selection
+    require_title: bool          # True = a write that stores CONTENT must caption it
     require_parent: bool         # False = sparse paths (create food.apple with no food row);
                                  # True = a node's parent path must already exist as a memory
     # history
@@ -106,6 +107,12 @@ class Config:
                                  #     deployment where an external worker embeds.
     embed_worker: bool           # a server process runs an in-process embed worker
     embed_worker_interval: float # seconds the idle worker sleeps between drains
+    usage_counters: bool         # count how often each memory surfaces in search and
+                                 # is read in full. Off makes reads pure again — for
+                                 # a read-only replica, or a deployment unwilling to
+                                 # pay one small write per read.
+    retention_sweep: bool        # this process runs the retention sweep
+    retention_sweep_interval: float  # seconds between retention sweeps (see retention_days)
     embed_max_attempts: int      # after this many failed embed attempts a row is a
                                  # dead letter — left flagged but out of the claim
                                  # rotation (logged), so one poison body can't wedge
@@ -151,6 +158,8 @@ class Config:
             raise ValueError("MEMGRES_FULL_BODY_MAX_CHARS must be >= 0")
         if self.embed_worker_interval <= 0:
             raise ValueError("MEMGRES_EMBED_WORKER_INTERVAL must be > 0")
+        if self.retention_sweep_interval <= 0:
+            raise ValueError("MEMGRES_RETENTION_SWEEP_INTERVAL must be > 0")
         if self.embed_max_attempts < 1:
             raise ValueError("MEMGRES_EMBED_MAX_ATTEMPTS must be >= 1")
         if self.embed_retry_backoff_s < 0:
@@ -200,6 +209,7 @@ def load() -> Config:
         admin_token_file=_str("MEMGRES_ADMIN_TOKEN_FILE", ""),
         admin_role=_str("MEMGRES_ADMIN_ROLE", "user_manager"),
         tree_enabled=_bool("MEMGRES_TREE", True),
+        require_title=_bool("MEMGRES_REQUIRE_TITLE", True),
         require_parent=_bool("MEMGRES_REQUIRE_PARENT", False),
         history_enabled=_bool("MEMGRES_HISTORY", True),
         fts_language=_str("MEMGRES_FTS_LANGUAGE", "simple"),
@@ -216,6 +226,9 @@ def load() -> Config:
         embed_dispatch=_str("MEMGRES_EMBED_DISPATCH", "inline"),
         embed_worker=_bool("MEMGRES_EMBED_WORKER", True),
         embed_worker_interval=_float("MEMGRES_EMBED_WORKER_INTERVAL", 1.0),
+        usage_counters=_bool("MEMGRES_USAGE_COUNTERS", True),
+        retention_sweep=_bool("MEMGRES_RETENTION_SWEEP", True),
+        retention_sweep_interval=_float("MEMGRES_RETENTION_SWEEP_INTERVAL", 3600.0),
         embed_max_attempts=_int("MEMGRES_EMBED_MAX_ATTEMPTS", 5),
         embed_retry_backoff_s=_float("MEMGRES_EMBED_RETRY_BACKOFF_S", 60.0),
         list_preview_chars=_int("MEMGRES_LIST_PREVIEW_CHARS", 120),

{memgres-0.6.0 → memgres-0.7.0}/memgres/embed_worker.py

@@ -9,84 +9,46 @@ One daemon thread, one dedicated connection. It backfills on start (so a restart
 catches up any rows left pending, including the one-time re-chunk after the
 schema upgrade), then polls. ``drain_once`` is the same code the loop runs and is
 directly callable from a test or a CLI. The real work lives in
-:func:`memgres.indexing.drain`; this is just its lifecycle.
+:func:`memgres.indexing.drain`; the lifecycle lives in
+:class:`memgres.periodic.PeriodicWorker`, shared with the retention sweep.
 """
 
 from __future__ import annotations
 
 import logging
-import threading
 from typing import Callable, Optional
 
 from .indexing import drain
+from .periodic import PeriodicWorker, maybe_start_sweeper
 
 _log = logging.getLogger("memgres.embed_worker")
 
 
-class EmbedWorker:
+class EmbedWorker(PeriodicWorker):
+    """Drains ``embed_pending`` — one tick is one drain pass.
+
+    The first tick IS the backfill (rows left pending across a restart or the
+    schema upgrade). It happens in the thread rather than in ``start()``, so
+    building a server never blocks on embedding a backlog."""
+
+    name = "memgres-embed"
+
     def __init__(self, cfg, embedder, backend,
                  connect: Callable[[], "object"]):
-        self.cfg = cfg
+        super().__init__(cfg, connect, cfg.embed_worker_interval)
         self.embedder = embedder
         self.backend = backend
-        self._connect = connect          # () -> a fresh psycopg connection
-        self._conn = None
-        self._stop = threading.Event()
-        self._thread: Optional[threading.Thread] = None
-
-    def _conn_ok(self):
-        if self._conn is None or getattr(self._conn, "closed", False):
-            self._conn = self._connect()
-        return self._conn
 
     def drain_once(self) -> int:
         """One synchronous drain pass over all currently-pending rows. Returns the
         count embedded. Used by the loop and directly by tests."""
         return drain(self._conn_ok(), self.cfg, self.embedder, self.backend)
 
-    def _run(self) -> None:
-        # The first iteration IS the backfill (catch up rows left pending across a
-        # restart / the schema upgrade). Done in the thread, not in start(), so
-        # building a server never blocks on embedding a backlog.
-        while not self._stop.is_set():
-            try:
-                self.drain_once()
-            except Exception:
-                _log.exception("embed worker drain failed; dropping connection, retrying")
-                self._reset_conn()
-            self._stop.wait(self.cfg.embed_worker_interval)
-
-    def _reset_conn(self) -> None:
-        try:
-            if self._conn is not None:
-                self._conn.close()
-        except Exception:
-            pass
-        self._conn = None
-
-    def start(self) -> "EmbedWorker":
-        if self._thread is not None:
-            return self
-        self._thread = threading.Thread(target=self._run, name="memgres-embed",
-                                        daemon=True)
-        self._thread.start()
-        return self
-
-    def serve(self) -> None:
-        """Run the drain loop in the CURRENT thread, blocking until ``stop()``.
-        Used by the standalone ``memgres-worker`` process (a signal handler calls
-        ``stop()``); the in-process server path uses ``start()`` instead."""
-        self._run()
-        self._reset_conn()
+    def _tick(self) -> None:
+        self.drain_once()
 
     def stop(self) -> None:
         self._stop.set()
-        if self._thread is not None:
-            self._thread.join(timeout=5)
-            self._thread = None
-        self._reset_conn()
-
-
 def maybe_start_worker(cfg, embedder, backend,
                        connect: Callable[[], "object"]) -> Optional[EmbedWorker]:
     """Start an :class:`EmbedWorker` when there's something to embed and the
@@ -99,12 +61,17 @@ def maybe_start_worker(cfg, embedder, backend,
 
 
 def wire_server(cfg, embedder):
-    """Server-side setup shared by the HTTP and MCP entrypoints: build the vector
-    backend ONCE, start the in-process embed worker if warranted, and return
+    """Server-side background setup shared by the HTTP and MCP entrypoints: build
+    the vector backend ONCE, start the in-process embed worker if warranted, start
+    the retention sweep if the deployment has a retention policy, and return
     ``(worker, cfg, backend)`` with ``cfg.embed_dispatch`` set to what actually
     holds. The caller injects ``backend`` into every per-request ``Store`` so a
     qdrant client isn't rebuilt each call.
 
+    The sweep is started here and not returned: like the embed worker at both
+    call sites, nothing holds it — it is a daemon thread that dies with the
+    process. Drive it directly (``RetentionSweeper.sweep_once``) in a test.
+
     Dispatch resolution:
       * an in-process worker started (``embed_worker`` on, embedder present) →
         writes defer to it → ``async``. The all-in-one server default: fast writes
@@ -120,9 +87,15 @@ def wire_server(cfg, embedder):
     from .vector.base import make_backend
 
     backend = make_backend(cfg, embedder)
-    worker = maybe_start_worker(
-        cfg, embedder, backend,
-        connect=lambda: psycopg.connect(cfg.database_url or ""))
+    connect = lambda: psycopg.connect(cfg.database_url or "")   # noqa: E731
+    worker = maybe_start_worker(cfg, embedder, backend, connect=connect)
+    # Retention answers to its own policy, not to whether embeddings are on: a
+    # lexical-only deployment must still stop holding expired data.
+    maybe_start_sweeper(cfg, connect, embedder, backend)
+    # One-time, synchronous: a server that starts serving `memory_links` before
+    # the graph exists would answer "nothing points here" for every memory.
+    from .relink import maybe_backfill
+    maybe_backfill(cfg, connect)
     dispatch = "async" if worker is not None else cfg.embed_dispatch
     if dispatch == "async" and worker is None and backend is not None:
         # async + no local worker: writes will flag embed_pending and this process

memgres-0.7.0/memgres/links.py

@@ -0,0 +1,188 @@
+"""`[[wiki links]]` in a body, parsed into edges.
+
+The convention arrived on its own: this repo's reference corpus already carried
+238 `[[…]]` links across 97 memories with no tool support at all, 91% of them
+resolving. What it did not carry was a way to ask "what points HERE" — 42 of
+those 97 memories had no inbound link and were reachable only by search — or any
+way to notice when a link stopped resolving.
+
+## The syntax
+
+    [[target]]                a link
+    [[target#anchor]]         …to a section of it
+    [[target|label]]          …shown as something else
+    [[target#anchor|label]]   both
+
+`#` before `|`, as in every wiki dialect since MediaWiki and in Obsidian today,
+so a link written from habit parses. The split is positional and not clever:
+first `|` ends the target part, first `#` inside that part starts the anchor. A
+`#` in the LABEL is therefore just text, which is the reading that surprises
+nobody.
+
+## What is a link, and what is merely square brackets
+
+Only two shapes are treated as links, and everything else is left alone:
+
+* a **path** — an ltree path in this memory's own namespace (`ops.x402.deploy`);
+* a **scheme** we know (`idea:some-slug`, `file:some-note`) — a pointer at
+  another store entirely, recorded as an edge but never resolved here.
+
+A URL is not a link for these purposes and neither is prose. The rule is stated
+POSITIVELY — recognise a path or a known scheme — rather than as a list of things
+to exclude, because the exclusions are unbounded and the inclusions are two.
+
+Two exclusions are still needed, and both come straight from the corpus:
+
+* **code spans and fenced blocks are not scanned.** Documentation that explains
+  the syntax writes ``[[path]]`` in backticks, and a validator that flags its own
+  documentation trains everyone to ignore it;
+* single-segment targets that name nothing here are still recorded as edges
+  rather than dropped. A link to something not yet written is a deliberate move
+  ("this deserves a memory") and the tooling must let it stand — dangling, and
+  visible as dangling.
+"""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+from typing import Dict, List, Optional
+
+# Stores other than this one that a memory may legitimately point at. Recorded as
+# edges so they are visible, never resolved — we do not own the address space.
+KNOWN_SCHEMES = ("idea", "file")
+
+# An ltree path: labels of [A-Za-z0-9_] joined by dots. Deliberately strict —
+# this is what tells a path apart from a slug belonging to another store, and
+# from prose that happens to sit in double brackets.
+_PATH = re.compile(r"^[A-Za-z0-9_]+(\.[A-Za-z0-9_]+)*$")
+
+_LINK = re.compile(r"\[\[([^\[\]\n]+)\]\]")
+_FENCE = re.compile(r"```.*?```|~~~.*?~~~", re.S)
+_CODE_SPAN = re.compile(r"`[^`\n]*`")
+
+
+@dataclass
+class Link:
+    """One parsed link. `target` is verbatim as written; `scheme` is None for an
+    ordinary path and the scheme name for a pointer at another store.
+
+    `start`/`end` are the link's half-open span in the ORIGINAL body — the whole
+    `[[…]]`, brackets included. They are what lets a rewriter replace one link
+    without touching the rest of the text, and in particular without touching a
+    `[[path]]` written inside backticks: the parser never saw it, so it has no
+    span, so a rewrite cannot reach it.
+    """
+    raw_target: str
+    label: Optional[str] = None
+    anchor: Optional[str] = None
+    scheme: Optional[str] = None
+    ord: int = 0
+    start: int = 0
+    end: int = 0
+
+
+def _blank_code(body: str) -> str:
+    """Replace code spans and fenced blocks with spaces of the same length.
+
+    Same length, not removal, so every offset in the returned text still lines up
+    with the original — the parser does not need that today, but an anchor
+    resolver or a renderer will, and a scanner that quietly shifts positions is a
+    bug waiting for the feature that depends on them.
+    """
+    def blank(m: re.Match) -> str:
+        return "".join(" " if c != "\n" else "\n" for c in m.group(0))
+    return _CODE_SPAN.sub(blank, _FENCE.sub(blank, body))
+
+
+def _classify(target: str) -> Optional[str]:
+    """None for a path in this namespace, the scheme name for a known foreign
+    store, and the string ``"ignore"`` for everything else."""
+    if ":" in target:
+        scheme = target.split(":", 1)[0].strip().lower()
+        return scheme if scheme in KNOWN_SCHEMES else "ignore"
+    return None if _PATH.match(target) else "ignore"
+
+
+def parse_links(body: Optional[str]) -> List[Link]:
+    """Every link in `body`, in the order it appears.
+
+    Matched against the ORIGINAL text, with the blanked copy used only to decide
+    whether a match sits inside code. Reading the fields out of the blanked copy
+    instead was harmless while nothing wrote them back — and destructive the
+    moment something did: a label containing an inline `code` span came back as
+    that many spaces, and a rewrite then committed the loss as a well-formed diff.
+    A match may also STRADDLE a blanked region (backticks that swallow a `]]` and
+    the `[[` after it), which in the blanked copy looks like one enormous link;
+    requiring the span to be untouched by blanking rejects that as well.
+    """
+    body = body or ""
+    blanked = _blank_code(body)
+    out: List[Link] = []
+    for m in _LINK.finditer(body):
+        # Is THIS `[[` inside code? Its own opening brackets are blanked if so.
+        # Deliberately not "is any part of the span blanked": a label may legally
+        # contain an inline `code` span, and rejecting the whole link for that
+        # would silently drop a real edge.
+        if blanked[m.start():m.start() + 2] != "[[":
+            continue
+        inner = m.group(1).strip()
+        target_part, _, label = inner.partition("|")
+        target, _, anchor = target_part.partition("#")
+        target, label, anchor = (target.strip(), label.strip(), anchor.strip())
+        if not target:
+            continue
+        kind = _classify(target)
+        if kind == "ignore":
+            continue
+        out.append(Link(raw_target=target, label=label or None,
+                        anchor=anchor or None,
+                        scheme=None if kind is None else kind,
+                        ord=len(out), start=m.start(), end=m.end()))
+    return out
+
+
+def render(target: str, anchor: Optional[str] = None,
+           label: Optional[str] = None) -> str:
+    """The canonical text for a link. Inverse of the parse, in its normal form:
+    whitespace the author put inside the brackets is not preserved, because there
+    is nothing to preserve it for and a single spelling is easier to read."""
+    inner = target
+    if anchor:
+        inner += "#" + anchor
+    if label:
+        inner += "|" + label
+    return f"[[{inner}]]"
+
+
+def rewrite_targets(body: str, new_targets: Dict[int, str]) -> str:
+    """Point selected links somewhere else, leaving the rest of the body alone.
+
+    `new_targets` maps a link's `ord` — its index among the links this parser
+    RECOGNISES, which is exactly what the edge table stores — to the target it
+    should now carry. Anchor and label ride along unchanged: a rename moves the
+    address, not what the author called it or which section they meant.
+
+    Addressed by ord rather than by text on purpose. A body that links to the same
+    path twice has two edges, and only one of them may be the one to change; and a
+    body that mentions the old path in prose or in a code span must not be touched
+    at all. Substituting text would get both of those wrong.
+
+    Assembled in one forward pass over the spans rather than by slicing the whole
+    body once per link: a memory at the default 256 KB ceiling can hold tens of
+    thousands of links, and rebuilding the string each time made a single rewrite
+    quadratic — seconds of CPU while holding row locks.
+    """
+    if not new_targets:
+        return body
+    links = sorted((l for l in parse_links(body) if l.ord in new_targets),
+                   key=lambda l: l.start)
+    if not links:
+        return body
+    parts, at = [], 0
+    for l in links:
+        parts.append(body[at:l.start])
+        parts.append(render(new_targets[l.ord], l.anchor, l.label))
+        at = l.end
+    parts.append(body[at:])
+    return "".join(parts)