memgres 0.3.2__tar.gz → 0.4.0__tar.gz

{memgres-0.3.2 → memgres-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: memgres
-Version: 0.3.2
+Version: 0.4.0
 Summary: Drop-in memory for AI agents: one Postgres, lexical + semantic recall, diff-versioned history, GDPR-erasable.
 Author: mozgsml
 License-Expression: MIT
@@ -41,7 +41,7 @@ Dynamic: license-file
 
 **Versioned document memory for AI agents — one Postgres, lexical *or* semantic recall, diff-based history, GDPR-erasable.**
 
-> Status: v0.2.0 — on [PyPI](https://pypi.org/project/memgres/) (`pip install memgres`) and [GHCR](https://ghcr.io/mozgsml/memgres). Core library, search (pgvector/Qdrant), multi-tenant identity (users / namespaces / scoped tokens), HTTP API and MCP server, all tested in CI against live Postgres + Qdrant.
+> Status: released on [PyPI](https://pypi.org/project/memgres/) (`pip install memgres`) and [GHCR](https://ghcr.io/mozgsml/memgres) — see the [releases](https://github.com/mozgsml/memgres/releases) page for the current version. Core library, search (pgvector/Qdrant), multi-tenant identity (users / namespaces / scoped tokens), HTTP API and MCP server, all tested in CI against live Postgres + Qdrant.
 
 memgres is a lightweight, drop-in memory layer — a Python library plus an optional HTTP/MCP service — backed by a single PostgreSQL database. You store **documents** (bodies of text an agent owns and edits), not facts an LLM guessed at. Every change is an authored diff with provenance, kept in a tamper-evident history that you can still delete when the law says you must.
 
@@ -71,7 +71,7 @@ Reach for memgres when you want **auditable, authored, versioned text memory**.
 | **Hash-chained, GDPR-deletable history** | Tamper-evident provenance you can *still* erase: `forget()` hard-deletes the row, its vectors, and crypto-shreds the chain — no ["ghost vectors" left reconstructible in the index](https://arxiv.org/pdf/2606.18497). |
 | **Lexical works with zero embeddings** | Deploy with no model, no API, no GPU — Postgres full-text search out of the box. Turn on semantic recall only when you want it. |
 | **Lexical *and* semantic (hybrid)** | Exact identifiers/codes go to lexical (where [dense retrieval alone stumbles](https://tianpan.co/blog/2026-04-12-hybrid-search-production-bm25-dense-embeddings)); meaning-based queries go to vectors; hybrid fuses both with RRF. |
-| **Snippets, not walls of text** | Recall returns the most relevant slice of each hit plus its line number — semantic hits pick their best segment (embedded once, then cached), lexical uses `ts_headline`. Pass `full_body=false` for just the snippet. |
+| **Snippets, not walls of text** | Recall returns one body view per hit — never both a slice and the whole thing. Long hits come back as the most relevant slice (`kind="snippet"`) with its `lines` range; semantic hits pick their best segment (embedded once, then cached), lexical uses a clean `ts_headline` (no markup). A body short enough that a slice would just repeat it comes back whole (`kind="full"`). Pass `full_body=true` to force whole bodies. |
 | **Embedding-model safety by construction** | The model id + dimension are stamped into the schema; a mismatch **hard-fails** instead of silently returning garbage. |
 | **Optional TTL, renewed on read** | Off by default — memory is kept forever. Turn on a retention window and active memory persists because it's used, while abandoned memory expires itself: storage self-cleans instead of growing. |
 | **Optional multi-tenant identity** | Users, namespaces and rotatable scoped tokens when you need isolation; nothing to configure for single-user. |
@@ -184,11 +184,14 @@ Or pull the container image (public, no login):
 docker pull ghcr.io/mozgsml/memgres:latest
 ```
 
-### Three ways to run it
+### Ways to run it
+
+Not sure which fits? Start with the decision guide: [docs/CHOOSING.md](docs/CHOOSING.md). In short — **more than one user → run the Docker server (shared), not a per-machine install.**
 
 1. **`docker compose up`** — `pgvector` + service, nothing to configure. For a dedicated vector service instead, `docker compose --profile qdrant up` and set `MEMGRES_VECTOR_BACKEND=qdrant` (Qdrant ranks vectors; Postgres still holds bodies and does tag/subtree/TTL filtering).
 2. **Your own Postgres** — install the `[server]` extra (above), point `MEMGRES_DATABASE_URL` at it, run `memgres-server` (migrates on startup).
 3. **Embedded library** — install the core package, use `Store` directly, no HTTP at all.
+4. **Split service (many clients)** — a stateless API tier that only flags writes plus a scalable `memgres-worker` tier that embeds; see [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) and `deploy/docker-compose.yml`. Switch the embedding model later with `memgres-reembed`.
 
 Semantic recall is optional: the default `MEMGRES_EMBED_PROVIDER=none` gives you lexical FTS with zero models. Turn on `local` (sentence-transformers), a cloud API (`openai`/`jina`), or any OpenAI-compatible server (LM Studio, Ollama, …) when you want meaning-based search — see [docs/EMBEDDINGS.md](docs/EMBEDDINGS.md) for choosing local vs cloud and [docs/BACKENDS.md](docs/BACKENDS.md) for copy-paste setups. The model id + dimension get stamped into the schema and a later mismatch hard-fails instead of silently returning garbage.
 
@@ -213,12 +216,20 @@ Everything is env, all optional (defaults suit a single-user embed). Full list i
 | `MEMGRES_HISTORY` | `true` | keep the hash-chained diff history (deleted with the record) |
 | `MEMGRES_FTS_LANGUAGE` | `simple` | Postgres FTS dictionary (`simple`/`english`/…) |
 | `MEMGRES_LEXICAL_MATCH` | `any` | lexical query words OR-ed (`any`) or AND-ed (`all`); per-call `match` overrides |
-| `MEMGRES_SNIPPET` | `true` | attach a best-match snippet + line to each hit (`MEMGRES_SNIPPET_*` tune size/semantic; `full_body` per call) |
+| `MEMGRES_SNIPPET` | `true` | extract a best-match slice per hit (`MEMGRES_SNIPPET_*` tune size/semantic); `false` returns whole bodies |
+| `MEMGRES_FULL_BODY` | `false` | force the whole body on every hit (off = auto: short whole, long sliced); `full_body` per call overrides |
+| `MEMGRES_FULL_BODY_MAX_CHARS` | `500` | a body this short is returned whole (`kind="full"`) instead of sliced |
 | `MEMGRES_LIST_PREVIEW_CHARS` | `120` | first-line preview length returned by `memory_list` |
+| `MEMGRES_INSTRUCTION` | — | server-side MCP instructions emitted at `initialize` (a client like Claude Code loads it once at connect); unset = omitted; capped at 2 KB |
 | `MEMGRES_VECTOR_BACKEND` | `pgvector` | `pgvector` (same DB) or `qdrant` (set `QDRANT_URL`, `QDRANT_API_KEY`, `MEMGRES_QDRANT_COLLECTION`) |
 | `MEMGRES_EMBED_PROVIDER` | `none` | `none` / `local` / `openai` / `jina` / `openai-compatible` (LM Studio, Ollama, vLLM, TEI…) |
 | `MEMGRES_EMBED_MODEL` / `_DIM` / `_API_KEY` / `_API_BASE` | — | model id · dimension (HTTP providers require it, `local` infers) · token · server URL |
 | `MEMGRES_EMBED_MAX_SEQ` | `0` | override the local model's max input length in tokens (`0` = the model's default) |
+| `MEMGRES_CHUNK_CHARS` / `_OVERLAP` | `400` / `80` | chunk size / overlap for the chunk index (legacy names `MEMGRES_SNIPPET_SEG_CHARS` / `_OVERLAP` still work) |
+| `MEMGRES_EMBED_DISPATCH` | `inline` | `inline` = embed within the write (safe default, no worker); `async` = flag `embed_pending` and let a worker embed. A server sets `async` when it starts an in-process worker; set `async` + `MEMGRES_EMBED_WORKER=off` for a split deployment with a separate `memgres-worker` |
+| `MEMGRES_EMBED_WORKER` | `true` | a server process runs an in-process embed worker |
+| `MEMGRES_EMBED_WORKER_INTERVAL` | `1.0` | worker idle poll seconds between drains |
+| `MEMGRES_EMBED_MAX_ATTEMPTS` / `_RETRY_BACKOFF_S` | `5` / `60` | a failing row is retried after the back-off; after this many attempts it's dead-lettered (left flagged, out of rotation, logged) so it can't wedge the queue |
 
 ## HTTP API
 

{memgres-0.3.2 → memgres-0.4.0}/README.md

@@ -5,7 +5,7 @@
 
 **Versioned document memory for AI agents — one Postgres, lexical *or* semantic recall, diff-based history, GDPR-erasable.**
 
-> Status: v0.2.0 — on [PyPI](https://pypi.org/project/memgres/) (`pip install memgres`) and [GHCR](https://ghcr.io/mozgsml/memgres). Core library, search (pgvector/Qdrant), multi-tenant identity (users / namespaces / scoped tokens), HTTP API and MCP server, all tested in CI against live Postgres + Qdrant.
+> Status: released on [PyPI](https://pypi.org/project/memgres/) (`pip install memgres`) and [GHCR](https://ghcr.io/mozgsml/memgres) — see the [releases](https://github.com/mozgsml/memgres/releases) page for the current version. Core library, search (pgvector/Qdrant), multi-tenant identity (users / namespaces / scoped tokens), HTTP API and MCP server, all tested in CI against live Postgres + Qdrant.
 
 memgres is a lightweight, drop-in memory layer — a Python library plus an optional HTTP/MCP service — backed by a single PostgreSQL database. You store **documents** (bodies of text an agent owns and edits), not facts an LLM guessed at. Every change is an authored diff with provenance, kept in a tamper-evident history that you can still delete when the law says you must.
 
@@ -35,7 +35,7 @@ Reach for memgres when you want **auditable, authored, versioned text memory**.
 | **Hash-chained, GDPR-deletable history** | Tamper-evident provenance you can *still* erase: `forget()` hard-deletes the row, its vectors, and crypto-shreds the chain — no ["ghost vectors" left reconstructible in the index](https://arxiv.org/pdf/2606.18497). |
 | **Lexical works with zero embeddings** | Deploy with no model, no API, no GPU — Postgres full-text search out of the box. Turn on semantic recall only when you want it. |
 | **Lexical *and* semantic (hybrid)** | Exact identifiers/codes go to lexical (where [dense retrieval alone stumbles](https://tianpan.co/blog/2026-04-12-hybrid-search-production-bm25-dense-embeddings)); meaning-based queries go to vectors; hybrid fuses both with RRF. |
-| **Snippets, not walls of text** | Recall returns the most relevant slice of each hit plus its line number — semantic hits pick their best segment (embedded once, then cached), lexical uses `ts_headline`. Pass `full_body=false` for just the snippet. |
+| **Snippets, not walls of text** | Recall returns one body view per hit — never both a slice and the whole thing. Long hits come back as the most relevant slice (`kind="snippet"`) with its `lines` range; semantic hits pick their best segment (embedded once, then cached), lexical uses a clean `ts_headline` (no markup). A body short enough that a slice would just repeat it comes back whole (`kind="full"`). Pass `full_body=true` to force whole bodies. |
 | **Embedding-model safety by construction** | The model id + dimension are stamped into the schema; a mismatch **hard-fails** instead of silently returning garbage. |
 | **Optional TTL, renewed on read** | Off by default — memory is kept forever. Turn on a retention window and active memory persists because it's used, while abandoned memory expires itself: storage self-cleans instead of growing. |
 | **Optional multi-tenant identity** | Users, namespaces and rotatable scoped tokens when you need isolation; nothing to configure for single-user. |
@@ -148,11 +148,14 @@ Or pull the container image (public, no login):
 docker pull ghcr.io/mozgsml/memgres:latest
 ```
 
-### Three ways to run it
+### Ways to run it
+
+Not sure which fits? Start with the decision guide: [docs/CHOOSING.md](docs/CHOOSING.md). In short — **more than one user → run the Docker server (shared), not a per-machine install.**
 
 1. **`docker compose up`** — `pgvector` + service, nothing to configure. For a dedicated vector service instead, `docker compose --profile qdrant up` and set `MEMGRES_VECTOR_BACKEND=qdrant` (Qdrant ranks vectors; Postgres still holds bodies and does tag/subtree/TTL filtering).
 2. **Your own Postgres** — install the `[server]` extra (above), point `MEMGRES_DATABASE_URL` at it, run `memgres-server` (migrates on startup).
 3. **Embedded library** — install the core package, use `Store` directly, no HTTP at all.
+4. **Split service (many clients)** — a stateless API tier that only flags writes plus a scalable `memgres-worker` tier that embeds; see [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) and `deploy/docker-compose.yml`. Switch the embedding model later with `memgres-reembed`.
 
 Semantic recall is optional: the default `MEMGRES_EMBED_PROVIDER=none` gives you lexical FTS with zero models. Turn on `local` (sentence-transformers), a cloud API (`openai`/`jina`), or any OpenAI-compatible server (LM Studio, Ollama, …) when you want meaning-based search — see [docs/EMBEDDINGS.md](docs/EMBEDDINGS.md) for choosing local vs cloud and [docs/BACKENDS.md](docs/BACKENDS.md) for copy-paste setups. The model id + dimension get stamped into the schema and a later mismatch hard-fails instead of silently returning garbage.
 
@@ -177,12 +180,20 @@ Everything is env, all optional (defaults suit a single-user embed). Full list i
 | `MEMGRES_HISTORY` | `true` | keep the hash-chained diff history (deleted with the record) |
 | `MEMGRES_FTS_LANGUAGE` | `simple` | Postgres FTS dictionary (`simple`/`english`/…) |
 | `MEMGRES_LEXICAL_MATCH` | `any` | lexical query words OR-ed (`any`) or AND-ed (`all`); per-call `match` overrides |
-| `MEMGRES_SNIPPET` | `true` | attach a best-match snippet + line to each hit (`MEMGRES_SNIPPET_*` tune size/semantic; `full_body` per call) |
+| `MEMGRES_SNIPPET` | `true` | extract a best-match slice per hit (`MEMGRES_SNIPPET_*` tune size/semantic); `false` returns whole bodies |
+| `MEMGRES_FULL_BODY` | `false` | force the whole body on every hit (off = auto: short whole, long sliced); `full_body` per call overrides |
+| `MEMGRES_FULL_BODY_MAX_CHARS` | `500` | a body this short is returned whole (`kind="full"`) instead of sliced |
 | `MEMGRES_LIST_PREVIEW_CHARS` | `120` | first-line preview length returned by `memory_list` |
+| `MEMGRES_INSTRUCTION` | — | server-side MCP instructions emitted at `initialize` (a client like Claude Code loads it once at connect); unset = omitted; capped at 2 KB |
 | `MEMGRES_VECTOR_BACKEND` | `pgvector` | `pgvector` (same DB) or `qdrant` (set `QDRANT_URL`, `QDRANT_API_KEY`, `MEMGRES_QDRANT_COLLECTION`) |
 | `MEMGRES_EMBED_PROVIDER` | `none` | `none` / `local` / `openai` / `jina` / `openai-compatible` (LM Studio, Ollama, vLLM, TEI…) |
 | `MEMGRES_EMBED_MODEL` / `_DIM` / `_API_KEY` / `_API_BASE` | — | model id · dimension (HTTP providers require it, `local` infers) · token · server URL |
 | `MEMGRES_EMBED_MAX_SEQ` | `0` | override the local model's max input length in tokens (`0` = the model's default) |
+| `MEMGRES_CHUNK_CHARS` / `_OVERLAP` | `400` / `80` | chunk size / overlap for the chunk index (legacy names `MEMGRES_SNIPPET_SEG_CHARS` / `_OVERLAP` still work) |
+| `MEMGRES_EMBED_DISPATCH` | `inline` | `inline` = embed within the write (safe default, no worker); `async` = flag `embed_pending` and let a worker embed. A server sets `async` when it starts an in-process worker; set `async` + `MEMGRES_EMBED_WORKER=off` for a split deployment with a separate `memgres-worker` |
+| `MEMGRES_EMBED_WORKER` | `true` | a server process runs an in-process embed worker |
+| `MEMGRES_EMBED_WORKER_INTERVAL` | `1.0` | worker idle poll seconds between drains |
+| `MEMGRES_EMBED_MAX_ATTEMPTS` / `_RETRY_BACKOFF_S` | `5` / `60` | a failing row is retried after the back-off; after this many attempts it's dead-lettered (left flagged, out of rotation, logged) so it can't wedge the queue |
 
 ## HTTP API
 

{memgres-0.3.2 → memgres-0.4.0}/memgres/__init__.py

@@ -15,6 +15,7 @@ Quick start::
     hits = store.recall(None, "what did I remember?")            # lexical or semantic
 """
 
+from ._version import __version__
 from .config import Config, load as load_config
 from .diffing import apply_diff, content_hash, make_diff, DiffConflict
 from .embeddings import Embedder, get_embedder
@@ -31,6 +32,7 @@ from .identity import (
 )
 
 __all__ = [
+    "__version__",
     "Config", "load_config",
     "Store", "Memory", "Conflict", "NotFound", "TooLarge", "NoParent",
     "make_diff", "apply_diff", "content_hash", "DiffConflict",

memgres-0.4.0/memgres/_version.py

@@ -0,0 +1,11 @@
+"""Single source of truth for the package version.
+
+Read at runtime by ``server_info`` (so an editable/dev checkout reports the code
+it is actually running, not stale install metadata) and at build time by
+``pyproject.toml`` (``[tool.setuptools.dynamic] version = {attr = ...}``). Bump
+here at release; nowhere else carries the number.
+
+PEP 440: a ``.devN`` suffix marks an unreleased build ahead of the last tag.
+"""
+
+__version__ = "0.4.0"

{memgres-0.3.2 → memgres-0.4.0}/memgres/blame.py

@@ -20,7 +20,8 @@ from typing import Iterable, List, Optional, Tuple
 
 from .diffing import _HUNK
 
-_ATTR_KEYS = ("seq", "op", "source", "reason", "created_at")
+_ATTR_KEYS = ("seq", "op", "source", "reason", "created_at",
+              "author_user_id", "author_token_id", "author_name")
 
 
 def _apply_attributed(src: List[Tuple[str, dict]], patch: str,

{memgres-0.3.2 → memgres-0.4.0}/memgres/config.py

@@ -35,6 +35,11 @@ def _bool(name: str, default: bool) -> bool:
     return raw.strip().lower() in ("1", "true", "yes", "on")
 
 
+def _float(name: str, default: float) -> float:
+    raw = os.environ.get(name)
+    return float(raw) if raw not in (None, "") else default
+
+
 def _str(name: str, default: str) -> str:
     raw = os.environ.get(name)
     return raw if raw not in (None, "") else default
@@ -47,11 +52,12 @@ class Config:
     max_write_bytes: int         # one write/diff payload ceiling (<= max_body)
     max_source_bytes: int        # provenance `source` field ceiling (per write)
     max_reason_bytes: int        # provenance `reason` field ceiling (per write)
+    max_title_bytes: int         # curated `title` field ceiling (per write)
     # retention
     retention_days: int          # 0 = forever; >0 = expire N days after last touch
     renew_on_read: bool          # a read pushes the expiry clock forward
     # multi-tenant isolation
-    token: str                   # default token used when a call passes none
+    default_token: str           # default token used when a call passes none
                                  # (set in MCP/env for a single-tenant deployment)
     # identity / tenancy (see docs/TENANCY.md)
     key_mode: str                # single | open | managed  (how tokens/users are minted)
@@ -66,14 +72,37 @@ class Config:
     fts_language: str            # Postgres FTS dict: simple | english | russian | …
     lexical_match: str           # any (OR-any words, default) | all (AND-all words)
     vector_backend: str          # pgvector (default) | qdrant
-    # snippets (a relevant slice of each recall hit's body + its line number)
-    snippet: bool                # attach a snippet+line to each hit
-    full_body: bool              # also return the whole body (off = snippet only)
+    # snippets (a relevant slice of each recall hit's body + its line range)
+    snippet: bool                # extract a relevant slice; off = return the body
+    full_body: bool              # force the whole body on every hit (off = auto:
+                                 # short bodies whole, long bodies sliced)
+    full_body_max_chars: int     # a body this short is returned whole (kind=full)
+                                 # instead of sliced — a slice would just repeat it
     snippet_semantic: bool       # semantic/hybrid hits use the best-matching
                                  # segment (needs the model); off = ts_headline,
                                  # avoiding per-query model calls on a paid API
-    snippet_seg_chars: int       # segment size for the best-segment snippet cache
-    snippet_seg_overlap: int     # chars shared between consecutive segments
+    chunk_chars: int             # chunk size for the chunk index (ranking+snippet);
+                                 # MEMGRES_CHUNK_CHARS (legacy MEMGRES_SNIPPET_SEG_CHARS)
+    chunk_overlap: int           # chars shared between consecutive chunks
+                                 # (MEMGRES_CHUNK_OVERLAP / legacy _SNIPPET_SEG_OVERLAP)
+    # embedding pipeline (chunks are the semantic index; see docs/EMBEDDINGS.md)
+    embed_dispatch: str          # how a write's chunk-embedding happens:
+                                 #   inline — embed within the write (safe default;
+                                 #     library/embedded use, no worker needed);
+                                 #   async  — flag embed_pending and return; a worker
+                                 #     (in-process or a separate memgres-worker)
+                                 #     drains it. A SERVER upgrades inline→async when
+                                 #     it starts an in-process worker (see
+                                 #     embed_worker.wire_server); set it to async
+                                 #     explicitly WITH embed_worker=off for a split
+                                 #     deployment where an external worker embeds.
+    embed_worker: bool           # a server process runs an in-process embed worker
+    embed_worker_interval: float # seconds the idle worker sleeps between drains
+    embed_max_attempts: int      # after this many failed embed attempts a row is a
+                                 # dead letter — left flagged but out of the claim
+                                 # rotation (logged), so one poison body can't wedge
+                                 # the queue behind it. A successful embed resets it.
+    embed_retry_backoff_s: float # seconds a failed row is skipped before retry
     # listing / browse
     list_preview_chars: int      # first-line preview length for memory_list (0 = none)
     # embeddings
@@ -95,14 +124,26 @@ class Config:
             raise ValueError("MEMGRES_MAX_SOURCE_BYTES must be >= 1")
         if self.max_reason_bytes < 1:
             raise ValueError("MEMGRES_MAX_REASON_BYTES must be >= 1")
+        if self.max_title_bytes < 1:
+            raise ValueError("MEMGRES_MAX_TITLE_BYTES must be >= 1")
         if self.embed_max_seq < 0:
             raise ValueError("MEMGRES_EMBED_MAX_SEQ must be >= 0")
         if self.list_preview_chars < 0:
             raise ValueError("MEMGRES_LIST_PREVIEW_CHARS must be >= 0")
-        if self.snippet_seg_chars < 1:
-            raise ValueError("MEMGRES_SNIPPET_SEG_CHARS must be >= 1")
-        if self.snippet_seg_overlap < 0:
-            raise ValueError("MEMGRES_SNIPPET_SEG_OVERLAP must be >= 0")
+        if self.chunk_chars < 1:
+            raise ValueError("MEMGRES_CHUNK_CHARS must be >= 1")
+        if self.chunk_overlap < 0:
+            raise ValueError("MEMGRES_CHUNK_OVERLAP must be >= 0")
+        if self.embed_dispatch not in ("inline", "async"):
+            raise ValueError(f"unknown MEMGRES_EMBED_DISPATCH: {self.embed_dispatch}")
+        if self.full_body_max_chars < 0:
+            raise ValueError("MEMGRES_FULL_BODY_MAX_CHARS must be >= 0")
+        if self.embed_worker_interval <= 0:
+            raise ValueError("MEMGRES_EMBED_WORKER_INTERVAL must be > 0")
+        if self.embed_max_attempts < 1:
+            raise ValueError("MEMGRES_EMBED_MAX_ATTEMPTS must be >= 1")
+        if self.embed_retry_backoff_s < 0:
+            raise ValueError("MEMGRES_EMBED_RETRY_BACKOFF_S must be >= 0")
         if self.max_write_bytes > self.max_body_bytes:
             raise ValueError(
                 "MEMGRES_MAX_WRITE_BYTES must be <= MEMGRES_MAX_BODY_BYTES"
@@ -132,9 +173,10 @@ def load() -> Config:
         max_write_bytes=_int("MEMGRES_MAX_WRITE_BYTES", 16_384),      # 16 KB
         max_source_bytes=_int("MEMGRES_MAX_SOURCE_BYTES", 2_048),     # 2 KB
         max_reason_bytes=_int("MEMGRES_MAX_REASON_BYTES", 1_024),     # 1 KB
+        max_title_bytes=_int("MEMGRES_MAX_TITLE_BYTES", 256),         # 256 B
         retention_days=_int("MEMGRES_RETENTION_DAYS", 0),
         renew_on_read=_bool("MEMGRES_RENEW_ON_READ", True),
-        token=_str("MEMGRES_TOKEN", ""),
+        default_token=_str("MEMGRES_TOKEN", ""),
         key_mode=_str("MEMGRES_KEY_MODE", "single"),
         admin_token=_str("MEMGRES_ADMIN_TOKEN", ""),
         tree_enabled=_bool("MEMGRES_TREE", True),
@@ -144,10 +186,18 @@ def load() -> Config:
         lexical_match=_str("MEMGRES_LEXICAL_MATCH", "any"),
         vector_backend=_str("MEMGRES_VECTOR_BACKEND", "pgvector"),
         snippet=_bool("MEMGRES_SNIPPET", True),
-        full_body=_bool("MEMGRES_FULL_BODY", True),
+        full_body=_bool("MEMGRES_FULL_BODY", False),
+        full_body_max_chars=_int("MEMGRES_FULL_BODY_MAX_CHARS", 500),
         snippet_semantic=_bool("MEMGRES_SNIPPET_SEMANTIC", True),
-        snippet_seg_chars=_int("MEMGRES_SNIPPET_SEG_CHARS", 400),
-        snippet_seg_overlap=_int("MEMGRES_SNIPPET_SEG_OVERLAP", 80),
+        chunk_chars=_int("MEMGRES_CHUNK_CHARS",
+                         _int("MEMGRES_SNIPPET_SEG_CHARS", 400)),
+        chunk_overlap=_int("MEMGRES_CHUNK_OVERLAP",
+                           _int("MEMGRES_SNIPPET_SEG_OVERLAP", 80)),
+        embed_dispatch=_str("MEMGRES_EMBED_DISPATCH", "inline"),
+        embed_worker=_bool("MEMGRES_EMBED_WORKER", True),
+        embed_worker_interval=_float("MEMGRES_EMBED_WORKER_INTERVAL", 1.0),
+        embed_max_attempts=_int("MEMGRES_EMBED_MAX_ATTEMPTS", 5),
+        embed_retry_backoff_s=_float("MEMGRES_EMBED_RETRY_BACKOFF_S", 60.0),
         list_preview_chars=_int("MEMGRES_LIST_PREVIEW_CHARS", 120),
         embed_provider=_str("MEMGRES_EMBED_PROVIDER", "none"),
         embed_model=_str("MEMGRES_EMBED_MODEL", ""),

memgres-0.4.0/memgres/embed_worker.py

@@ -0,0 +1,134 @@
+"""Background embed worker: builds chunk vectors off the write path.
+
+When a server runs this worker, writes only flag ``embed_pending`` and return
+immediately; the worker drains those rows — segment, embed, index — on its own
+connection and thread. That's what keeps a write fast even for a large body (the
+embedding no longer runs inside the request).
+
+One daemon thread, one dedicated connection. It backfills on start (so a restart
+catches up any rows left pending, including the one-time re-chunk after the
+schema upgrade), then polls. ``drain_once`` is the same code the loop runs and is
+directly callable from a test or a CLI. The real work lives in
+:func:`memgres.indexing.drain`; this is just its lifecycle.
+"""
+
+from __future__ import annotations
+
+import logging
+import threading
+from typing import Callable, Optional
+
+from .indexing import drain
+
+_log = logging.getLogger("memgres.embed_worker")
+
+
+class EmbedWorker:
+    def __init__(self, cfg, embedder, backend,
+                 connect: Callable[[], "object"]):
+        self.cfg = cfg
+        self.embedder = embedder
+        self.backend = backend
+        self._connect = connect          # () -> a fresh psycopg connection
+        self._conn = None
+        self._stop = threading.Event()
+        self._thread: Optional[threading.Thread] = None
+
+    def _conn_ok(self):
+        if self._conn is None or getattr(self._conn, "closed", False):
+            self._conn = self._connect()
+        return self._conn
+
+    def drain_once(self) -> int:
+        """One synchronous drain pass over all currently-pending rows. Returns the
+        count embedded. Used by the loop and directly by tests."""
+        return drain(self._conn_ok(), self.cfg, self.embedder, self.backend)
+
+    def _run(self) -> None:
+        # The first iteration IS the backfill (catch up rows left pending across a
+        # restart / the schema upgrade). Done in the thread, not in start(), so
+        # building a server never blocks on embedding a backlog.
+        while not self._stop.is_set():
+            try:
+                self.drain_once()
+            except Exception:
+                _log.exception("embed worker drain failed; dropping connection, retrying")
+                self._reset_conn()
+            self._stop.wait(self.cfg.embed_worker_interval)
+
+    def _reset_conn(self) -> None:
+        try:
+            if self._conn is not None:
+                self._conn.close()
+        except Exception:
+            pass
+        self._conn = None
+
+    def start(self) -> "EmbedWorker":
+        if self._thread is not None:
+            return self
+        self._thread = threading.Thread(target=self._run, name="memgres-embed",
+                                        daemon=True)
+        self._thread.start()
+        return self
+
+    def serve(self) -> None:
+        """Run the drain loop in the CURRENT thread, blocking until ``stop()``.
+        Used by the standalone ``memgres-worker`` process (a signal handler calls
+        ``stop()``); the in-process server path uses ``start()`` instead."""
+        self._run()
+        self._reset_conn()
+
+    def stop(self) -> None:
+        self._stop.set()
+        if self._thread is not None:
+            self._thread.join(timeout=5)
+            self._thread = None
+        self._reset_conn()
+
+
+def maybe_start_worker(cfg, embedder, backend,
+                       connect: Callable[[], "object"]) -> Optional[EmbedWorker]:
+    """Start an :class:`EmbedWorker` when there's something to embed and the
+    deployment wants one (``MEMGRES_EMBED_WORKER``, default on). Returns the
+    running worker, or ``None`` — in which case the caller must keep writes
+    synchronous (embed inline), so semantic recall never silently lags."""
+    if embedder is None or backend is None or not cfg.embed_worker:
+        return None
+    return EmbedWorker(cfg, embedder, backend, connect).start()
+
+
+def wire_server(cfg, embedder):
+    """Server-side setup shared by the HTTP and MCP entrypoints: build the vector
+    backend ONCE, start the in-process embed worker if warranted, and return
+    ``(worker, cfg, backend)`` with ``cfg.embed_dispatch`` set to what actually
+    holds. The caller injects ``backend`` into every per-request ``Store`` so a
+    qdrant client isn't rebuilt each call.
+
+    Dispatch resolution:
+      * an in-process worker started (``embed_worker`` on, embedder present) →
+        writes defer to it → ``async``. The all-in-one server default: fast writes
+        with a local drainer, never a flag nothing embeds.
+      * no in-process worker → keep the operator's ``embed_dispatch``. That is how
+        a SPLIT deployment works: an API container sets ``async`` + ``embed_worker
+        =off`` and a separate ``memgres-worker`` drains. Left at the default
+        (``inline``), a workerless server just embeds inline — safe, never a
+        silent gap."""
+    import psycopg
+    from dataclasses import replace
+
+    from .vector.base import make_backend
+
+    backend = make_backend(cfg, embedder)
+    worker = maybe_start_worker(
+        cfg, embedder, backend,
+        connect=lambda: psycopg.connect(cfg.database_url or ""))
+    dispatch = "async" if worker is not None else cfg.embed_dispatch
+    if dispatch == "async" and worker is None and backend is not None:
+        # async + no local worker: writes will flag embed_pending and this process
+        # embeds nothing. Correct ONLY in a split deployment with a separate
+        # memgres-worker. Warn loudly so a missing/failed worker isn't a silent gap.
+        _log.warning("MEMGRES_EMBED_DISPATCH=async with no in-process worker: writes "
+                     "will be flagged but NOT embedded here — a separate "
+                     "memgres-worker MUST be running, or semantic recall will lag.")
+    return worker, replace(cfg, embed_dispatch=dispatch), backend

{memgres-0.3.2 → memgres-0.4.0}/memgres/identity.py

@@ -32,6 +32,19 @@ from typing import List, Optional, Tuple
 # ─── token format: mgk_ + 43 url-safe chars (256-bit) ────────────────────────
 TOKEN_RE = re.compile(r"^mgk_[A-Za-z0-9_-]{43}$")
 
+
+def bearer_token(authorization: Optional[str],
+                 x_memgres_token: Optional[str]) -> Optional[str]:
+    """Extract a raw memgres token from request auth headers — ``Authorization:
+    Bearer <tok>`` first, else the ``X-Memgres-Token`` header — or ``None`` if
+    neither is present. One definition for both the HTTP and MCP transports so
+    header parsing can't drift between them."""
+    if authorization and authorization[:7].lower() == "bearer ":
+        return authorization[7:].strip() or None
+    if x_memgres_token:
+        return x_memgres_token.strip() or None
+    return None
+
 # permission lattice
 _RANK = {"read": 1, "write": 2, "admin": 3}
 

memgres-0.4.0/memgres/indexing.py

@@ -0,0 +1,144 @@
+"""Chunk embedding: turn a memory body into chunk vectors, sync or via worker.
+
+One place builds the chunk index for a memory (`index_memory`), reached two ways:
+
+  * **synchronously**, inline in the write path (embedded/library mode, or a
+    server with the worker turned off) — the default, so semantic recall never
+    silently lags behind a write;
+  * **asynchronously**, by the background embed worker draining `embed_pending`
+    rows (the server's default: writes return fast, embedding runs off the
+    request path).
+
+Idempotent and crash-safe: `index_memory` skips a row whose chunks already match
+the body's content_hash, and clears `embed_pending` only when the body it
+embedded is still current — a concurrent edit bumps the hash and re-flags the
+row, so it's picked up again rather than lost. `drain` commits per row, so a
+crash mid-batch leaves the unfinished rows flagged for the next pass.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import Optional
+
+from .diffing import content_hash
+from .segments import segment
+
+_log = logging.getLogger("memgres.indexing")
+
+
+def index_memory(conn, cfg, embedder, backend, memory_id: str, body: str,
+                 ns: str, src_hash: Optional[str] = None) -> bool:
+    """Build and store the chunk vectors for one memory, then clear its pending
+    flag. Returns True if it (re)embedded, False if the chunks were already
+    current (or there's nothing to embed). Does NOT manage the transaction — the
+    caller owns commit/rollback (the write path folds this into its own tx; the
+    worker commits per row)."""
+    src = src_hash or content_hash(body)
+    if backend is None or embedder is None:
+        _clear_pending(conn, memory_id, src)   # no vectors; don't leave it pending
+        return False
+    if backend.chunk_src_hash(conn, memory_id, ns) == src:
+        _clear_pending(conn, memory_id, src)   # already current for this body
+        return False
+    spans = segment(body, cfg.chunk_chars, cfg.chunk_overlap)
+    if spans:
+        vecs = embedder.embed_documents([body[s:e] for (s, e) in spans])
+        chunks = [(i, s, e, v) for i, ((s, e), v) in enumerate(zip(spans, vecs))]
+        backend.index_chunks(conn, memory_id, ns, src, chunks)
+    else:
+        backend.delete_chunks(conn, memory_id, ns)   # empty body → no chunks
+    _clear_pending(conn, memory_id, src)
+    return True
+
+
+def _clear_pending(conn, memory_id: str, src: str) -> None:
+    """Clear ``embed_pending`` (and reset the retry counters) only if the body is
+    still the one we embedded (guarded by content_hash). A concurrent edit changed
+    the hash and re-set the flag, so leaving it set hands the row to the next
+    pass. A success zeroes ``embed_attempts``/``embed_failed_at`` so a later
+    legitimate edit re-embeds cleanly rather than inheriting a stale failure."""
+    with conn.cursor() as cur:
+        cur.execute(
+            "UPDATE memory SET embed_pending=false, embed_attempts=0, "
+            "embed_failed_at=NULL "
+            "WHERE id=%s AND content_hash=%s AND embed_pending",
+            (memory_id, src))
+
+
+def _record_failure(conn, cfg, memory_id: str) -> None:
+    """Record a failed embed attempt in its OWN transaction (the embed tx rolled
+    back). The claim then skips this row for a back-off window, and once it has
+    failed ``embed_max_attempts`` times the claim drops it entirely — a dead
+    letter, left flagged and logged, so one poison body can't wedge the queue."""
+    try:
+        with conn.transaction():
+            with conn.cursor() as cur:
+                cur.execute(
+                    "UPDATE memory SET embed_attempts=embed_attempts+1, "
+                    "embed_failed_at=now() WHERE id=%s RETURNING embed_attempts",
+                    (memory_id,))
+                r = cur.fetchone()
+        attempts = r[0] if r else 0
+        if attempts >= cfg.embed_max_attempts:
+            _log.error("embed: memory %s failed %d× — dead-lettered (still flagged, "
+                       "out of rotation); investigate its body or the model",
+                       memory_id, attempts)
+        else:
+            _log.warning("embed: memory %s failed (attempt %d/%d); backing off %.0fs",
+                         memory_id, attempts, cfg.embed_max_attempts,
+                         cfg.embed_retry_backoff_s)
+    except Exception:
+        _log.exception("embed: could not record failure for %s", memory_id)
+
+
+def drain(conn, cfg, embedder, backend, limit: Optional[int] = None) -> int:
+    """Embed eligible pending memories oldest-first until none remain. Returns the
+    count successfully embedded.
+
+    Claim-based, so MANY workers (separate memgres-worker containers, or several
+    server replicas each with an in-process worker) drain the same queue without
+    duplicating work: each row is claimed with ``FOR UPDATE SKIP LOCKED`` inside
+    the transaction that embeds it, so a concurrent worker skips a row already
+    being handled rather than embedding it again or blocking on it. The lock is
+    held only for that one row's embed and released on commit; the flag is cleared
+    in the same transaction, so a crash mid-embed rolls back and leaves the row
+    pending for another pass (crash-safe).
+
+    A row that FAILS to embed does not stop the pass and does not wedge the queue:
+    its attempt count + failure time are recorded (``_record_failure``), the claim
+    skips it for a back-off window, and after ``embed_max_attempts`` it drops out
+    of rotation (a logged dead letter) so newer rows always make progress. Only a
+    claim/connection-level error ends the pass (the next tick reconnects).
+
+    ``limit`` caps how many rows this call embeds (default: everything eligible)."""
+    total = 0
+    while limit is None or total < limit:
+        row = None
+        try:
+            with conn.transaction():
+                with conn.cursor() as cur:
+                    cur.execute(
+                        "SELECT id, body, namespace, content_hash FROM memory "
+                        "WHERE embed_pending AND embed_attempts < %s "
+                        "AND (embed_failed_at IS NULL "
+                        "     OR embed_failed_at < now() - make_interval(secs => %s)) "
+                        "ORDER BY updated_at LIMIT 1 FOR UPDATE SKIP LOCKED",
+                        (cfg.embed_max_attempts, cfg.embed_retry_backoff_s))
+                    row = cur.fetchone()
+                if row is None:
+                    break                # nothing eligible & unclaimed → done
+                mid, body, ns, chash = row
+                index_memory(conn, cfg, embedder, backend, str(mid),
+                             body or "", ns, chash)
+            total += 1                    # committed → row done
+        except Exception:
+            if row is None:               # the claim itself failed → connection issue
+                _log.exception("embed worker: claim failed; ending this pass")
+                break
+            # A single row's embed failed: record it and SKIP FORWARD, so a poison
+            # row never blocks the rest of the queue.
+            _log.warning("embed worker: embedding memory %s failed", row[0])
+            _record_failure(conn, cfg, str(row[0]))
+            continue
+    return total

{memgres-0.3.2 → memgres-0.4.0}/memgres/info.py

@@ -11,20 +11,29 @@ from __future__ import annotations
 
 from typing import Optional
 
+from ._version import __version__
 from .config import Config
+from .schema import SCHEMA_VERSION
 
 
 def server_info(cfg: Config, embed_dim: Optional[int] = None) -> dict:
     """Effective limits + capabilities, from the loaded config (and the live
-    embedder's dimension when available). Never includes secrets."""
+    embedder's dimension when available). Never includes secrets.
+
+    ``version`` is the running package version (from code, so an editable/dev
+    checkout reports what it's actually running); ``schema_version`` is the DB
+    layout this build migrates to."""
     lexical_only = cfg.embed_provider == "none"
     dim = embed_dim if embed_dim is not None else (cfg.embed_dim or None)
     return {
+        "version": __version__,
+        "schema_version": SCHEMA_VERSION,
         "limits": {
             "max_body_bytes": cfg.max_body_bytes,
             "max_write_bytes": cfg.max_write_bytes,
             "max_source_bytes": cfg.max_source_bytes,
             "max_reason_bytes": cfg.max_reason_bytes,
+            "max_title_bytes": cfg.max_title_bytes,
         },
         "embed": {
             "provider": cfg.embed_provider,