mnemo-engine 0.4.1__tar.gz → 0.5.0__tar.gz

{mnemo_engine-0.4.1 → mnemo_engine-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mnemo-engine
-Version: 0.4.1
+Version: 0.5.0
 Classifier: Programming Language :: Rust
 Classifier: Programming Language :: Python :: 3
 Classifier: License :: OSI Approved :: Apache Software License
@@ -161,6 +161,10 @@ its own agent.
 | `memories()` | Enumerate every live (non-deleted, non-expired) memory as a list of dicts |
 | `dimensions()` | Embedding dimensionality this database was created with (also on `stats()["dimensions"]`) |
 | `rekey(new_passphrase, fast=False)` | Change the passphrase. Re-derives the KEK and re-wraps the DEK; encrypted pages are not rewritten (O(1) in db size). `fast=True` uses the cheap-Argon2 test params — omit for production |
+| `set_embedder(fn, *, name, dimensions, normalize=True)` | Register a Python callable `embed(texts: list[str]) -> list[list[float]]` as this handle's text embedder (Phase 3). Dimension-checks against the database's width; identity-checks against (and on first use upserts) the scaffold manifest's `embedder_name` / `embedder_dimensions` / `embedder_normalize` keys, so swapping embedders between opens fails loudly |
+| `remember_text(content, memory_type, **kwargs)` | Embed `content` via the registered embedder and `remember` the result. Accepts the same per-memory kwargs as `remember` (`agent_id`, `importance`, `session_id`, `ttl_secs`, `shared`, `metadata`) |
+| `recall_text(query, top_k=10, memory_types=None, agent_id=None, track_access=True)` | Embed `query` via the registered embedder and run `recall`. Same return shape as `recall` |
+| `cache_get_semantic_text(namespace, text, threshold=0.97)` | Text-side semantic cache lookup; uses the embedder's `name` as the semantic cache's `model` component |
 | `stats()` | Summary statistics |
 | `export_encrypted(dest)` | Copy the (already-encrypted) file elsewhere |
 | `len(db)` | Live memory count |
@@ -370,6 +374,61 @@ your embedder's output — `text-embedding-3-small` is 1536-dim,
 consistent across hosts (use Redis / DynamoDB DAX). MNemo's cache is
 optimized for a single-host agent that owns its cache file.
 
+### Recipe: `set_embedder` + text-only `remember` / `recall`
+
+Phase 3 ships a per-handle embedder hook so callers never have to
+pass vectors by hand. Register once at open time; every subsequent
+`remember_text` / `recall_text` / `cache_get_semantic_text` call on
+the handle uses it.
+
+```python
+import mnemo
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer("all-MiniLM-L6-v2")  # 384-dim, normalized
+
+def embed(texts: list[str]) -> list[list[float]]:
+    return model.encode(texts, normalize_embeddings=True).tolist()
+
+db = mnemo.open("agent.mnemo", "pw", dimensions=384)
+db.set_embedder(embed, name="all-MiniLM-L6-v2",
+                dimensions=384, normalize=True)
+
+db.remember_text("the user prefers tea", "semantic",
+                 importance=0.8, agent_id="assistant")
+hits = db.recall_text("how does the user like their drinks?", top_k=5)
+for h in hits:
+    print(h["score"], h["content"])
+```
+
+Or with OpenAI's hosted embeddings:
+
+```python
+from openai import OpenAI
+client = OpenAI()
+
+def embed(texts: list[str]) -> list[list[float]]:
+    r = client.embeddings.create(model="text-embedding-3-small", input=texts)
+    return [item.embedding for item in r.data]
+
+db = mnemo.open("agent.mnemo", "pw", dimensions=1536)
+db.set_embedder(embed, name="text-embedding-3-small",
+                dimensions=1536, normalize=False)
+```
+
+The identity (`name`, `dimensions`, `normalize`) is persisted on the
+scaffold manifest the first time you `set_embedder`. Reopening with a
+*different* embedder — say you swap MiniLM for BGE on the same file —
+raises `RuntimeError: embedder identity mismatch: database manifest
+records embedder (name="all-MiniLM-L6-v2", …), but this handle was
+passed embedder (name="bge-large-en-v1.5", …). Mixing embedders
+silently corrupts recall quality — open the file with the original
+embedder, or create a fresh database for the new one.`
+
+Mnemo also now refuses any `remember` call with an all-zero vector,
+so a mis-wired embedder (returning `[0.0] * dims`) fails loudly on
+the first write instead of polluting recall silently.
+
 ## License
 
 Apache-2.0.

{mnemo_engine-0.4.1 → mnemo_engine-0.5.0}/README.md

@@ -147,6 +147,10 @@ its own agent.
 | `memories()` | Enumerate every live (non-deleted, non-expired) memory as a list of dicts |
 | `dimensions()` | Embedding dimensionality this database was created with (also on `stats()["dimensions"]`) |
 | `rekey(new_passphrase, fast=False)` | Change the passphrase. Re-derives the KEK and re-wraps the DEK; encrypted pages are not rewritten (O(1) in db size). `fast=True` uses the cheap-Argon2 test params — omit for production |
+| `set_embedder(fn, *, name, dimensions, normalize=True)` | Register a Python callable `embed(texts: list[str]) -> list[list[float]]` as this handle's text embedder (Phase 3). Dimension-checks against the database's width; identity-checks against (and on first use upserts) the scaffold manifest's `embedder_name` / `embedder_dimensions` / `embedder_normalize` keys, so swapping embedders between opens fails loudly |
+| `remember_text(content, memory_type, **kwargs)` | Embed `content` via the registered embedder and `remember` the result. Accepts the same per-memory kwargs as `remember` (`agent_id`, `importance`, `session_id`, `ttl_secs`, `shared`, `metadata`) |
+| `recall_text(query, top_k=10, memory_types=None, agent_id=None, track_access=True)` | Embed `query` via the registered embedder and run `recall`. Same return shape as `recall` |
+| `cache_get_semantic_text(namespace, text, threshold=0.97)` | Text-side semantic cache lookup; uses the embedder's `name` as the semantic cache's `model` component |
 | `stats()` | Summary statistics |
 | `export_encrypted(dest)` | Copy the (already-encrypted) file elsewhere |
 | `len(db)` | Live memory count |
@@ -356,6 +360,61 @@ your embedder's output — `text-embedding-3-small` is 1536-dim,
 consistent across hosts (use Redis / DynamoDB DAX). MNemo's cache is
 optimized for a single-host agent that owns its cache file.
 
+### Recipe: `set_embedder` + text-only `remember` / `recall`
+
+Phase 3 ships a per-handle embedder hook so callers never have to
+pass vectors by hand. Register once at open time; every subsequent
+`remember_text` / `recall_text` / `cache_get_semantic_text` call on
+the handle uses it.
+
+```python
+import mnemo
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer("all-MiniLM-L6-v2")  # 384-dim, normalized
+
+def embed(texts: list[str]) -> list[list[float]]:
+    return model.encode(texts, normalize_embeddings=True).tolist()
+
+db = mnemo.open("agent.mnemo", "pw", dimensions=384)
+db.set_embedder(embed, name="all-MiniLM-L6-v2",
+                dimensions=384, normalize=True)
+
+db.remember_text("the user prefers tea", "semantic",
+                 importance=0.8, agent_id="assistant")
+hits = db.recall_text("how does the user like their drinks?", top_k=5)
+for h in hits:
+    print(h["score"], h["content"])
+```
+
+Or with OpenAI's hosted embeddings:
+
+```python
+from openai import OpenAI
+client = OpenAI()
+
+def embed(texts: list[str]) -> list[list[float]]:
+    r = client.embeddings.create(model="text-embedding-3-small", input=texts)
+    return [item.embedding for item in r.data]
+
+db = mnemo.open("agent.mnemo", "pw", dimensions=1536)
+db.set_embedder(embed, name="text-embedding-3-small",
+                dimensions=1536, normalize=False)
+```
+
+The identity (`name`, `dimensions`, `normalize`) is persisted on the
+scaffold manifest the first time you `set_embedder`. Reopening with a
+*different* embedder — say you swap MiniLM for BGE on the same file —
+raises `RuntimeError: embedder identity mismatch: database manifest
+records embedder (name="all-MiniLM-L6-v2", …), but this handle was
+passed embedder (name="bge-large-en-v1.5", …). Mixing embedders
+silently corrupts recall quality — open the file with the original
+embedder, or create a fresh database for the new one.`
+
+Mnemo also now refuses any `remember` call with an all-zero vector,
+so a mis-wired embedder (returning `[0.0] * dims`) fails loudly on
+the first write instead of polluting recall silently.
+
 ## License
 
 Apache-2.0.

{mnemo_engine-0.4.1 → mnemo_engine-0.5.0}/mnemo/Cargo.lock

@@ -462,7 +462,7 @@ checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79"
 
 [[package]]
 name = "mnemo-engine"
-version = "0.4.1"
+version = "0.5.0"
 dependencies = [
  "aes-gcm",
  "argon2",
@@ -485,7 +485,7 @@ dependencies = [
 
 [[package]]
 name = "mnemo-node"
-version = "0.4.1"
+version = "0.5.0"
 dependencies = [
  "mnemo-engine",
  "napi",

{mnemo_engine-0.4.1 → mnemo_engine-0.5.0}/mnemo/Cargo.toml

@@ -6,7 +6,7 @@
 # below stay `mnemo`, so consumers still write `use mnemo::...` and invoke
 # `mnemo` on the command line after `cargo add mnemo-engine`.
 name = "mnemo-engine"
-version = "0.4.1"
+version = "0.5.0"
 edition = "2021"
 rust-version = "1.75"
 description = "Encrypted, single-file, portable agent memory engine with multi-signal recall, sessions, snapshots, and an IVF+PQ index"

{mnemo_engine-0.4.1 → mnemo_engine-0.5.0}/mnemo/README.md

@@ -41,10 +41,55 @@ mnemo serve --mcp path/to/agent.mnemo
 
 The passphrase is read from `MNEMO_PASSPHRASE` (never a CLI flag — that
 would leak into shell history and process listings). Tools exposed:
-`about`, `remember`, `recall`, `forget`, `list`, `snapshot_list`, `stats`.
-Every mutation flushes before returning. `remember` and `recall` are
-embedder-agnostic — the caller supplies vectors; text-only variants
-land with Phase 3 of the level-up plan.
+`about`, `remember`, `recall`, `forget`, `list`, `snapshot_list`, `stats`,
+`cache_put`, `cache_get`, `cache_stats`, `cache_put_semantic`,
+`cache_get_semantic`. Every mutation flushes before returning.
+
+#### Text-in via `--embed-cmd` (Phase 3)
+
+`remember`, `recall`, `cache_put_semantic`, and `cache_get_semantic`
+accept **either** a caller-supplied `vector`/`query` **or** a `text`
+string that the server embeds for them. Enable the text path by passing
+an embedder command at serve start:
+
+```sh
+MNEMO_PASSPHRASE=... mnemo serve --mcp agent.mnemo \
+  --embed-cmd "python3 mnemo/examples/embedder_server.py" \
+  --embed-name "sentence-transformers/all-MiniLM-L6-v2"
+```
+
+The embedder is a **persistent child process** spawned once at serve
+start. Protocol: it reads one JSON-encoded string per line from stdin
+and writes one JSON array of floats per line to stdout, flushed after
+every reply. See `mnemo/examples/embedder_server.py` for a 10-line
+reference implementation (sentence-transformers + MiniLM, 384-dim).
+
+> **Stdout is for replies only.** Any line the embedder prints to
+> stdout outside a reply — startup banners, model-load progress,
+> warnings, "model loaded" messages — is a protocol violation and
+> would shift every subsequent reply by one line. EmbedCmd detects
+> it, kills the child, and surfaces a clear error ("stdout is
+> reserved for replies only; log to stderr instead"); the next call
+> respawns the child. **Log to stderr.** Python: `print("...",
+> file=sys.stderr, flush=True)`, or configure `logging` with
+> `stream=sys.stderr`. Stderr is inherited to the parent and is
+> safe for anything you want to see while the server runs.
+
+Per-request timeout defaults to 60 seconds; override with
+`--embed-timeout <secs>` on `serve`, `remember`, and `recall`. Set
+it longer than your slowest cold-start reply (sentence-transformers'
+first `encode()` on CPU can be 3–5s). On timeout — or any other
+protocol error (malformed JSON, wrong dimensionality, zero vector,
+unexpected EOF) — the child is killed and the next call respawns a
+fresh child with the same argv/env/timeout. The guarantee is
+asymmetric and deliberate: **an error is allowed; a mismatched
+vector never is.**
+
+The `--embed-name` string is matched against (and persisted on) the
+scaffold manifest's `metadata.embedder = { name, dimensions, normalize }`
+object. Starting a server with a different `--embed-name` on an existing
+file fails loudly — mixing embedders in one file silently corrupts
+recall quality.
 
 Claude Desktop config snippet (`~/Library/Application Support/Claude/claude_desktop_config.json`):
 
@@ -53,13 +98,39 @@ Claude Desktop config snippet (`~/Library/Application Support/Claude/claude_desk
   "mcpServers": {
     "mnemo": {
       "command": "mnemo",
-      "args": ["serve", "--mcp", "/absolute/path/to/agent.mnemo"],
+      "args": [
+        "serve", "--mcp", "/absolute/path/to/agent.mnemo",
+        "--embed-cmd", "python3 /absolute/path/to/embedder_server.py",
+        "--embed-name", "sentence-transformers/all-MiniLM-L6-v2"
+      ],
       "env": { "MNEMO_PASSPHRASE": "your-passphrase-here" }
     }
   }
 }
 ```
 
+Omit `--embed-cmd`/`--embed-name` to keep the pre-Phase-3
+embedder-agnostic behavior (`vector`/`query` only).
+
+#### CLI text path
+
+The same `--embed-cmd` + `--embed-name` pair works on the one-shot
+CLI commands:
+
+```sh
+mnemo remember agent.mnemo --text "the user prefers tea" \
+  --type semantic --importance 0.8 --agent assistant \
+  --embed-cmd "python3 embedder_server.py" \
+  --embed-name "sentence-transformers/all-MiniLM-L6-v2"
+
+mnemo recall agent.mnemo --text "what drink?" --top-k 5 \
+  --embed-cmd "python3 embedder_server.py" \
+  --embed-name "sentence-transformers/all-MiniLM-L6-v2" \
+  --no-track
+```
+
+`--text` is mutually exclusive with `--query`/`--query-file`.
+
 The server holds an exclusive write lock on the file for its lifetime
 (see [Concurrency](#concurrency--read-only-opens)), so two MCP servers
 on the same file will conflict — one will fail to start with

mnemo_engine-0.5.0/mnemo/examples/embedder_server.py

@@ -0,0 +1,51 @@
+"""Reference embedder for `mnemo` --embed-cmd / MCP serve (Phase 3 PR 10b).
+
+Protocol:
+  * Read one JSON-encoded string per line from stdin.
+  * Write one JSON array of floats per line to stdout.
+  * Flush stdout after every reply, or mnemo will time out waiting.
+  * Close on EOF of stdin (which mnemo sends when the handle drops).
+
+The script runs persistently — one spawn per `Mnemo` handle, reused for
+every subsequent remember/recall/cache call. Keep heavy imports (model
+load) at module scope so the cost is paid once.
+
+**Stdout is reserved for replies only.** Any `print(...)` without
+`file=sys.stderr` — a startup banner, a "model loaded" line, a
+progress bar, a warning — shifts every subsequent reply by one line
+and mnemo's EmbedCmd will kill the child with a clear error.
+Log to stderr instead: `print("...", file=sys.stderr, flush=True)`.
+`sentence-transformers`/`transformers`/`torch` already log to stderr
+by default, so this is only a hazard for your own debug prints.
+
+Usage::
+
+    # CLI, one-shot:
+    mnemo remember notes.mnemo \\
+        --text "the user prefers tea" \\
+        --embed-cmd "python3 embedder_server.py" \\
+        --embed-name "sentence-transformers/all-MiniLM-L6-v2"
+
+    # MCP, persistent server:
+    MNEMO_PASSPHRASE=... mnemo serve --mcp notes.mnemo \\
+        --embed-cmd "python3 embedder_server.py" \\
+        --embed-name "sentence-transformers/all-MiniLM-L6-v2"
+
+Requires `sentence-transformers` + `torch`. Swap the model slug (and
+pass the matching --embed-name) to use something else; just keep the
+database's `dimensions` consistent with the embedder's output size.
+"""
+
+import json
+import sys
+
+from sentence_transformers import SentenceTransformer
+
+MODEL = "sentence-transformers/all-MiniLM-L6-v2"  # 384-dim, normalized
+
+model = SentenceTransformer(MODEL)
+
+for line in sys.stdin:
+    text = json.loads(line)
+    vector = model.encode([text], normalize_embeddings=True)[0].tolist()
+    print(json.dumps(vector), flush=True)