loop-memory 0.4.17__tar.gz → 0.4.18__tar.gz

{loop_memory-0.4.17 → loop_memory-0.4.18}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: loop-memory
-Version: 0.4.17
+Version: 0.4.18
 Summary: A general-purpose, local memory system for every AI agent you run. Loop Memory auto-captures conversations from Codex / Claude / Hermes / OpenClaw, scores them by importance × recency × usage × feedback, distils them into a curated wiki, and serves everything from a single web UI.
 Author: Loop Memory contributors <loop-memory@users.noreply.github.com>
 License-Expression: MIT
@@ -66,6 +66,8 @@ Dynamic: license-file
 
 ---
 
+> **What's new in 0.4.18** — **the BM25 channel no longer publishes a score that collapses to zero, and there is finally a benchmark that measures it** (audit 2026-11-01, weekly ecosystem sync). SQLite's FTS5 `bm25()` computes the classic idf `log((N - n + 0.5) / (n + 0.5))` and **clamps every non-positive idf to `1e-6`** — so a query term present in 49 % of the corpus keeps a real idf while one present in 51 % has its weight annihilated, a factor of ~40 000 between two almost equally discriminative terms. With the `trigram` tokenizer that is the *normal* case for CJK (`知识图` alone occurs in more than half of any knowledge-graph-heavy corpus), and the visible symptom was the 0.4.15 evidence chain reporting **`bm25: 0.0`** — "this hit because bm25 0.0". The published score is now a textbook BM25 (`k1 = 1.2`, `b = 0.75`) with the Lucene / BM25+ idf `log(1 + (N - n + 0.5) / (n + 0.5))`, which is strictly positive for every term, so the scale cannot collapse. **A document that contains the query verbatim now outranks one that merely contains the same scattered trigrams** — the transferable half of `rohitg00/agentmemory` v0.9.31's prefix-score fix (2026-10-26: strict recall-all@5 **65.74 % → 84.68 %**, +18.94 pp), whose release note is careful to call its numbers "component retrieval results on inspected development data, not held-out answer accuracy". The `LIKE` fallback and the FTS5 hits are finally on the **same scale** (a `LIKE`-only hit used to publish the memory's `importance` under the name `bm25`). The candidate set is unchanged — the fix re-ranks, it does not add or drop documents. New **`loop-memory bench`**: a deterministic retrieval-quality benchmark over a fixed, versioned fixture (30 memories, 12 wiki pages, 11 judged queries) reporting **recall@1/3/5, MRR@10, nDCG@10** with the fixture's revision and the SHA-256 of its bytes; the run is fully offline (**no provider API is called, $0, no network**) and builds its store in a throwaway directory. On the shipped fixture the channel scored **MRR@10 = 0.6061, recall@1 = 0.2273, nDCG@10 = 0.7078** before and **1.0000 / 0.9545 / 1.0000** after — and the report keeps the raw FTS5 score next to the repaired one, so the defect stays visible instead of being quietly fixed and forgotten. Source: *agentmemory* v0.9.31 evaluation methodology + the FTS5 idf clamp in SQLite's `fts5_aux.c` (Apache-2.0 — *pattern only, no code copied*); honcho v3.4.1's "a filter must not silently drop a term" and v3.5.0's "a conclusion must cite its source" were reviewed and need no code (AGPL-3.0 — *pattern only*). 37 new regression cases; suite at **995 passed / 4 skipped**. [Full CHANGELOG →](./CHANGELOG.md#0418---2026-11-01)
+> **What's new in 0.4.17** — **a dry-run dream pass no longer writes to the database** (audit 2026-10-25). `loop-memory dream` and `POST /api/v1/dream` have always documented themselves as *"nothing is written unless `apply=True`"* — but the pass's decay stage called `MemoryStore.rescore_all()` unconditionally, which rewrote the `memories.score` column of **every** row while claiming to only inspect the store. The expiry and merge decisions were correctly left unapplied; the score decay was not. `rescore_all()` now takes a `dry_run` flag: it computes every new score and reports how many rows *would* change but issues no `UPDATE`, and the dream pass runs it with `dry_run=not apply`. A dry run still reports the size of the decay job through `report.rescored` while touching nothing. The contract wording is now exact rather than aspirational — a dry run changes **no memory data** (no expiry, no supersession, no rescore); the single write it still performs is one `cognitive_audit` row per proposed action with `action="suggest"` instead of `"applied"`, because that row *is* the reviewable plan the dashboard shows before you commit to it (`record_audit=False` suppresses even that). Also fixed in this release: `publish.yml` now sends `make_latest` as a JSON **boolean** (`gh api -F`) instead of a JSON **string** — GitHub's release API ignores the string form, so the "mark as latest" step had been returning 200 and reporting success while promoting nothing, leaving `/releases/latest` pointing at v0.4.14 through the 0.4.15 and 0.4.16 ships. The step now also re-reads `/releases/latest` and fails the job when the tag does not match, instead of trusting the exit code. 3 new regression cases (`tests/test_dream_2026_10_25.py`); suite at **958 passed / 4 skipped**. [Full CHANGELOG →](./CHANGELOG.md#0417---2026-10-25)
 > **What's new in 0.4.16** — **subject-key logical fact slots + `latest_only` recall + the "dream" sleep-time consolidation pass** (audit 2026-10-25, weekly ecosystem sync — Mem0 v2.2.1 "Dream", arXiv 2605.27366 "In Defense of Forgotten", Zep/Graphiti temporal edges, Letta memory blocks). New nullable `subject_key` column on `memories` names the *logical slot* a fact occupies (`"editor.preference"`); `upsert_memory(subject_key=...)` makes the new row the live version and soft-supersedes the previous one with a `superseded_by` pointer, so the version history survives and `recall_as_of()` can still answer "what did we believe last week?". The content-hash dedup path is now slot-aware — two different slots holding the same text stay two rows. `SCHEMA_VERSION` bumps 12 → 13 with an idempotent in-place migration. New `latest_only=False` flag on `recall()` / `recall_hybrid()` (Mem0 v2 `SearchMemoryOptions.latest_only` parity) collapses the answer in three layers: rows outside their validity window at *now* are dropped, a subject-key slot resolves to its newest **live** row (looked up in the store, so a stale version never comes back just because it matched the query better), and near-duplicate texts cluster by token Jaccard (default 0.85) with the newest row winning. Wired through `GET /api/recall?latest_only=1`, `GET /api/v1/recall?latest_only=1`, `loop-memory recall --latest-only` and `MemoryClient.recall(latest_only=True)`. New `loop_memory/jobs/dream.py` implements the sleep-time consolidation Mem0 v2 calls the "Dream" — deterministic, zero-dep and **non-destructive**: it decays scores, then judges every row past its deadline on **utility** (`0.45 × importance + 0.30 × usage + 0.25 × recency`) so a frequently-recalled old fact is *protected* while a never-used stale row is **soft-expired** (`invalid_at` stamped, row kept, history kept) instead of deleted, and finally merges near-duplicates into a supersession chain. Run it with `loop-memory dream [--apply] [--json]` or `POST /api/v1/dream` — it is a **dry run by default** and every decision lands in `cognitive_audit` so the dashboard can show and revert it. New store primitives `expire_memory()` / `revive_memory()`, and `memory_stats()` now reports `subject_key`, `ttl`, `valid_at`, `invalid_at` and a derived `ttl_state`. **Also fixed a real defect**: the hybrid recall path (BM25 / LIKE / semantic channels) did not filter `superseded_by`, so `recall_hybrid()` — the production path behind the CLI, MCP server and dashboard — returned both versions of a superseded fact. Source: Mem0 v2.2.1 Dream + ADD-only architecture + arXiv 2605.27366 utility-aware eviction + Zep/Graphiti bi-temporal edges (pattern only, no code copied). 77 new regression cases. [Full CHANGELOG →](./CHANGELOG.md#0416---2026-10-25)
 > **What's new in 0.4.15** — **content-hash dedup + bi-temporal validity window + evidence chain + `@agent_memory` decorator** (audit 2026-10-18, weekly ecosystem sync). Three new columns on `memories`: `content_hash` (SHA-256 of normalised text, indexed), `valid_at` (epoch the memory becomes active), `invalid_at` (epoch it expires — NULL means still open-ended). The schema migration bumps `SCHEMA_VERSION` 11 → 12 with an idempotent backfill so legacy DBs upgrade in-place. New `upsert_memory(valid_at=, invalid_at=, dedup=)` parameters let callers schedule future-dated or expiring memories and opt out of content-hash dedup for episodic event streams. `recall()` and `recall_hybrid()` now accept an optional `as_of_ts` (float epoch or ISO-8601) — when set, both methods drop memories whose `valid_at > as_of` or `invalid_at <= as_of`, complementing the existing `recall_as_of()` supersession-chain walk. Every recall hit now carries an `evidence: [{field, value, weight}]` array — BM25 / semantic / entity / rrf_total / high_importance channels are exposed individually so a UI can render "this hit because ... [bm25 1.0] [entity 0.7]" instead of a black-box score (Cognee 1.0 / Mem0 v2 evidence-citation pattern, zero-dep). New `loop_memory.decorators` exposes `@agent_memory(store, agent_id=, limit=)` — a one-line wrapper that recalls the top-k relevant memories before the body runs and writes a fresh `kind="interaction"` memory after. Strings returned from the wrapped function are stored verbatim; dict/list returns are JSON-serialised; the recall list is injected as a `memory_context` keyword when the function declares it, otherwise attached to dict results under `memories`. Sister `remember(store, text, ...)` helper for seeding. The BM25 `_like_fallback` now falls through to per-token OR-LIKE when the whole-string match returns nothing — natural-language questions like "What is the deploy process?" now hit stored facts even when FTS5 trigram can't match quoted exact phrases. Source: Cognee 1.0 ECL pipeline + Mem0 v2 hybrid fusion + Zep/Graphiti bi-temporal validity fields + Letta block-memory (pattern only, no code copied). 59 new regression cases. [Full CHANGELOG →](./CHANGELOG.md#0415---2026-10-18)
 > **What's new in 0.4.14** — **`loop-memory doctor` deep store-health check** (audit 2026-10-12). Ten diagnostic checks — `db_exists`, `db_integrity`, `schema_version`, `audit_triggers`, `audit_log_writable`, `audit_log_size`, `hooks_installed`, `version_consistent`, `wiki_no_orphans`, `recall_paths_works` — report `ok`/`info`/`warn`/`error` for the live `MemoryStore`. `--fix` auto-repairs the `audit_triggers` check. `--json` emits a stable shape for CI. `--list` prints the check function names without touching the store. `--verbose` shows `info`-level checks. Exit code `1` on any `error` status, `0` otherwise. Source: `cortexkit/magic-context doctor` auto-diagnostic pattern (MIT, 2026-10-12) + `dsh-memento` write-approval-gate pattern (Apache-2.0, 2026-10-12). Stdlib-only — we re-implement from scratch against our own `MemoryStore`. Also fixed a Python 3.14 bug: chained `store._conn().__enter__()` immediately triggers the contextmanager's `finally` and closes the connection mid-query — both call sites in `doctor.py` now use explicit `with` form. 27 new regression cases. [Full CHANGELOG →](./CHANGELOG.md#0414---2026-10-12)
@@ -251,6 +253,9 @@ loop-memory consolidate          # rescore + GC + dedupe
 # 4. Optional: sleep-time consolidation (soft expiry, never deletes)
 loop-memory dream                # dry run -- what would expire / merge
 loop-memory dream --apply        # write it (soft-expire + supersede)
+
+# 5. Optional: measure the retrieval quality (offline, no API calls)
+loop-memory bench                # recall@k / MRR / nDCG over a fixed fixture
 ```
 
 ---

{loop_memory-0.4.17 → loop_memory-0.4.18}/README.md

@@ -22,6 +22,8 @@
 
 ---
 
+> **What's new in 0.4.18** — **the BM25 channel no longer publishes a score that collapses to zero, and there is finally a benchmark that measures it** (audit 2026-11-01, weekly ecosystem sync). SQLite's FTS5 `bm25()` computes the classic idf `log((N - n + 0.5) / (n + 0.5))` and **clamps every non-positive idf to `1e-6`** — so a query term present in 49 % of the corpus keeps a real idf while one present in 51 % has its weight annihilated, a factor of ~40 000 between two almost equally discriminative terms. With the `trigram` tokenizer that is the *normal* case for CJK (`知识图` alone occurs in more than half of any knowledge-graph-heavy corpus), and the visible symptom was the 0.4.15 evidence chain reporting **`bm25: 0.0`** — "this hit because bm25 0.0". The published score is now a textbook BM25 (`k1 = 1.2`, `b = 0.75`) with the Lucene / BM25+ idf `log(1 + (N - n + 0.5) / (n + 0.5))`, which is strictly positive for every term, so the scale cannot collapse. **A document that contains the query verbatim now outranks one that merely contains the same scattered trigrams** — the transferable half of `rohitg00/agentmemory` v0.9.31's prefix-score fix (2026-10-26: strict recall-all@5 **65.74 % → 84.68 %**, +18.94 pp), whose release note is careful to call its numbers "component retrieval results on inspected development data, not held-out answer accuracy". The `LIKE` fallback and the FTS5 hits are finally on the **same scale** (a `LIKE`-only hit used to publish the memory's `importance` under the name `bm25`). The candidate set is unchanged — the fix re-ranks, it does not add or drop documents. New **`loop-memory bench`**: a deterministic retrieval-quality benchmark over a fixed, versioned fixture (30 memories, 12 wiki pages, 11 judged queries) reporting **recall@1/3/5, MRR@10, nDCG@10** with the fixture's revision and the SHA-256 of its bytes; the run is fully offline (**no provider API is called, $0, no network**) and builds its store in a throwaway directory. On the shipped fixture the channel scored **MRR@10 = 0.6061, recall@1 = 0.2273, nDCG@10 = 0.7078** before and **1.0000 / 0.9545 / 1.0000** after — and the report keeps the raw FTS5 score next to the repaired one, so the defect stays visible instead of being quietly fixed and forgotten. Source: *agentmemory* v0.9.31 evaluation methodology + the FTS5 idf clamp in SQLite's `fts5_aux.c` (Apache-2.0 — *pattern only, no code copied*); honcho v3.4.1's "a filter must not silently drop a term" and v3.5.0's "a conclusion must cite its source" were reviewed and need no code (AGPL-3.0 — *pattern only*). 37 new regression cases; suite at **995 passed / 4 skipped**. [Full CHANGELOG →](./CHANGELOG.md#0418---2026-11-01)
+> **What's new in 0.4.17** — **a dry-run dream pass no longer writes to the database** (audit 2026-10-25). `loop-memory dream` and `POST /api/v1/dream` have always documented themselves as *"nothing is written unless `apply=True`"* — but the pass's decay stage called `MemoryStore.rescore_all()` unconditionally, which rewrote the `memories.score` column of **every** row while claiming to only inspect the store. The expiry and merge decisions were correctly left unapplied; the score decay was not. `rescore_all()` now takes a `dry_run` flag: it computes every new score and reports how many rows *would* change but issues no `UPDATE`, and the dream pass runs it with `dry_run=not apply`. A dry run still reports the size of the decay job through `report.rescored` while touching nothing. The contract wording is now exact rather than aspirational — a dry run changes **no memory data** (no expiry, no supersession, no rescore); the single write it still performs is one `cognitive_audit` row per proposed action with `action="suggest"` instead of `"applied"`, because that row *is* the reviewable plan the dashboard shows before you commit to it (`record_audit=False` suppresses even that). Also fixed in this release: `publish.yml` now sends `make_latest` as a JSON **boolean** (`gh api -F`) instead of a JSON **string** — GitHub's release API ignores the string form, so the "mark as latest" step had been returning 200 and reporting success while promoting nothing, leaving `/releases/latest` pointing at v0.4.14 through the 0.4.15 and 0.4.16 ships. The step now also re-reads `/releases/latest` and fails the job when the tag does not match, instead of trusting the exit code. 3 new regression cases (`tests/test_dream_2026_10_25.py`); suite at **958 passed / 4 skipped**. [Full CHANGELOG →](./CHANGELOG.md#0417---2026-10-25)
 > **What's new in 0.4.16** — **subject-key logical fact slots + `latest_only` recall + the "dream" sleep-time consolidation pass** (audit 2026-10-25, weekly ecosystem sync — Mem0 v2.2.1 "Dream", arXiv 2605.27366 "In Defense of Forgotten", Zep/Graphiti temporal edges, Letta memory blocks). New nullable `subject_key` column on `memories` names the *logical slot* a fact occupies (`"editor.preference"`); `upsert_memory(subject_key=...)` makes the new row the live version and soft-supersedes the previous one with a `superseded_by` pointer, so the version history survives and `recall_as_of()` can still answer "what did we believe last week?". The content-hash dedup path is now slot-aware — two different slots holding the same text stay two rows. `SCHEMA_VERSION` bumps 12 → 13 with an idempotent in-place migration. New `latest_only=False` flag on `recall()` / `recall_hybrid()` (Mem0 v2 `SearchMemoryOptions.latest_only` parity) collapses the answer in three layers: rows outside their validity window at *now* are dropped, a subject-key slot resolves to its newest **live** row (looked up in the store, so a stale version never comes back just because it matched the query better), and near-duplicate texts cluster by token Jaccard (default 0.85) with the newest row winning. Wired through `GET /api/recall?latest_only=1`, `GET /api/v1/recall?latest_only=1`, `loop-memory recall --latest-only` and `MemoryClient.recall(latest_only=True)`. New `loop_memory/jobs/dream.py` implements the sleep-time consolidation Mem0 v2 calls the "Dream" — deterministic, zero-dep and **non-destructive**: it decays scores, then judges every row past its deadline on **utility** (`0.45 × importance + 0.30 × usage + 0.25 × recency`) so a frequently-recalled old fact is *protected* while a never-used stale row is **soft-expired** (`invalid_at` stamped, row kept, history kept) instead of deleted, and finally merges near-duplicates into a supersession chain. Run it with `loop-memory dream [--apply] [--json]` or `POST /api/v1/dream` — it is a **dry run by default** and every decision lands in `cognitive_audit` so the dashboard can show and revert it. New store primitives `expire_memory()` / `revive_memory()`, and `memory_stats()` now reports `subject_key`, `ttl`, `valid_at`, `invalid_at` and a derived `ttl_state`. **Also fixed a real defect**: the hybrid recall path (BM25 / LIKE / semantic channels) did not filter `superseded_by`, so `recall_hybrid()` — the production path behind the CLI, MCP server and dashboard — returned both versions of a superseded fact. Source: Mem0 v2.2.1 Dream + ADD-only architecture + arXiv 2605.27366 utility-aware eviction + Zep/Graphiti bi-temporal edges (pattern only, no code copied). 77 new regression cases. [Full CHANGELOG →](./CHANGELOG.md#0416---2026-10-25)
 > **What's new in 0.4.15** — **content-hash dedup + bi-temporal validity window + evidence chain + `@agent_memory` decorator** (audit 2026-10-18, weekly ecosystem sync). Three new columns on `memories`: `content_hash` (SHA-256 of normalised text, indexed), `valid_at` (epoch the memory becomes active), `invalid_at` (epoch it expires — NULL means still open-ended). The schema migration bumps `SCHEMA_VERSION` 11 → 12 with an idempotent backfill so legacy DBs upgrade in-place. New `upsert_memory(valid_at=, invalid_at=, dedup=)` parameters let callers schedule future-dated or expiring memories and opt out of content-hash dedup for episodic event streams. `recall()` and `recall_hybrid()` now accept an optional `as_of_ts` (float epoch or ISO-8601) — when set, both methods drop memories whose `valid_at > as_of` or `invalid_at <= as_of`, complementing the existing `recall_as_of()` supersession-chain walk. Every recall hit now carries an `evidence: [{field, value, weight}]` array — BM25 / semantic / entity / rrf_total / high_importance channels are exposed individually so a UI can render "this hit because ... [bm25 1.0] [entity 0.7]" instead of a black-box score (Cognee 1.0 / Mem0 v2 evidence-citation pattern, zero-dep). New `loop_memory.decorators` exposes `@agent_memory(store, agent_id=, limit=)` — a one-line wrapper that recalls the top-k relevant memories before the body runs and writes a fresh `kind="interaction"` memory after. Strings returned from the wrapped function are stored verbatim; dict/list returns are JSON-serialised; the recall list is injected as a `memory_context` keyword when the function declares it, otherwise attached to dict results under `memories`. Sister `remember(store, text, ...)` helper for seeding. The BM25 `_like_fallback` now falls through to per-token OR-LIKE when the whole-string match returns nothing — natural-language questions like "What is the deploy process?" now hit stored facts even when FTS5 trigram can't match quoted exact phrases. Source: Cognee 1.0 ECL pipeline + Mem0 v2 hybrid fusion + Zep/Graphiti bi-temporal validity fields + Letta block-memory (pattern only, no code copied). 59 new regression cases. [Full CHANGELOG →](./CHANGELOG.md#0415---2026-10-18)
 > **What's new in 0.4.14** — **`loop-memory doctor` deep store-health check** (audit 2026-10-12). Ten diagnostic checks — `db_exists`, `db_integrity`, `schema_version`, `audit_triggers`, `audit_log_writable`, `audit_log_size`, `hooks_installed`, `version_consistent`, `wiki_no_orphans`, `recall_paths_works` — report `ok`/`info`/`warn`/`error` for the live `MemoryStore`. `--fix` auto-repairs the `audit_triggers` check. `--json` emits a stable shape for CI. `--list` prints the check function names without touching the store. `--verbose` shows `info`-level checks. Exit code `1` on any `error` status, `0` otherwise. Source: `cortexkit/magic-context doctor` auto-diagnostic pattern (MIT, 2026-10-12) + `dsh-memento` write-approval-gate pattern (Apache-2.0, 2026-10-12). Stdlib-only — we re-implement from scratch against our own `MemoryStore`. Also fixed a Python 3.14 bug: chained `store._conn().__enter__()` immediately triggers the contextmanager's `finally` and closes the connection mid-query — both call sites in `doctor.py` now use explicit `with` form. 27 new regression cases. [Full CHANGELOG →](./CHANGELOG.md#0414---2026-10-12)
@@ -207,6 +209,9 @@ loop-memory consolidate          # rescore + GC + dedupe
 # 4. Optional: sleep-time consolidation (soft expiry, never deletes)
 loop-memory dream                # dry run -- what would expire / merge
 loop-memory dream --apply        # write it (soft-expire + supersede)
+
+# 5. Optional: measure the retrieval quality (offline, no API calls)
+loop-memory bench                # recall@k / MRR / nDCG over a fixed fixture
 ```
 
 ---

loop_memory-0.4.18/loop_memory/bench/__init__.py

@@ -0,0 +1,42 @@
+"""Retrieval-quality benchmark for Loop Memory.
+
+Audit 2026-11-01 (loop-memory 0.4.18).
+
+Loop Memory shipped hybrid retrieval (BM25 + semantic + entity
+fused with RRF) without ever *measuring* whether the BM25 channel
+ranked anything sensibly.  That gap is the reason a FTS5 scoring
+quirk could sit unnoticed for a dozen releases: the ``trigram``
+tokenizer makes common CJK trigrams reach a prevalence where
+SQLite's ``bm25()`` collapses its inverse document frequency to a
+constant, which silently flattened the ranking.
+
+``rohitg00/agentmemory`` v0.9.31 (2026-10-26) fixed the same class
+of bug in its prefix-expansion path and reported the delta on a
+deterministic benchmark (strict recall-all@5 65.74% -> 84.68%,
++18.94 points on LongMemEval S500).  Its methodology is what this
+package copies:
+
+* a **fixed fixture** with an explicit revision number, so two runs
+  measure the same corpus;
+* a **SHA-256 of the fixture file** recorded in every report, so a
+  report says exactly which bytes it scored;
+* **explicit limitations** -- these numbers are *retrieval* results
+  over a synthetic corpus, not held-out answer accuracy, and no
+  provider API is called, so the run costs $0 and needs no network.
+
+The runner is deliberately dependency-free and offline.
+"""
+
+from .runner import (
+    BenchmarkReport,
+    default_fixture_path,
+    load_fixture,
+    run_benchmark,
+)
+
+__all__ = [
+    "BenchmarkReport",
+    "default_fixture_path",
+    "load_fixture",
+    "run_benchmark",
+]

loop_memory-0.4.18/loop_memory/bench/runner.py

@@ -0,0 +1,361 @@
+"""Deterministic retrieval-quality benchmark runner.
+
+Audit 2026-11-01 (loop-memory 0.4.18).
+
+Measures the **ranking** quality of the BM25 / lexical channel, not
+answer accuracy.  The numbers are computed over a fixed synthetic
+corpus so two runs are directly comparable; the fixture's SHA-256 is
+recorded in every report so a reader can tell exactly which bytes
+produced a score.
+
+Metrics
+-------
+``recall@k``
+    Fraction of the judged-relevant documents that appear in the top
+    ``k`` results (``k`` in 1, 3, 5).  Macro-averaged over queries.
+``mrr@10``
+    Mean reciprocal rank of the *first* relevant hit within the top
+    10.  ``1.0`` means the right answer was ranked first every time.
+``ndcg@10``
+    Normalised discounted cumulative gain with binary gains -- a
+    stricter view that also rewards putting *all* relevant hits high.
+``score_scale``
+    ``(min, max)`` of the raw channel score over the whole run.  This
+    is a *diagnostic*, not a quality metric: SQLite's ``bm25()``
+    clamps a non-positive inverse document frequency to ``1e-6``, so a
+    corpus in which a query term is common collapses the reported
+    scores to ~1e-6 while the *ordering* survives.  A run whose
+    ``score_scale`` max is below 1e-4 is reporting a degenerate scale
+    even if the ranking happens to look fine.
+
+Limitations (deliberate, and stated in every report)
+----------------------------------------------------
+* Synthetic corpus.  These are **development retrieval results**, not
+  held-out answer accuracy on a real user corpus.
+* No provider API is called; the run is offline and costs $0.
+* Only the lexical channel is measured.  The semantic channel needs an
+  embedding model and the entity channel needs a graph build, so
+  neither is included -- a benchmark that silently depends on a
+  downloaded model would not be reproducible.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import math
+import shutil
+import tempfile
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any
+
+DEFAULT_TOP_K = 10
+DEFAULT_K_CUTS = (1, 3, 5)
+
+#: A reported channel score below this counts as "degenerate scale".
+#: SQLite's FTS5 ``bm25()`` floors a non-positive idf at 1e-6, and a
+#: single clamped term in a two-term query keeps the whole score near
+#: that floor.  1e-4 leaves a wide margin over that floor while still
+#: flagging a real collapse.
+DEGENERATE_SCORE_FLOOR = 1e-4
+
+
+@dataclass
+class QueryResult:
+    """Per-query outcome of one benchmark run."""
+
+    query_id: str
+    query: str
+    channel: str
+    relevant: list[str]
+    ranked: list[str]
+    recall_at: dict[int, float] = field(default_factory=dict)
+    reciprocal_rank: float = 0.0
+    ndcg: float = 0.0
+    first_relevant_rank: int | None = None
+    score_min: float = 0.0
+    score_max: float = 0.0
+    fts_score_min: float = 0.0
+    fts_score_max: float = 0.0
+    degenerate_fts_score: bool = False
+    note: str = ""
+
+    def to_dict(self) -> dict[str, Any]:
+        return {
+            "query_id": self.query_id,
+            "query": self.query,
+            "channel": self.channel,
+            "relevant": list(self.relevant),
+            "ranked": list(self.ranked),
+            "recall_at": {str(k): round(v, 6) for k, v in self.recall_at.items()},
+            "reciprocal_rank": round(self.reciprocal_rank, 6),
+            "ndcg": round(self.ndcg, 6),
+            "first_relevant_rank": self.first_relevant_rank,
+            "score_min": self.score_min,
+            "score_max": self.score_max,
+            "fts_score_min": self.fts_score_min,
+            "fts_score_max": self.fts_score_max,
+            "degenerate_fts_score": self.degenerate_fts_score,
+            "note": self.note,
+        }
+
+
+@dataclass
+class BenchmarkReport:
+    """Aggregate outcome of one benchmark run."""
+
+    fixture_name: str
+    fixture_revision: int
+    fixture_sha256: str
+    top_k: int
+    queries: list[QueryResult] = field(default_factory=list)
+    metrics: dict[str, float] = field(default_factory=dict)
+    score_scale: tuple[float, float] = (0.0, 0.0)
+    degenerate_score_scale: bool = False
+    fts_score_scale: tuple[float, float] = (0.0, 0.0)
+    degenerate_fts_queries: int = 0
+    limitations: list[str] = field(default_factory=list)
+
+    def to_dict(self) -> dict[str, Any]:
+        return {
+            "fixture_name": self.fixture_name,
+            "fixture_revision": self.fixture_revision,
+            "fixture_sha256": self.fixture_sha256,
+            "top_k": self.top_k,
+            "metrics": {k: round(v, 6) for k, v in self.metrics.items()},
+            "score_scale": [self.score_scale[0], self.score_scale[1]],
+            "degenerate_score_scale": self.degenerate_score_scale,
+            "fts_score_scale": [self.fts_score_scale[0], self.fts_score_scale[1]],
+            "degenerate_fts_queries": self.degenerate_fts_queries,
+            "queries": [q.to_dict() for q in self.queries],
+            "limitations": list(self.limitations),
+        }
+
+
+def default_fixture_path() -> Path:
+    """Absolute path of the fixture shipped with the package."""
+    return Path(__file__).resolve().parent / "fixtures" / "default.json"
+
+
+def load_fixture(path: str | Path | None = None) -> tuple[dict[str, Any], str]:
+    """Load a benchmark fixture.
+
+    Returns ``(fixture, sha256)`` where ``sha256`` hashes the *raw
+    file bytes* -- the report therefore pins the exact input, not a
+    re-serialisation of it.
+    """
+    p = Path(path).expanduser() if path else default_fixture_path()
+    raw = p.read_bytes()
+    digest = hashlib.sha256(raw).hexdigest()
+    data = json.loads(raw.decode("utf-8"))
+    if not isinstance(data, dict):
+        raise ValueError(f"fixture {p} must contain a JSON object")
+    data.setdefault("name", p.stem)
+    data.setdefault("revision", 1)
+    data.setdefault("corpus", [])
+    data.setdefault("wiki", [])
+    data.setdefault("queries", [])
+    return data, digest
+
+
+def _recall(ranked: list[str], relevant: list[str], k: int) -> float:
+    if not relevant:
+        return 0.0
+    top = set(ranked[:k])
+    hit = sum(1 for r in relevant if r in top)
+    return hit / float(len(relevant))
+
+
+def _reciprocal_rank(ranked: list[str], relevant: list[str], top_k: int) -> tuple[float, int | None]:
+    rel = set(relevant)
+    for idx, doc_id in enumerate(ranked[:top_k], start=1):
+        if doc_id in rel:
+            return 1.0 / idx, idx
+    return 0.0, None
+
+
+def _ndcg(ranked: list[str], relevant: list[str], top_k: int) -> float:
+    if not relevant:
+        return 0.0
+    rel = set(relevant)
+    dcg = 0.0
+    for idx, doc_id in enumerate(ranked[:top_k], start=1):
+        if doc_id in rel:
+            dcg += 1.0 / math.log2(idx + 1)
+    ideal_hits = min(len(rel), top_k)
+    idcg = sum(1.0 / math.log2(i + 1) for i in range(1, ideal_hits + 1))
+    return (dcg / idcg) if idcg > 0 else 0.0
+
+
+def _ingest(store: Any, fixture: dict[str, Any]) -> dict[str, str]:
+    """Load the fixture into ``store``; return a slug->id map."""
+    for row in fixture.get("corpus") or []:
+        store.upsert_memory(
+            id=row["id"],
+            kind=row.get("kind", "fact"),
+            text=row["text"],
+            importance=float(row.get("importance", 0.5)),
+            tags=list(row.get("tags") or []),
+            source=row.get("source"),
+        )
+    slug_to_id: dict[str, str] = {}
+    for row in fixture.get("wiki") or []:
+        store.upsert_wiki_page(
+            slug=row["slug"],
+            title=row.get("title", row["slug"]),
+            body=row.get("body", ""),
+            summary=row.get("summary"),
+            tags=list(row.get("tags") or []),
+            importance=float(row.get("importance", 0.5)),
+        )
+        page = store.get_wiki_page_by_slug(row["slug"])
+        if page:
+            slug_to_id[row["slug"]] = page["id"]
+    return slug_to_id
+
+
+def run_benchmark(
+    fixture: dict[str, Any] | None = None,
+    *,
+    fixture_path: str | Path | None = None,
+    fixture_sha256: str | None = None,
+    top_k: int = DEFAULT_TOP_K,
+) -> BenchmarkReport:
+    """Run the benchmark and return a :class:`BenchmarkReport`.
+
+    The store is built in a throwaway temporary directory so the
+    benchmark never touches the user's real database.
+    """
+    from ..storage.retrieval import bm25_search
+    from ..storage.sqlite_store import MemoryStore
+
+    if fixture is None:
+        fixture, digest = load_fixture(fixture_path)
+    else:
+        digest = fixture_sha256 or ""
+    if not digest:
+        raw = json.dumps(fixture, ensure_ascii=False, sort_keys=True).encode("utf-8")
+        digest = hashlib.sha256(raw).hexdigest()
+
+    tmpdir = tempfile.mkdtemp(prefix="loop-memory-bench-")
+    try:
+        store = MemoryStore(Path(tmpdir) / "bench.db")
+        slug_to_id = _ingest(store, fixture)
+
+        results: list[QueryResult] = []
+        score_min = float("inf")
+        score_max = float("-inf")
+        fts_min = float("inf")
+        fts_max = float("-inf")
+
+        for q in fixture.get("queries") or []:
+            channel = q.get("channel", "memories")
+            kind = "wiki" if channel == "wiki" else "memories"
+            relevant = []
+            for ref in q.get("relevant") or []:
+                relevant.append(slug_to_id.get(ref, ref))
+            hits = bm25_search(store, q["query"], kind=kind, limit=top_k)
+            ranked = [h["id"] for h in hits]
+            scores = [float(h.get("_score") or 0.0) for h in hits]
+            if scores:
+                score_min = min(score_min, min(scores))
+                score_max = max(score_max, max(scores))
+            fts_scores = [
+                float(h["_fts_score"])
+                for h in hits
+                if h.get("_fts_score") is not None
+            ]
+            if fts_scores:
+                fts_min = min(fts_min, min(fts_scores))
+                fts_max = max(fts_max, max(fts_scores))
+            degenerate_fts = bool(fts_scores) and max(fts_scores) < DEGENERATE_SCORE_FLOOR
+            rr, first_rank = _reciprocal_rank(ranked, relevant, top_k)
+            results.append(
+                QueryResult(
+                    query_id=q.get("id", q["query"]),
+                    query=q["query"],
+                    channel=channel,
+                    relevant=relevant,
+                    ranked=ranked,
+                    recall_at={k: _recall(ranked, relevant, k) for k in DEFAULT_K_CUTS},
+                    reciprocal_rank=rr,
+                    ndcg=_ndcg(ranked, relevant, top_k),
+                    first_relevant_rank=first_rank,
+                    score_min=min(scores) if scores else 0.0,
+                    score_max=max(scores) if scores else 0.0,
+                    fts_score_min=min(fts_scores) if fts_scores else 0.0,
+                    fts_score_max=max(fts_scores) if fts_scores else 0.0,
+                    degenerate_fts_score=degenerate_fts,
+                    note=q.get("note", ""),
+                )
+            )
+
+        n = len(results) or 1
+        metrics = {
+            f"recall@{k}": sum(r.recall_at[k] for r in results) / n for k in DEFAULT_K_CUTS
+        }
+        metrics[f"mrr@{top_k}"] = sum(r.reciprocal_rank for r in results) / n
+        metrics[f"ndcg@{top_k}"] = sum(r.ndcg for r in results) / n
+
+        if score_min is float("inf"):
+            score_min = score_max = 0.0
+        if fts_min is float("inf"):
+            fts_min = fts_max = 0.0
+
+        return BenchmarkReport(
+            fixture_name=fixture.get("name", "unknown"),
+            fixture_revision=int(fixture.get("revision", 1)),
+            fixture_sha256=digest,
+            top_k=top_k,
+            queries=results,
+            metrics=metrics,
+            score_scale=(score_min, score_max),
+            degenerate_score_scale=score_max < DEGENERATE_SCORE_FLOOR,
+            fts_score_scale=(fts_min, fts_max),
+            degenerate_fts_queries=sum(1 for r in results if r.degenerate_fts_score),
+            limitations=[
+                "Synthetic corpus: these are development retrieval results, "
+                "not held-out answer accuracy on a real user corpus.",
+                "No provider API is called; the run is offline and costs $0.",
+                "Only the lexical channel is measured; the semantic channel "
+                "needs an embedding model and the entity channel needs a "
+                "graph build, so neither is included.",
+            ],
+        )
+    finally:
+        shutil.rmtree(tmpdir, ignore_errors=True)
+
+
+def compare_reports(
+    baseline: dict[str, Any], current: dict[str, Any]
+) -> dict[str, Any]:
+    """Diff two report dicts (baseline first, current second).
+
+    Returns the per-metric deltas plus the fixture identity check.  A
+    comparison across *different* fixtures is meaningless, so the
+    result says so explicitly instead of printing a bogus delta.
+    """
+    same_fixture = (
+        baseline.get("fixture_sha256") == current.get("fixture_sha256")
+        and baseline.get("fixture_revision") == current.get("fixture_revision")
+    )
+    base_metrics = baseline.get("metrics") or {}
+    curr_metrics = current.get("metrics") or {}
+    deltas: dict[str, float] = {}
+    for key in sorted(set(base_metrics) | set(curr_metrics)):
+        deltas[key] = round(float(curr_metrics.get(key, 0.0)) - float(base_metrics.get(key, 0.0)), 6)
+    return {
+        "comparable": same_fixture,
+        "baseline_fixture": {
+            "name": baseline.get("fixture_name"),
+            "revision": baseline.get("fixture_revision"),
+            "sha256": baseline.get("fixture_sha256"),
+        },
+        "current_fixture": {
+            "name": current.get("fixture_name"),
+            "revision": current.get("fixture_revision"),
+            "sha256": current.get("fixture_sha256"),
+        },
+        "metrics": deltas,
+    }

loop_memory-0.4.18/loop_memory/cli/commands/bench.py

@@ -0,0 +1,116 @@
+"""``loop-memory bench`` -- deterministic retrieval-quality benchmark.
+
+Audit 2026-11-01 (loop-memory 0.4.18).
+
+Loop Memory had hybrid retrieval but no way to *measure* whether the
+lexical channel ranked anything sensibly.  This command scores the
+lexical channel against a fixed, versioned fixture and prints the
+result as JSON, so a ranking change can be reviewed the way a test
+failure would be.
+
+The methodology follows ``rohitg00/agentmemory`` v0.9.31 (2026-10-26):
+a fixed corpus with an explicit revision, the SHA-256 of the fixture
+bytes recorded in every report, and an explicit statement of what the
+numbers do *not* prove.
+
+The run is fully offline and calls no provider API.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+
+def _print_text(report: dict, comparison: dict | None = None) -> None:
+    """Human-readable rendering of a benchmark report."""
+    lines = []
+    lines.append(
+        f"fixture : {report['fixture_name']} (revision {report['fixture_revision']})"
+    )
+    lines.append(f"sha256  : {report['fixture_sha256']}")
+    lines.append(f"top-k   : {report['top_k']}")
+    lines.append("")
+    lines.append("metrics")
+    for key in sorted(report.get("metrics") or {}):
+        lines.append(f"  {key:<12} {report['metrics'][key]:.4f}")
+    scale = report.get("score_scale") or [0, 0]
+    flag = "  <-- degenerate scale" if report.get("degenerate_score_scale") else ""
+    lines.append("")
+    lines.append(f"score scale : {scale[0]:.6g} .. {scale[1]:.6g}{flag}")
+    lines.append("")
+    lines.append("per query")
+    for q in report.get("queries") or []:
+        rank = q.get("first_relevant_rank")
+        rank_txt = f"rank {rank}" if rank is not None else "not in top-k"
+        lines.append(
+            f"  {q['query_id']:<4} {q['query'][:34]:<34} "
+            f"rr={q['reciprocal_rank']:.3f} ndcg={q['ndcg']:.3f} {rank_txt}"
+        )
+    if comparison is not None:
+        lines.append("")
+        lines.append("comparison vs baseline")
+        if not comparison.get("comparable"):
+            lines.append("  NOT COMPARABLE: the two runs used different fixtures.")
+        for key in sorted(comparison.get("metrics") or {}):
+            delta = comparison["metrics"][key]
+            sign = "+" if delta >= 0 else ""
+            lines.append(f"  {key:<12} {sign}{delta:.4f}")
+    lines.append("")
+    lines.append("limitations")
+    for item in report.get("limitations") or []:
+        lines.append(f"  - {item}")
+    print("\n".join(lines))
+
+
+def run_bench(args: list) -> int:
+    """``loop-memory bench [--fixture PATH] [--json] [--out PATH] [--compare PATH]``"""
+    p = argparse.ArgumentParser(prog="loop-memory bench")
+    p.add_argument(
+        "--fixture",
+        default=None,
+        help="path to a benchmark fixture (defaults to the packaged fixture)",
+    )
+    p.add_argument("--top-k", type=int, default=10, help="results to consider per query")
+    p.add_argument("--json", action="store_true", help="print the raw JSON report")
+    p.add_argument("--out", default=None, help="write the JSON report to this path")
+    p.add_argument(
+        "--compare",
+        default=None,
+        help="path to a previous JSON report; prints the per-metric delta",
+    )
+    ns = p.parse_args(args)
+
+    from ...bench.runner import compare_reports, load_fixture, run_benchmark
+
+    fixture, digest = load_fixture(ns.fixture)
+    report = run_benchmark(
+        fixture, fixture_sha256=digest, top_k=ns.top_k
+    ).to_dict()
+
+    comparison = None
+    if ns.compare:
+        try:
+            baseline = json.loads(Path(ns.compare).expanduser().read_text("utf-8"))
+        except Exception as exc:  # noqa: BLE001 - CLI boundary
+            print(f"cannot read baseline report: {exc}", file=sys.stderr)
+            return 2
+        comparison = compare_reports(baseline, report)
+
+    if ns.out:
+        out_path = Path(ns.out).expanduser()
+        payload = {"report": report}
+        if comparison is not None:
+            payload["comparison"] = comparison
+        out_path.parent.mkdir(parents=True, exist_ok=True)
+        out_path.write_text(
+            json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8"
+        )
+
+    if ns.json:
+        print(json.dumps(report, ensure_ascii=False, indent=2, default=str))
+    else:
+        _print_text(report, comparison)
+    return 0

{loop_memory-0.4.17 → loop_memory-0.4.18}/loop_memory/cli/main.py

@@ -32,12 +32,14 @@ Usage:
     loop-memory graph-edge <src> <dst> [--kind K] [--weight W]  # push a relation
     loop-memory subgraph <query>  # print a small subgraph
     loop-memory graph-rebuild  # rebuild entities + entity_mentions
+    loop-memory bench          # retrieval-quality benchmark (fixed fixture, offline)
 """
 
 from __future__ import annotations
 
 import sys
 
+from .commands import bench as bench_cmd
 from .commands import cognitive as cognitive_cmd
 from .commands import diag as diag_cmd
 from .commands import graph as graph_cmd
@@ -136,6 +138,9 @@ COMMANDS = {
     # Audit 2026-09-27: OKF v0.2 bundle import (rune + okf-agent-memory #12-#15 + Deja-Vu).
     "import-okf": cognitive_cmd.run_import_okf,
     "rules": rules_cmd.run_rules,
+    # Audit 2026-11-01 (loop-memory 0.4.18) -- deterministic
+    # retrieval-quality benchmark over a fixed, versioned fixture.
+    "bench": bench_cmd.run_bench,
     "version": _run_version,
 }
 
@@ -187,6 +192,7 @@ COMMAND_HELP: dict[str, str] = {
     # Audit 2026-09-13 — per-agent identity bootstrap (Mem0 CLI init --agent pattern).
     "init":                   "loop-memory init --agent <name> [--install-hooks]   # register an agent + (optionally) install hooks.",
     "rules":                 "loop-memory rules [--agent codex|claude|hermes|openclaw] [--write] [--force]\n  Print, or append into the agent's rule file, the three-phase memory discipline (task start / mid-task / wrap-up). Never overwrites user content.",
+    "bench":                 "loop-memory bench [--fixture PATH] [--top-k N] [--json] [--out PATH] [--compare PATH]   # deterministic retrieval-quality benchmark over a fixed, versioned fixture (audit 2026-11-01; agentmemory v0.9.31 methodology). Offline, no provider API called.",
 }