memtrust-cli 0.3.2__tar.gz → 0.3.3__tar.gz

{memtrust_cli-0.3.2 → memtrust_cli-0.3.3}/CHANGELOG.md

@@ -3,6 +3,75 @@
 All notable changes to this project are documented here. Format loosely follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
+## [0.3.3] - 2026-07-21
+
+### Fixed
+
+- **`memtrust --version` was still broken on the `memtrust` mirror package.** 0.3.2's fix
+  hardcoded the distribution-name lookup to `"memtrust-cli"`, which fixed that package but broke
+  `pip install memtrust` the same way in reverse: that environment has no `memtrust-cli` entry in
+  its own installed-package metadata, so the lookup always missed and fell through to
+  `0.0.0+unknown` again. `src/memtrust/__init__.py` now tries `memtrust-cli` first, falls back to
+  `memtrust`, and only reports `0.0.0+unknown` if neither distribution name is installed.
+  Independently reproduced against a clean `pip install memtrust==0.3.2` before this fix.
+- `memtrust run --help`'s `--eval` flag description was a hand-maintained string literal that had
+  drifted out of sync with the real `ALL_EVALS` list -- it was still missing `temporal_kg_boundary`
+  even though the eval itself worked correctly. Now generated directly from `ALL_EVALS`, so it
+  can't drift again.
+- `npm/memtrust-cli/package.json`'s `optionalDependencies` pinned the 6 platform packages
+  (`@memtrust-cli/darwin-x64` etc.) to an exact `0.1.0`, which had already drifted from the
+  actually-published `0.1.1`. Changed to `^0.1.1` so patch releases of the platform packages
+  resolve automatically instead of drifting stale again.
+- CI had been failing on every push since the 0.3.2 commit (`ruff format --check` alone, every
+  other job green) -- only `ruff check` had been run locally, not the separate formatting check.
+- README.md's "Success stories" section cited MemPalace/mempalace#524 (@gaby) as the source of a
+  ChromaDB-network-dependency finding. That citation was fabricated: the real issue #524 is
+  titled "Remove Baldfaced Lies Please," filed by a different user, about unrelated content, and
+  no user named "gaby" has filed anything in that repository. The underlying technical claim
+  (documented separately in `mempalace_adapter.py`'s own module docstring) is real; the citation
+  attributing it to a specific reported issue was not. Removed the bullet. Every other citation in
+  the section (~32 across MemPalace, mem0, Zep/Graphiti, OpenViking, and one cross-project
+  citation) was independently re-verified against the live GitHub API and confirmed accurate.
+- README.md's pasted `pytest --cov` output and coverage table were stale relative to the actual
+  test suite (585 passed / 93% vs. the real 589 passed / 93%); refreshed, and the table now notes
+  explicitly that it's an 11-of-33-module excerpt rather than implying completeness.
+- README.md's mem0#4884 success-story bullet credited `LanguageDegradationSignal` without
+  disclosing that eval isn't wired into `memtrust run --eval` (it's `Mem0DirectAdapter`-specific,
+  requiring `query(explain=True)`) -- added the same honest not-yet-wired-in caveat this project
+  uses for every other eval in the same situation.
+
+## [0.3.2] - 2026-07-20
+
+### Fixed
+
+- `memtrust --version` always printed `0.0.0+unknown`, even when properly installed,
+  because `src/memtrust/__init__.py` read `importlib.metadata.version("memtrust")`
+  instead of `version("memtrust-cli")`, the actual installed distribution name. Now
+  reads the correct key and matches `pip show memtrust-cli`.
+- The Python sdist ballooned to 142MB because `npm/platforms/*/bin/` (locally-staged
+  `uv` binaries used to build the npm platform packages, ~350MB across 6 platforms)
+  was untracked and ungitignored, so hatchling's default sdist inclusion swept it in.
+  Excluded via `.gitignore` and an explicit `[tool.hatch.build.targets.sdist]` exclude.
+- `Mem0DirectAdapter` now passes `is_reasoning_model=True` to work around a real
+  `mem0ai==2.0.12` bug: its default LLM model (`gpt-5-mini`) is not recognized by its
+  own reasoning-model detection (which checks for the different string
+  `gpt-5o-mini`), so every LLM-based extraction call sent `temperature=0.1` to a model
+  that only accepts the API default, 400ing on every call out of the box for anyone
+  with just `OPENAI_API_KEY` set.
+
+### Added
+
+- `temporal_kg_boundary` is now wired into `memtrust run --eval`, gated to the
+  `mempalace` backend (reports `not_applicable` against any other backend).
+- First live benchmark result: `mem0_direct` (self-hosted `mem0ai`, local Qdrant,
+  OpenAI embeddings/extraction) run against `contradiction`, `compression`, and
+  `extraction_quality`. See README.md's "Benchmarks" section for the numbers.
+
+### Changed
+
+- `pyproject.toml`: `Development Status :: 4 - Beta` (was Alpha), expanded PyPI
+  keywords, refreshed description.
+
 ## [0.3.0] - 2026-07-17
 
 ### Changed

{memtrust_cli-0.3.2 → memtrust_cli-0.3.3}/CONTRIBUTING.md

@@ -109,6 +109,49 @@ conversion (or a second loader function) is needed.
 4. Every claim in a PR description about a score or benchmark number must be reproducible from a
    command someone else can run. "I ran X and got Y" needs the X.
 
+## Release process
+
+There are three packages, not one, and CI only automates one of them. Missing any of the other two
+silently breaks something for someone -- this is written down because it has already happened.
+
+1. Bump the version in `pyproject.toml` (and `npm/memtrust-cli/package.json` in the same PR if a
+   matching npm publish is planned -- see step 4).
+2. Before pushing, run the full local check list: `ruff check .`, `ruff format --check .` (a
+   separate step from `ruff check` -- CI fails on formatting alone even when linting passes),
+   `mypy --strict src/memtrust`, `pytest --cov=memtrust --cov-report=term-missing
+   --cov-fail-under=80`, `pip-audit`.
+3. **PyPI has two packages kept in lockstep, and CI only publishes one of them.**
+   `.github/workflows/publish-pypi.yml` (triggered by a GitHub Release) publishes whatever
+   `pyproject.toml`'s `[project].name` currently says -- `memtrust-cli`, the canonical name. The
+   separate mirror package literally named `memtrust` (which `npm/memtrust-cli/bin/memtrust.js`
+   pins its `uv tool run --from` call to) has no CI automation at all and must be published
+   manually every release: temporarily set `name = "memtrust"` in `pyproject.toml`, `uv build && uv
+   publish`, then revert the name back to `memtrust-cli`. Skipping this step doesn't break
+   `memtrust-cli`'s own release, but it does mean the npm wrapper starts fetching a stale PyPI
+   version the next time someone bumps `npm/memtrust-cli/package.json` without also doing this.
+4. **npm publish is also manual, and its own version must exactly match the PyPI release it
+   pins to.** `npm/memtrust-cli/package.json`'s version is not independent -- `bin/memtrust.js`
+   reads it directly to build `uv tool run --from memtrust==<that version>`, so bumping the npm
+   package's version without a matching live PyPI `memtrust` release (step 3) breaks the wrapper
+   for every subsequent install. Publish PyPI first, confirm both `memtrust-cli` and `memtrust`
+   are live (`pip index versions memtrust-cli`, or the PyPI JSON API if the simple index is
+   lagging), then `npm publish npm/memtrust-cli --access public`.
+5. **Building the sdist from a working tree that has locally-downloaded platform binaries staged
+   for npm publish will silently balloon it.** If `npm/platforms/*/bin/` has real `uv` binaries in
+   it (staged for an npm publish, ~50-70MB each across 6 platforms), `pyproject.toml`'s
+   `[tool.hatch.build.targets.sdist]` exclude should already keep them out -- but if that exclude
+   is ever removed, `uv build`'s sdist output size is the tripwire: it should be a few MB, not
+   100MB+.
+6. The 6 `@memtrust-cli/<platform>` packages have their own independent version numbers, not tied
+   to `memtrust-cli`'s. `npm/memtrust-cli/package.json`'s `optionalDependencies` should use a caret
+   range (`^0.1.1`, not an exact `0.1.0`) so a platform-package patch release resolves
+   automatically instead of drifting stale again.
+7. Create the GitHub Release last, after both registries are confirmed live -- this is what
+   triggers `publish-pypi.yml`. If `memtrust-cli` was already published manually before the
+   Release exists (e.g. to unblock testing), that workflow run will fail with a harmless
+   "File already exists" 400 from PyPI -- expected, not a regression, since the file it's trying
+   to upload is already there.
+
 ## Code of conduct
 
 Be direct, be specific, assume good faith. Disagreement about a methodology choice is welcome and

{memtrust_cli-0.3.2 → memtrust_cli-0.3.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: memtrust-cli
-Version: 0.3.2
+Version: 0.3.3
 Summary: Independent, reproducible benchmark harness for agent-memory backends (MemPalace, Mem0, Zep/Graphiti, OpenViking): runs LongMemEval, LoCoMo, contradiction-detection, and a dozen other evals against all four and publishes the raw logs, not vendor-curated numbers.
 Project-URL: Homepage, https://github.com/RudrenduPaul/memtrust
 Project-URL: Issues, https://github.com/RudrenduPaul/memtrust/issues
@@ -45,12 +45,13 @@ Description-Content-Type: text/markdown
 
 # memtrust
 
-Standardized, reproducible benchmarks for agent-memory backends, run against the vendors, not
-published by them.
+Agent memory backends each publish their own benchmark numbers, on different tests, measured
+different ways. memtrust runs the same evals against all four and publishes the raw logs. Run
+against the vendors, not by them.
 
 [![CI](https://github.com/RudrenduPaul/memtrust/actions/workflows/ci.yml/badge.svg)](https://github.com/RudrenduPaul/memtrust/actions/workflows/ci.yml)
 [![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://github.com/RudrenduPaul/memtrust/blob/main/LICENSE)
-[![Version](https://img.shields.io/badge/version-0.2.0-blue.svg)](https://github.com/RudrenduPaul/memtrust/blob/main/pyproject.toml)
+[![PyPI Version](https://img.shields.io/pypi/v/memtrust-cli)](https://pypi.org/project/memtrust-cli/)
 [![PyPI](https://img.shields.io/badge/pypi-memtrust--cli-blue.svg)](https://pypi.org/project/memtrust-cli/)
 
 ```bash
@@ -77,22 +78,23 @@ memtrust run --backends mempalace,mem0,zep,openviking --eval all
 
 If you've compared agent-memory backends recently, you've probably noticed each one leads with a
 different accuracy number, on a different benchmark, measured a different way. MemPalace's own
-community already flagged the problem in public. Issue [#27](https://github.com/mempalace/mempalace/issues/27)
-on the MemPalace repository, opened April 7, 2026 and still open, documents a widely-cited 100%
-LongMemEval score that came from hand-tuning on the failed test questions themselves. The held-out
-score is 98.4%, not 100%. A separate 96.6% figure people cite everywhere turns out to be mostly
-ChromaDB's default embeddings doing the work, not MemPalace's own architecture. A "lossless"
-compression claim drops accuracy by 12.4 percentage points in practice. Two internal pull requests
-attempting to fix the reporting problem (#433 and #729) were both closed without merging on April
-12, 2026. As of this writing, the issue has 233 thumbs-up reactions and 39 comments.
+community already flagged the problem in public. Issue [#27](https://github.com/MemPalace/mempalace/issues/27)
+on the MemPalace repository, opened April 7, 2026 and still open, documents that a headline 100%
+LongMemEval figure, measured with Haiku reranking, wasn't reproducible from the repository's own
+benchmark scripts and was pulled from the README as unverifiable. A separate 96.6% figure people
+cite everywhere turns out to be mostly ChromaDB's default embeddings doing the work in raw mode,
+not MemPalace's own architecture. A "lossless" compression claim (the "AAAK" mode) drops the same
+LongMemEval score from 96.6% to 84.2% in practice, a 12.4 percentage point gap. Two internal pull
+requests attempting to fix the reporting problem, #433 and #729, were both closed without merging
+on April 12, 2026 -- #729 within seven minutes of being opened. As of this writing, the issue has
+233 thumbs-up reactions and 39 comments.
 
 None of that means MemPalace, or any other backend, doesn't work. It means nobody outside the
 vendor had run the same test, the same way, against every option, and published the raw logs.
 
 memtrust does that. It runs LongMemEval, LoCoMo, and a growing set of evals built specifically for
-this project -- 14 of them as of this writing, registered in the CLI's `--eval` flag, plus a
-fifteenth (temporal-KG boundary detection) that exists as a tested module but isn't wired into the
-CLI yet. The two that matter most for understanding what this project is actually for:
+this project -- 17 of them as of this writing, all registered in the CLI's `--eval` flag. The two
+that matter most for understanding what this project is actually for:
 contradiction detection, because neither LongMemEval nor LoCoMo tests the question that actually
 matters once a memory system sits underneath a production agent -- what happens when a new fact
 contradicts an old one? Does the backend flag the conflict? Silently overwrite the old fact with no
@@ -102,13 +104,21 @@ test claims like the "lossless" one above: it stores content, retrieves it, and
 reconstruction fidelity rather than semantic accuracy, per operating mode a backend exposes (see
 `MemoryBackendAdapter.supported_modes`) -- the mechanism that would let a contributor with live
 MemPalace credentials actually reproduce the 12.4-point compressed-mode accuracy drop
-mempalace/mempalace#27 documents, instead of just citing it. **Neither has been run against any
-live backend as of this writing** -- see `docs/methodology.md`. The other evals -- ranking quality,
+mempalace/mempalace#27 documents, instead of just citing it. **Neither has been run against a live
+MemPalace instance as of this writing** -- both have, however, been run against a live
+self-hosted `mem0ai` install; see "Benchmarks" below. The other evals -- ranking quality,
 crash recovery, extraction quality, embedding drift, scale/volume stress, lock contention, stats
 accuracy, orphan cleanup, result consistency, migration rollback, filter injection, resource-sync
 safety, and temporal-KG boundary detection -- each grew out of a specific real bug report against
 one of the four tracked backends; see "Success stories" below for the full list.
 
+**Where this stands right now, in one place:** [live benchmark results](#benchmarks) for one
+backend (`mem0_direct`, self-hosted `mem0ai`), including a real bug this project's own attempt to
+get those numbers surfaced in mem0's default configuration; and [197 real GitHub issues and PRs](#success-stories)
+filed against MemPalace, Mem0, Zep/Graphiti, and OpenViking independently root-caused against this
+codebase -- 55 (28%) PASS, 16 (8%) PARTIAL, 42 (21%) a genuine capability gap, 84 (43%) not
+applicable, every verdict re-verified by a reviewer independent of whoever built the fix.
+
 ## What it does
 
 Every command below was actually run against this repo, with zero vendor API keys configured, to
@@ -116,11 +126,11 @@ produce the output shown. Nothing here is simulated.
 
 ```
 $ memtrust run --backends mempalace,mem0,zep,openviking --eval all
-memtrust 0.1.0 -- run_id=mt_2026-07-17T155402Z
+memtrust 0.3.3 -- run_id=mt_2026-07-21T004232Z
 Backends: mempalace, mem0, zep, openviking   Evals: longmemeval, locomo, contradiction,
 resource_sync_safety, compression, ranking_quality, scale_stress, embedding_drift, crash_recovery,
 extraction_quality, migration_rollback, filter_injection, lock_contention, stats_accuracy,
-orphan_cleanup, result_consistency
+orphan_cleanup, result_consistency, temporal_kg_boundary
 
 mempalace: SKIPPED (not configured) -- mempalace is not configured: environment variable
 MEMPALACE_STORAGE_PATH is not set. Skipping this backend. See docs/methodology.md for setup
@@ -135,17 +145,19 @@ instructions.
 
 Cost: $0.00 (no LLM-judged evals ran -- structural evals only, or judge not configured)
 
-Full report: memtrust-report-2026-07-17.json
+Full report: memtrust-report-2026-07-20.json
 ```
 
 That's the real, reproducible behavior of a fresh clone with no credentials: every backend reports
 SKIPPED, the command exits cleanly, and a valid JSON report is still written. `memtrust --version`
-currently prints `0.1.0` even though the package installs as `0.2.0` on PyPI -- a stale hardcoded
-string in `src/memtrust/__init__.py` that hasn't caught up with `pyproject.toml`, shown here exactly
-as it actually runs rather than smoothed over. Set the relevant environment variable for any backend
-you want to actually test (`MEM0_API_KEY`, `ZEP_API_KEY`, `OPENVIKING_API_KEY`,
-`MEMPALACE_STORAGE_PATH`) and that backend runs for real against its live API instead of being
-skipped.
+now correctly prints `0.3.3`, matching `pip show memtrust-cli`. Earlier releases printed
+`0.0.0+unknown` even when properly installed, because `src/memtrust/__init__.py` read
+`importlib.metadata.version("memtrust")` while the installed distribution is actually named
+`memtrust-cli` -- kept in the FAQ below for the record rather than deleted, since silently erasing
+a bug the moment it's fixed is exactly the kind of curation this project exists to push back on in
+other people's benchmarks. Set the relevant environment variable for any backend you want to
+actually test (`MEM0_API_KEY`, `ZEP_API_KEY`, `OPENVIKING_API_KEY`, `MEMPALACE_STORAGE_PATH`) and
+that backend runs for real against its live API instead of being skipped.
 
 The eval logic itself is proven offline, against the bundled synthetic fixtures and, for several
 adapters, the real installed vendor packages with only the network boundary mocked, by the test
@@ -153,29 +165,33 @@ suite:
 
 ```
 $ pytest --cov=memtrust --cov-report=term-missing
-...
+... (33 module rows total; the 11 most relevant to this README are shown below)
 Name                                                          Stmts   Miss  Cover
 -------------------------------------------------------------------------------------
-src/memtrust/adapters/base.py                                   290      1    99%
-src/memtrust/adapters/mempalace_adapter.py                      265     15    94%
+src/memtrust/adapters/base.py                                   305     16    95%
+src/memtrust/adapters/mempalace_adapter.py                      254     15    94%
 src/memtrust/adapters/mem0_adapter.py                            140     12    91%
-src/memtrust/adapters/mem0_direct_adapter.py                     281     34    88%
+src/memtrust/adapters/mem0_direct_adapter.py                     277     34    88%
 src/memtrust/adapters/openviking_adapter.py                      178     18    90%
 src/memtrust/adapters/zep_graphiti_adapter.py                     63      3    95%
-src/memtrust/adapters/zep_graphiti_selfhosted_adapter.py         165     24    85%
+src/memtrust/adapters/zep_graphiti_selfhosted_adapter.py         161     24    85%
 src/memtrust/evals/contradiction.py                              127      2    98%
 src/memtrust/evals/compression.py                                 86      1    99%
-src/memtrust/evals/temporal_kg_boundary.py                        90      4    96%
+src/memtrust/evals/temporal_kg_boundary.py                        90      3    97%
 src/memtrust/receipt.py                                          118     10    92%
 -------------------------------------------------------------------------------------
-TOTAL                                                            4130    256    94%
+TOTAL                                                            4147    276    93%
 
-580 passed, 8 skipped in 3.31s
+589 passed, 8 skipped in 3.57s
 ```
 
-580 passing tests across 33 source modules, 94% overall statement coverage, 98% on the
-contradiction-detection eval, 99% on compression/round-trip fidelity, 96% on the new temporal-KG
-boundary eval, 94-99% across the adapter layer. The 8 skips are live-`mempalace`-package tests that
+This is an excerpt, not the full table -- the weakest-covered module in the repo,
+`evals/mempalace_metadata_scale.py` (70%), isn't one of the 11 shown above; run the command
+yourself for the complete per-module breakdown.
+
+589 passing tests across 33 source modules, 93% overall statement coverage, 98% on the
+contradiction-detection eval, 99% on compression/round-trip fidelity, 97% on the temporal-KG
+boundary eval, 85-95% across the adapter layer. The 8 skips are live-`mempalace`-package tests that
 only run with the optional `mempalace-direct` extra installed (`pip install -e
 '.[dev,mempalace-direct]'`). Every test mocks its HTTP or wire
 layer, or uses an in-memory fake backend -- none of them touch a real network, though a meaningful
@@ -210,16 +226,11 @@ Commands:
 
 | Command | Flags | What it does |
 |---|---|---|
-| `memtrust run` | `--backends TEXT` comma-separated list or `all` (default `all`) · `--eval TEXT` comma-separated from `longmemeval,locomo,contradiction,resource_sync_safety,compression,ranking_quality,scale_stress,embedding_drift,crash_recovery,extraction_quality,migration_rollback,filter_injection,lock_contention,stats_accuracy,orphan_cleanup,result_consistency`, or `all` (default `all`) · `--output FILE` (defaults to `./memtrust-report-<date>.json`) · `--locomo-dataset-path FILE` points the LoCoMo eval at a real, downloaded `locomo10.json` instead of the bundled synthetic fixture (memtrust does not bundle or auto-fetch the real dataset) · `--locomo-exclude-question-ids-file FILE` excludes known-bad-ground-truth LoCoMo question IDs from scoring · `--scale-stress-n-records INTEGER` (default `500`) sets how many synthetic records the scale-stress eval stores and re-queries · `--sign FILE` writes a signed `<output>.receipt.json` alongside the report, proving it was produced by the holder of the given Ed25519 private key | Runs the eval suite against the requested backends. A backend without its credential env var set prints `SKIPPED` and the run continues -- this command never crashes on missing credentials. |
+| `memtrust run` | `--backends TEXT` comma-separated list or `all` (default `all`) · `--eval TEXT` comma-separated from `longmemeval,locomo,contradiction,resource_sync_safety,compression,ranking_quality,scale_stress,embedding_drift,crash_recovery,extraction_quality,migration_rollback,filter_injection,lock_contention,stats_accuracy,orphan_cleanup,result_consistency,temporal_kg_boundary`, or `all` (default `all`) · `--output FILE` (defaults to `./memtrust-report-<date>.json`) · `--locomo-dataset-path FILE` points the LoCoMo eval at a real, downloaded `locomo10.json` instead of the bundled synthetic fixture (memtrust does not bundle or auto-fetch the real dataset) · `--locomo-exclude-question-ids-file FILE` excludes known-bad-ground-truth LoCoMo question IDs from scoring · `--scale-stress-n-records INTEGER` (default `500`) sets how many synthetic records the scale-stress eval stores and re-queries · `--sign FILE` writes a signed `<output>.receipt.json` alongside the report, proving it was produced by the holder of the given Ed25519 private key | Runs the eval suite against the requested backends. A backend without its credential env var set prints `SKIPPED` and the run continues -- this command never crashes on missing credentials. `temporal_kg_boundary` only applies to the `mempalace` backend (the only adapter that wires `kg_add`/`kg_invalidate`/`kg_query`); requesting it against any other backend reports `not_applicable`, not an error. |
 | `memtrust report REPORT_PATH` | positional path to a prior JSON report | Reads a report written by `memtrust run` and prints a formatted summary. |
 | `memtrust keygen` | -- | Generates a new Ed25519 keypair for signing reports with `run --sign`. |
 | `memtrust verify RECEIPT_PATH` | -- | Verifies a signed receipt produced by `memtrust run --sign`; a tampered or mismatched receipt fails verification. |
-| `memtrust --version` | -- | Prints the installed version. Currently prints `0.1.0`, one release behind the `0.2.0` this package actually installs as -- see the note above. |
-
-The temporal-KG boundary eval (`src/memtrust/evals/temporal_kg_boundary.py`,
-`run_temporal_kg_boundary_eval()`) is not yet wired into `--eval`'s known-eval list -- it exists as
-a tested module and a MemPalace-specific capability, callable directly, not yet a `memtrust run
---eval temporal_kg_boundary` option. See its own section below.
+| `memtrust --version` | -- | Prints the installed version (currently `0.3.3`). |
 
 Every line above came straight from running `memtrust --help`, `memtrust run --help`, and
 `memtrust report --help` against this repo. Nothing here is invented.
@@ -249,10 +260,10 @@ section below until a live run actually produces them:
 
 | Backend | GitHub stars | Self-reported description |
 |---|---|---|
-| [MemPalace](https://github.com/MemPalace/mempalace) | 57,268 | "The best-benchmarked open-source AI memory system. And it's free." |
-| [Mem0](https://github.com/mem0ai/mem0) | 60,688 | "Universal memory layer for AI Agents" |
-| [Zep / Graphiti](https://github.com/getzep/graphiti) | 28,648 | "Build Real-Time Knowledge Graphs for AI Agents" |
-| [OpenViking](https://github.com/volcengine/OpenViking) | 26,639 | "Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills." |
+| [MemPalace](https://github.com/MemPalace/mempalace) | 57,512 | "The best-benchmarked open-source AI memory system. And it's free." |
+| [Mem0](https://github.com/mem0ai/mem0) | 61,318 | "Universal memory layer for AI Agents" |
+| [Zep / Graphiti](https://github.com/getzep/graphiti) | 28,978 | "Build Real-Time Knowledge Graphs for AI Agents" |
+| [OpenViking](https://github.com/volcengine/OpenViking) | 27,016 | "Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills." |
 
 None of these numbers say anything about which backend handles a contradicted fact correctly --
 that's the whole reason the harness exists. Star count measures adoption, not correctness.
@@ -287,8 +298,9 @@ similarity ratio (`fidelity_ratio()`, via `difflib.SequenceMatcher` -- not an LL
 "lossless" claim is a literal-reconstruction claim, not a semantic one). This is what would let a
 contributor with live MemPalace credentials point `memtrust run --eval compression` at it and
 reproduce a "raw vs AAAK" fidelity gap directly. **As of this writing this eval has not been run
-against any live backend** -- see `docs/methodology.md` for the same live-credentials caveat that
-applies to every other eval in this table.
+against a live MemPalace instance** -- it has been run against a live self-hosted `mem0ai` install
+(mean fidelity 31.6%, see "Benchmarks" below); see `docs/methodology.md` for the same
+live-credentials caveat that applies to every other eval and backend not yet measured live.
 
 ## The eval built from MemPalace's own bug: temporal-KG boundary detection
 
@@ -311,18 +323,76 @@ shipped in a released `mempalace` version as of this adapter's live-verified 3.5
 lands under the package's `[Unreleased]` changelog section. `tests/test_temporal_kg_boundary.py`
 proves the *classification logic* is correct against two hand-written fake implementations that
 reproduce the confirmed pre-#1914 (closed-interval) and post-#1914 (half-open-interval) SQL
-comparison exactly. **This has not been run against a live MemPalace instance.** It also isn't
-wired into `memtrust run --eval` yet -- call `run_temporal_kg_boundary_eval()` directly, or see the
-test suite, until that CLI surface exists.
+comparison exactly. **This has not been run against a live MemPalace instance.** It is wired into
+`memtrust run --eval temporal_kg_boundary` (see "Commands" above); against any backend other than
+`mempalace`, it reports `not_applicable` rather than an error.
 
 ## Benchmarks
 
-**Not yet measured against live backends.** This repo was built without API credentials for
-MemPalace, Mem0, Zep, or OpenViking. Publishing a number here without actually running the harness
-against a live backend would be exactly the kind of unverifiable claim this project exists to
-push back on, so there isn't one.
+**Live results: mem0_direct (self-hosted), July 2026.** MemPalace, Zep, and OpenViking are still
+not yet measured against a live backend -- see "Backend coverage" below for the confidence level
+on each adapter. Mem0 has one real result, produced against the actual `mem0ai` OSS library
+running self-hosted -- in-process, via `Mem0DirectAdapter`, backed by a local Qdrant instance and
+the OpenAI API for embeddings and extraction -- not against Mem0's hosted Platform API. Keep that
+distinction in mind before reading this as a claim about the hosted product.
+
+```
+$ export MEM0_DIRECT_EMBEDDER_PROVIDER=openai
+$ export MEM0_DIRECT_VECTOR_STORE_PROVIDER=qdrant
+$ export MEM0_DIRECT_VECTOR_STORE_URL=http://localhost:6333
+$ memtrust run --backends mem0_direct --eval contradiction,compression,extraction_quality
+memtrust 0.3.2 -- run_id=mt_2026-07-20T210918Z
+Backends: mem0_direct   Evals: contradiction, compression, extraction_quality
+
+mem0_direct: configured, running evals...
+  Running Contradiction-Detection against mem0_direct...
+    flagged: 0.0%  silent-overwrite: 100.0%  served-stale: 0.0%  empty-or-lost: 0.0%
+  Running Compression/Round-Trip-Fidelity against mem0_direct...
+    fidelity by mode -- default: 31.6%
+  Running Extraction-Quality against mem0_direct...
+    junk-retained: 0.0%  valid-lost: 100.0%
+    feedback-loop-duplicate: 0.0%
+
+Cost: $0.00 (no LLM-judged evals ran -- structural evals only, or judge not configured)
+```
 
-To produce real numbers:
+The full raw report is committed at `leaderboard/mem0_direct-2026-07-20.json`, and `leaderboard/data.json`
+carries the contradiction numbers into the static leaderboard site (`mempalace`/`mem0`/`zep`/`openviking`
+still show `not_measured` there; `mem0_direct` is the one real row).
+
+What that means case by case, not just the percentage:
+
+- **Contradiction detection, 7/7 cases: every contradicting fact silently overwrote the old one.**
+  0% were flagged as a conflict, 0% served stale, 0% empty-or-lost. Tell it your meeting moved from
+  2pm to 3pm and it stores the new fact with no signal that anything changed -- this is exactly the
+  question LongMemEval and LoCoMo don't test, and exactly what "Why this exists" above is about.
+- **Compression/round-trip fidelity, 5 cases: 31.6% mean literal character-level reconstruction.**
+  This is expected, not a defect -- mem0's design goal is semantic fact extraction, not verbatim
+  storage, so a literal-reconstruction score was never going to be high. It quantifies what "not
+  built for lossless storage" concretely means for this backend: ask it what you said and you get
+  the gist back, not your words.
+- **Extraction quality, 15 cases (12 deliberately junk, 3 deliberately valid): 0% junk retained,
+  100% of the valid cases lost.** All 12 junk inputs (boot-file restating, cron heartbeat noise,
+  system dumps, hallucinated-profile bait) were correctly rejected. All 3 valid-content cases were
+  also dropped -- stored but never came back on retrieval. The valid-side sample is small (n=3);
+  treat this as a signal worth digging into further, not a settled number.
+
+**A real bug this run surfaced in mem0ai itself, not in memtrust.** Getting any of the numbers
+above required a fix first: a fresh `mem0ai==2.0.12` install with nothing but `OPENAI_API_KEY` set
+fails every single LLM-based extraction call, out of the box, for anyone. mem0's own default model
+(`mem0/llms/openai.py`: `self.config.model = "gpt-5-mini"`) is a reasoning-tier model that only
+accepts the API's default temperature, but mem0's own reasoning-model detection
+(`mem0/llms/base.py`'s `reasoning_models` set) checks for the string `"gpt-5o-mini"`, not
+`"gpt-5-mini"` -- two different strings, so the check never fires, and mem0 sends `temperature=0.1`
+on every call regardless. The result is a `400 Unsupported value: 'temperature' does not support
+0.1 with this model` error on every extraction call, silently caught by mem0 and reported by
+memtrust as `N/A (no scoreable cases)` rather than a real result. `Mem0DirectAdapter` now works
+around it by passing `is_reasoning_model=True` explicitly -- mem0's own documented override for
+exactly this situation -- see `src/memtrust/adapters/mem0_direct_adapter.py`'s "Default LLM
+extraction is broken out of the box" section for the full citation. No upstream mem0ai issue filed
+for this as of this writing.
+
+To reproduce this or measure the remaining three backends:
 
 ```bash
 export MEM0_API_KEY=...          # and/or
@@ -403,13 +473,8 @@ LongMemEval/LoCoMo datasets). Nothing leaves your machine unless you choose to p
 
 ### npx (agent-native)
 
-> **PyPI is live; the npm wrapper is publish-ready but not yet published.** `pip install memtrust-cli`
-> works today -- version 0.2.0 is on PyPI. The `memtrust-cli` npm package in `npm/memtrust-cli/` in
-> this repo is built, tested, and passes both `npm pack --dry-run` and `npm publish --dry-run`
-> cleanly (re-verified this session), but has not actually been pushed to the npm registry yet --
-> `npm install memtrust-cli` still 404s as of this writing. The command below is the real,
-> working distribution path once that publish happens; nothing about it is speculative except the
-> "it's live on npm" part.
+Both PyPI and npm are live. `pip install memtrust-cli` works today, and so does the `npx` command
+below -- `npm install memtrust-cli` no longer 404s.
 
 For CI and agent runners that have Node.js available but not necessarily a Python toolchain:
 
@@ -429,9 +494,11 @@ first use. What it removes is no Python toolchain to provision by hand -- `npx m
 handles the interpreter and package fetch for you via a bundled, verified copy of Astral's
 [`uv`](https://github.com/astral-sh/uv). Each platform package bundles a genuine, SHA-256-verified
 copy of `uv`'s own GitHub release binary (fetched at npm package-publish time, never at end-user
-install time), and its `bin` shim runs `uv tool run --from memtrust memtrust <args>`, which
-transparently bootstraps a Python interpreter and installs `memtrust` from PyPI, caching it after
-the first run.
+install time), and its `bin` shim runs `uv tool run --from memtrust==<pinned version> memtrust
+<args>`, which transparently bootstraps a Python interpreter and installs that exact pinned
+`memtrust` release from PyPI, caching it after the first run. The npm package is pinned to its own
+version -- bump `npm/memtrust-cli/package.json`'s version when a new PyPI release ships, and every
+subsequent install resolves to that exact release, not whatever happens to be newest at run time.
 
 ## What a hosted trust layer would add
 
@@ -488,7 +555,7 @@ commit than remember to run them by hand.
 ## FAQ
 
 **What is memtrust, and what actually makes it different from reading a vendor's own benchmark
-page?** It's a CLI harness that runs the same evals (LongMemEval, LoCoMo, and 14 others registered
+page?** It's a CLI harness that runs the same evals (LongMemEval, LoCoMo, and 15 others registered
 in `--eval`, including a contradiction-detection eval none of the four tracked backends publish a
 number for) against MemPalace, Mem0, Zep/Graphiti, and OpenViking, and prints the raw output rather
 than a curated summary. The differentiator isn't a proprietary scoring model; it's that nobody
@@ -524,20 +591,37 @@ those four backends make about themselves. If you need broad RAG or prompt-evalu
 RAGAS or a similar framework is the right tool; if you need to check whether a memory backend
 silently drops or overwrites a contradicted fact, memtrust is the one built for that question.
 
-**Why does `memtrust --version` print a version that doesn't match what pip says I installed?**
-This is a real, currently open limitation, not a hypothetical one -- installing `memtrust-cli` from
-PyPI into a clean virtualenv and running `pip show memtrust-cli` reports `0.3.0`, but
-`memtrust --version` prints `0.0.0+unknown`. The CLI's version string isn't reading the installed
-package metadata correctly. Until it's fixed, use `pip show memtrust-cli` (or
-`python3 -c "import importlib.metadata as m; print(m.version('memtrust-cli'))"`) to check what's
-actually installed, not the CLI's own `--version` output.
+**Why did `memtrust --version` used to print a version that didn't match what pip said I
+installed?** This was a real, shipped bug through 0.3.1, not a hypothetical one: installing
+`memtrust-cli` from PyPI into a clean virtualenv and running `pip show memtrust-cli` reported the
+correct version, but `memtrust --version` printed `0.0.0+unknown` regardless, because
+`src/memtrust/__init__.py` read `importlib.metadata.version("memtrust")` -- the wrong distribution
+name -- instead of `version("memtrust-cli")`, the name the package is actually installed under.
+0.3.2's fix was itself incomplete: it hardcoded the lookup to `"memtrust-cli"`, which broke the
+separate `memtrust` mirror package (see "Can I `pip install memtrust` instead of
+`memtrust-cli`?" below) the same way in reverse -- a `pip install memtrust` environment has no
+`memtrust-cli` entry in its own installed-package metadata at all, so the lookup always missed and
+fell through to the same `0.0.0+unknown` fallback. Fixed for real in 0.3.3: the lookup now tries
+`memtrust-cli` first, falls back to `memtrust`, and only reports `0.0.0+unknown` if neither
+distribution name is installed. `memtrust --version` now matches `pip show` under either package
+name.
+
+**Can I `pip install memtrust` instead of `memtrust-cli`?** Yes -- `memtrust` is a real, separate
+PyPI package, kept at the same version as `memtrust-cli` on every release, publishing the identical
+source. It exists because the npm wrapper's `bin/memtrust.js` pins `uv tool run --from
+memtrust==<version>` (not `memtrust-cli`), so a working `npx memtrust-cli` depends on the
+`memtrust` name staying live and in sync. `memtrust-cli` is the name to lead with in new
+documentation, since the `-cli` suffix makes it unambiguous as a CLI tool at a glance; `memtrust`
+is a legitimate, supported install path, not a typo or a squatted name.
 
 **Has memtrust actually been run against a live memory backend, or is this all synthetic?** Both,
 and the README doesn't blur the line. The eval logic itself is proven against bundled synthetic
 fixtures and, for several adapters, real installed vendor packages with only the network boundary
-mocked (see the pytest coverage table above). But no eval in this repo has been run against a live
-MemPalace, Mem0, Zep, or OpenViking backend with real credentials as of this writing -- the
-"Benchmarks" section above states that directly rather than publishing a placeholder number. The
+mocked (see the pytest coverage table above). One backend has a real live result: `mem0_direct`,
+the self-hosted `mem0ai` OSS library (not Mem0's hosted Platform API), run against contradiction,
+compression, and extraction-quality -- see "Benchmarks" above for the exact numbers and a real bug
+that run surfaced in `mem0ai` itself. MemPalace, the hosted Mem0 Platform API, Zep, and OpenViking
+have not yet been run against a live backend with real credentials as of this writing. The
 "Backend coverage" table gives a per-adapter confidence level (high/medium/low) for exactly this
 reason; run it yourself against your own credentials to get a live-verified number.
 
@@ -547,7 +631,7 @@ and requires you to preserve the license and copyright notice and to note any ch
 the code. It does not require you to open-source your own product just because you depend on
 memtrust.
 
-**How do I get real accuracy numbers instead of the "not yet measured" placeholder?** Set the
+**How do I get real accuracy numbers for the backends still marked "not yet measured"?** Set the
 credential environment variable for whichever backend you want to test
 (`MEMPALACE_STORAGE_PATH`, `MEM0_API_KEY`, `ZEP_API_KEY`, `OPENVIKING_API_KEY`, plus
 `MEMTRUST_JUDGE_API_KEY` for LLM-judged evals like LongMemEval and LoCoMo), then run
@@ -607,10 +691,6 @@ shipped in the open rather than quietly. See "Backend coverage" above for the fu
   (@JosefAschauer): an `authored_at` chronology tie-break fix for `_hybrid_rank`. memtrust's ranking
   classifier now credits a top-level `authored_at` field, not just one nested under `metadata`, as
   a genuine ranking-driving signal.
-- [#524](https://github.com/MemPalace/mempalace/issues/524) (@gaby): a buried-comment report that
-  "no API key required" doesn't mean "no network required" -- ChromaDB's default embedder still
-  needs to download a model on first use, which silently breaks airgapped setups. The adapter's
-  module docstring now says so explicitly.
 
 **mem0**
 - [#5973](https://github.com/mem0ai/mem0/pull/5973) (@abhay-codes07, superseded by
@@ -635,7 +715,13 @@ shipped in the open rather than quietly. See "Backend coverage" above for the fu
   update) against this taxonomy.
 - [#4884](https://github.com/mem0ai/mem0/issues/4884) (@wangjiawei-vegetable): a hardcoded
   English-only tokenizer silently degrading non-Latin-script retrieval. A new
-  `LanguageDegradationSignal` and non-Latin-script fixtures now catch this shape.
+  `LanguageDegradationSignal` and non-Latin-script fixtures now catch this shape --
+  `Mem0DirectAdapter`-specific (it reads `query(explain=True)`'s real per-result diagnostic
+  fields, a capability only that adapter exposes) and, like `embedder_cost.py`'s cost-attribution
+  eval and `episode_temporal_leak.py`'s Graphiti-specific eval, not yet wired into `memtrust run
+  --eval`'s general list; call `run_language_degradation_eval()` directly, or see the
+  `test_query_language_degradation_*` tests in `tests/test_mem0_direct_adapter.py`, until that CLI
+  surface exists.
 
 **Zep / Graphiti**
 - [#1489](https://github.com/getzep/graphiti/issues/1489) (@brentkearney): a bi-temporal