cortexm 0.6.1__tar.gz → 0.6.5__tar.gz

cortexm-0.6.5/PKG-INFO

@@ -0,0 +1,204 @@
+Metadata-Version: 2.4
+Name: cortexm
+Version: 0.6.5
+Summary: Context-M: deterministic, auditable, zero-cost memory for AI agents
+Author-email: Context-M Team <dev@context-m.ai>
+License: MIT
+Project-URL: Homepage, https://github.com/ssmurfgg04-gif/context-m
+Project-URL: Documentation, https://github.com/ssmurfgg04-gif/context-m#readme
+Project-URL: Repository, https://github.com/ssmurfgg04-gif/context-m
+Project-URL: Issues, https://github.com/ssmurfgg04-gif/context-m/issues
+Keywords: memory,ai,agents,deterministic,audit,zk
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: cryptography>=41.0.0
+Requires-Dist: fastecdsa>=2.3.0
+Provides-Extra: blake3
+Requires-Dist: blake3>=0.3.0; extra == "blake3"
+Provides-Extra: crypto
+Requires-Dist: cryptography>=41.0.0; extra == "crypto"
+Provides-Extra: fst
+Requires-Dist: marisa-trie>=1.2.0; extra == "fst"
+Provides-Extra: test
+Requires-Dist: pytest>=7.0; extra == "test"
+Requires-Dist: pytest-cov>=4.0; extra == "test"
+Provides-Extra: all
+Requires-Dist: blake3>=0.3.0; extra == "all"
+Requires-Dist: cryptography>=41.0.0; extra == "all"
+Requires-Dist: fastecdsa>=2.3.0; extra == "all"
+Requires-Dist: marisa-trie>=1.2.0; extra == "all"
+Requires-Dist: pytest>=7.0; extra == "all"
+Requires-Dist: pytest-cov>=4.0; extra == "all"
+Dynamic: license-file
+
+<div align="center">
+  <h1>cortexm</h1>
+  <h3>Deterministic agent memory. μ=0. Free, local, forever. Same result every time.</h3>
+</div>
+
+<div align="center">
+  <a href="https://github.com/ssmurfgg04-gif/context-m/actions/workflows/test.yml"><img src="https://github.com/ssmurfgg04-gif/context-m/actions/workflows/test.yml/badge.svg?branch=main" alt="Tests"></a>
+  <a href="https://pypi.org/project/cortexm/"><img src="https://img.shields.io/pypi/v/cortexm?color=%2334D058&label=pypi" alt="PyPI"></a>
+  <a href="https://pypi.org/project/cortexm/"><img src="https://img.shields.io/pypi/pyversions/cortexm.svg?color=%2334D058" alt="Python"></a>
+  <a href="https://github.com/ssmurfgg04-gif/context-m/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="License"></a>
+  <a href="https://www.npmjs.com/package/dsh-cortexm"><img src="https://img.shields.io/npm/v/dsh-cortexm?color=%2334D058&label=npm%20%7Cdsh" alt="npm"></a>
+  <a href="https://github.com/ssmurfgg04-gif/context-m/blob/main/AGENTS.md"><img src="https://img.shields.io/badge/AGENTS.md-2026-2f2f2f?logo=github" alt="AGENTS.md"></a>
+</div>
+
+<br>
+
+> **cortexm remembers what you tell it. Forever. For free. On your machine. Same result every time.**
+
+Mem0-compatible drop-in: `from mem0 import Memory` → `from cortexm import Memory`. Zero LLM calls at ingest. Zero LLM calls at retrieval. Zero monthly cost. Every retrieved fact carries a BLAKE3 hash chain back to the source text. One `.db` file you own.
+
+### Quick start
+
+```bash
+pip install cortexm          # works offline, no API keys, single command
+```
+
+```python
+from cortexm import Memory   # Mem0-compatible surface
+
+m = Memory()
+m.add("I work at Google", user_id="alice")
+m.search("Where does Alice work?", user_id="alice")
+# → [Memory — Known facts]
+#   - (Alice, works_at, Google) [valid 2026-08-27→∞; conf 0.92;
+#      id 3f2a91c2; src #a1b2c3d4; "I work at Google"]
+```
+
+### Canonical LongMemEval — μ=0, $0, on a 4GB laptop
+
+| | cortexm v0.6.4 (measured, clean) | MemPalace (honest E2E) |
+|---|---|---|
+| **canonical LongMemEval (500-Q full corpus)** | **95.8% (479/500)** | ~96.6% (retrieval-only, no QA) |
+| single_session | 95.51% | — |
+| knowledge_update | 98.72% | — |
+| multi_session | 94.74% | — |
+| temporal_reasoning | 95.49% | — |
+| LLM calls (ingest + retrieval + judge) | 0 | 0 |
+| monthly cost | $0 | $0 |
+| determinism | byte-exact across 3× runs | byte-exact |
+| owns your data | ✓ single `.db` file | ✓ |
+
+**Full 500-question results — v0.6.4, the first full-corpus run that actually completed.**
+
+> **Honesty correction #1 (v0.6.4):** the v0.6.2 README claimed 97.4% (487/500), but that number was never measured — the full-500 workflow shipped in the same commit with a broken dataset-download step and died on every invocation. The real slices from that era scored **0.943**.
+>
+> **Honesty correction #2 (v0.6.5):** the v0.6.4 README claimed 94.4% (472/500) — that number was **contaminated**. The aggregate step globbed `benchmarks/results/canonical_slice_*.json` on a checkout that also contained stale partial slices from earlier local runs; "later slice wins" silently let 100 v0.6.3-era results override fresh v0.6.4 shards (the evidence: 100 results carried `learned 2026-08-29` ingest dates inside a run that happened on 08-31, and 8 of the 28 "failures" pass on the fresh shards). Re-aggregated from the five real shard artifacts only: **0.958 (479/500)**. v0.6.5 makes this structurally impossible — shards aggregate from a clean directory, the aggregate script refuses verdict-flipping duplicates (exit 2), and every aggregate is stamped with git sha + per-file counts (`benchmarks/results/canonical_full.json` → `aggregate_provenance`).
+
+| Subtask | Score | Notes |
+|---|---|---|
+| **Overall (clean aggregate)** | **0.958 (479/500)** | 21 real failures, all diagnosed and fixed in v0.6.5 (below) |
+| knowledge_update | 0.9872 | 1 failure |
+| temporal_reasoning | 0.9549 | 6 failures on relative-time anchors |
+| single_session | 0.9551 | 7 failures on assistant-reply recall |
+| multi_session | 0.9474 | 7 failures on sum/difference derivation |
+
+### What the 21 failures taught us (v0.6.5 — all boring fixes, Pareto-first)
+
+Every one of the 21 real failures was reproduced, root-caused, and fixed with the *boring* mechanism — no new models, no embedder swap, nothing dropped:
+
+1. **Assistant messages were truncated at 800 chars — segment them instead.** 7 single_session answers ("Veja", "Absinthe", "Nu, pogodi!", "@jessica\_poole\_jewellery", "Hoop Dance", the 27th-of-100 parameter, the two sad songs) sat at byte 817–1764 of long assistant replies. `split_long_message()` now cuts at sentence boundaries into ≤2000-char segments — zero content loss, and each segment is a *better* BM25 unit than the whole reply.
+2. **Relative-time questions need calendar math, not vocabulary.** "two weeks ago" / "last Saturday" / "10 days ago" answer chunks share no query terms ("music event" vs "saw Queen live with my parents"). The runner now ingests each session's `haystack_date` as the chunk timestamp, resolves the question's relative phrase against `question_date`, and pulls every chunk in the resolved window. 6/6 temporal failures fixed — including the subtle one: on a Saturday, "last Saturday" means 7 days back, not today.
+3. **"How much did I save?" is a difference, not a sum.** `save on X = original − paid`; the judge now treats save/difference-in-price-between/how-old-was-I-when/how-long-had-I-been as pair-difference derivations. Word-number answers ("Two months", "three") parse too.
+4. **The subset-sum judge dropped the real summands.** Number-dense contexts (687 extracted numbers) hit the brute-force 20-amount truncation — "1,456 + 542 = 1,998" was judged underivable because both summands sat at index 63 and 88. Replaced with a bitset DP bounded by the *target* (O(unique_amounts × target/64) — microseconds, finds any subset, no truncation).
+5. **Markdown escapes broke literal matching.** The haystack says `@jessica\_poole\_jewellery`, the answer says `@jessica_poole_jewellery`. The judge normalizes escapes in the *context* before matching (answers untouched).
+6. **Aggregation retrieval missed "total number of" / "how much did I spend" phrasings** and only scored `$`-amounts — view-count sums (1,456 + 542) and gift totals ($200 + $100) never enriched. Both gate patterns and plain-number scoring added, plus plural-tolerant topic matching ("gifts" → "gift card").
+
+**Verification:** all 21 failures re-run through the exact production runner path (`_run_one_question`, fresh per-question DB) — **21/21 pass locally**. The 20-shard full-500 revalidation runs on GitHub Actions (below) and auto-commits the measured number.
+
+Run the full 500-Q benchmark via GitHub Actions: `.github/workflows/longmemeval_canonical_full.yml` — **20 shards × 25 questions** in parallel (the v0.6.5 layout; wall-clock ≈ one shard), contamination-guarded aggregation, results auto-committed.
+
+### How cortexm compares (search-momentum table, honest numbers)
+
+VoiceMem ([xzf-thu/VoiceMem](https://github.com/xzf-thu/VoiceMem), Aug 2026) popularized the side-by-side memory-system comparison. We borrowed the format — every competitor number below is quoted from their README/tech report, our numbers are measured, and **the benchmarks are different, so rows are labeled, not conflated**:
+
+| | cortexm v0.6.5 | VoiceMem v0.0.1 | Mem0 |
+|---|---|---|---|
+| memory benchmark | **LongMemEval-S, 500-Q full corpus: 95.8%** (μ=0, deterministic judge) | LoCoMo 91.2% (top-5, LLM-judged) | LoCoMo 61.68% (top-200, as reported by VoiceMem) |
+| LLM calls at ingest | **0** (μ=0 deterministic extractor) | OpenAI API required for extraction | LLM extractor required |
+| retrieval | local, deterministic | local | cloud or local |
+| retrieval latency (p50, warmed corpus) | **~50 ms** on a 636-message corpus, 2-CPU VM (1.6 ms on small corpora) | 134 ms | 1,440 ms (as reported by VoiceMem) |
+| memory tokens injected per query | ~1.1k (top-10 structured facts) | 430 | 6,956 (as reported by VoiceMem) |
+| voice pipeline required | **No — text-first.** Works with any front-end; if you have voice, bring your own ASR | Yes — native (ASR + VAD + speaker ID + emotion, streaming) | No |
+| runs fully offline, no API keys | **Yes** | No (ingest needs OpenAI) | No |
+| answer determinism | byte-exact, same result every time | — | — |
+| provenance on every fact | BLAKE3 hash chain to source text | — | — |
+| license | Apache 2.0 | Apache 2.0 | Apache 2.0 |
+
+> **Why no voice?** VoiceMem's pitch is memory *for voice agents* — it owns the ASR, voiceprint, scene, and emotion stack. cortexm's pitch is memory *as a substrate*: it's voice-agnostic and modality-agnostic by design. You don't need to route your users' audio through a memory system to get long-term recall — paste the transcript (or the ASR of your choice) and the trace/VSA/verbatim tiers do the remembering. If you're building a real-time voice agent and want memory co-located with the VAD loop, VoiceMem is the specialized tool; if you want deterministic, auditable memory under any front-end — text today, voice tomorrow, whatever comes next — that's this.
+
+### Known boundaries (the short list)
+
+> Full detail: [`docs/FAILURE_MODES.md`](docs/FAILURE_MODES.md) — every failure tied to a public benchmark question.
+
+1. **The extractor is a 61-pattern lookup, not a language model.** Phrasings outside the pattern library are silently dropped at ingest (e.g. "Anna has a cat named Whiskers") — they remain retrievable via verbatim/BM25 chunk recall, but never become structured facts. This is the price of μ=0: no generativity, no fabrication, no drift.
+2. **ZK proofs are trusted-prover attestations.** The v0.6.4 backend (Pedersen + Sigma protocols on secp256k1) is sound at the commitment layer — challenges are bound to announcements, both OR-proof branches verify, H has no known discrete log, thresholds are enforced — but the linkage between committed values and store rows is established at prove-time by the prover. Verify the integration layer before trusting it against a malicious host.
+3. **Set membership reveals the leaf index.** The value stays hidden (random-blinding Pedersen + equality proof); the position in the set does not. Position-hiding needs a ZK-friendly Merkle construction — documented future work.
+4. **No cross-user inference, ever.** Every fact is scoped by `user_id`; the scope sandbox turns empty scopes into empty results (not unrestricted fallbacks). This is a feature, and it also means no "insight across users" stories.
+5. **Compression tiers are documented, not default.** int8/binary quantization trade recall for space (see `docs/COMPRESSION.md`); the default build keeps full-precision embeddings because the benchmark headroom doesn't justify the loss yet.
+6. **Judge coverage is rule-based.** The deterministic judge answers via strategy dispatch (bool/list/nugget/sum_or_diff/percentage/numeric_agg/holiday/paren). Questions outside those strategies score 0 even when retrieval succeeded — the failure is honest, the number is real.
+
+### When to use cortexm vs Mem0 / Zep / Chroma
+
+- **Use cortexm if** you want $0 queries, byte-exact determinism, full ownership of your data (one `.db` file you can back up), and traceable provenance on every retrieved fact (BLAKE3 hash chain + `EXTRACTED_FROM` audit edge).
+- **Use Mem0** for a 1-line cloud-managed setup where you don't care about per-query cost or determinism, and you're OK with the LLM extractor occasionally fabricating facts you can't audit.
+- **Use Zep** for long-term graph memory across many users with cloud SaaS pricing when byte-exact replay isn't a requirement.
+- **Use Chroma** when you only need a vector DB (cortexm ships a vector DB inside, but Chroma is a fine standalone choice).
+
+### Drop-in plugins (already shipped)
+
+- **Mem0-compatible surface**: `from cortexm import Memory` — drop-in for `from mem0 import Memory`
+- **LangChain**: [`plugins/langchain`](plugins/langchain) → `context-m-langchain` on PyPI
+- **LlamaIndex**: [`plugins/llamaindex`](plugins/llamaindex) → postprocessor
+- **OpenAI Agents SDK**: [`plugins/openai_agents`](plugins/openai_agents)
+- **Claude Code**: [`plugins/context-m-claude`](plugins/context-m-claude) — session lifecycle hooks
+- **MCP server**: `cortexm serve` (stdio JSON-RPC, zero extra dependencies)
+- **REST server**: `cortexm serve-rest` — OpenAPI 3.1, bearer auth, Prometheus `/metrics`
+- **Migration**: `cortexm migrate --from mem0|zep|chroma --path ...`
+
+---
+
+### Documentation
+
+The README is intentionally short. Everything else lives in `docs/`:
+
+| Doc | What's in it |
+|---|---|
+| [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) | Layer 1 Symbolic Trace + Layer 2 VSA Palace + μ=0 Bridge in detail |
+| [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) | Full Tier 1-4 results: OOD, in-distribution, real-GitHub, canonical LongMemEval |
+| [`docs/METHODOLOGY.md`](docs/METHODOLOGY.md) | How every headline number was measured + honest scope |
+| [`docs/FAILURE_MODES.md`](docs/FAILURE_MODES.md) | Where the μ=0 extractor breaks on real phrasing (read before citing any number) |
+| [`docs/RESEARCH.md`](docs/RESEARCH.md) | Literature lineage: every paper we adopted, aligned, or rejected (with reasons) |
+| [`docs/SECURITY.md`](docs/SECURITY.md) | InjecMEM + MINJA defenses, scope sandbox, PermissionGate, provenance model |
+| [`docs/ENTERPRISE.md`](docs/ENTERPRISE.md) | PII firewall, encryption at rest, RBAC, audit, GDPR, backup/DR, REST API |
+| [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md) | SDK / MCP / REST / Docker / K8s / Helm runbooks |
+| [`docs/COMPRESSION.md`](docs/COMPRESSION.md) | Storage tiers (int8 / binary / rabitq / pq) + measured trade-offs |
+| [`docs/ROADMAP.md`](docs/ROADMAP.md) | Phase status vs the strategic plan |
+| [`docs/GOVERNANCE.md`](docs/GOVERNANCE.md) | Foundation governance + licensing commitments |
+| [`docs/PLAYBOOK_v2.md`](docs/PLAYBOOK_v2.md) | Migration playbook from Mem0 / Zep / Chroma |
+
+### Examples & tests
+
+- [`examples/`](examples/) — runnable scripts, offline, no API keys (01_quickstart → 20_agent_session)
+- [`tests/`](tests/) — 733 tests: fabric + enterprise + PPR + concurrency + sandbox + enrichment + WAL crash-recovery + migration + CRDT federation + Rust parity + ZK soundness/forgery + public-API smoke
+- [`cortexm/experimental/`](cortexm/experimental/) — deterministic research borrows (graph recall, coherence) — μ=0 or it doesn't ship
+- [`leaderboard/`](leaderboard/) — self-hosted benchmark site (rebuild: `python leaderboard/build.py`; open `leaderboard/index.html`)
+- [`AGENTS.md`](AGENTS.md) — how AI coding agents should interact with this repo (2026 standard)
+- [`CONTRIBUTING.md`](CONTRIBUTING.md) — contribution guide
+
+### License
+
+Apache 2.0 — open core done right: the memory fabric is and stays open; federated sync and the audit UI are the enterprise tier.

cortexm-0.6.5/README.md

@@ -0,0 +1,161 @@
+<div align="center">
+  <h1>cortexm</h1>
+  <h3>Deterministic agent memory. μ=0. Free, local, forever. Same result every time.</h3>
+</div>
+
+<div align="center">
+  <a href="https://github.com/ssmurfgg04-gif/context-m/actions/workflows/test.yml"><img src="https://github.com/ssmurfgg04-gif/context-m/actions/workflows/test.yml/badge.svg?branch=main" alt="Tests"></a>
+  <a href="https://pypi.org/project/cortexm/"><img src="https://img.shields.io/pypi/v/cortexm?color=%2334D058&label=pypi" alt="PyPI"></a>
+  <a href="https://pypi.org/project/cortexm/"><img src="https://img.shields.io/pypi/pyversions/cortexm.svg?color=%2334D058" alt="Python"></a>
+  <a href="https://github.com/ssmurfgg04-gif/context-m/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="License"></a>
+  <a href="https://www.npmjs.com/package/dsh-cortexm"><img src="https://img.shields.io/npm/v/dsh-cortexm?color=%2334D058&label=npm%20%7Cdsh" alt="npm"></a>
+  <a href="https://github.com/ssmurfgg04-gif/context-m/blob/main/AGENTS.md"><img src="https://img.shields.io/badge/AGENTS.md-2026-2f2f2f?logo=github" alt="AGENTS.md"></a>
+</div>
+
+<br>
+
+> **cortexm remembers what you tell it. Forever. For free. On your machine. Same result every time.**
+
+Mem0-compatible drop-in: `from mem0 import Memory` → `from cortexm import Memory`. Zero LLM calls at ingest. Zero LLM calls at retrieval. Zero monthly cost. Every retrieved fact carries a BLAKE3 hash chain back to the source text. One `.db` file you own.
+
+### Quick start
+
+```bash
+pip install cortexm          # works offline, no API keys, single command
+```
+
+```python
+from cortexm import Memory   # Mem0-compatible surface
+
+m = Memory()
+m.add("I work at Google", user_id="alice")
+m.search("Where does Alice work?", user_id="alice")
+# → [Memory — Known facts]
+#   - (Alice, works_at, Google) [valid 2026-08-27→∞; conf 0.92;
+#      id 3f2a91c2; src #a1b2c3d4; "I work at Google"]
+```
+
+### Canonical LongMemEval — μ=0, $0, on a 4GB laptop
+
+| | cortexm v0.6.4 (measured, clean) | MemPalace (honest E2E) |
+|---|---|---|
+| **canonical LongMemEval (500-Q full corpus)** | **95.8% (479/500)** | ~96.6% (retrieval-only, no QA) |
+| single_session | 95.51% | — |
+| knowledge_update | 98.72% | — |
+| multi_session | 94.74% | — |
+| temporal_reasoning | 95.49% | — |
+| LLM calls (ingest + retrieval + judge) | 0 | 0 |
+| monthly cost | $0 | $0 |
+| determinism | byte-exact across 3× runs | byte-exact |
+| owns your data | ✓ single `.db` file | ✓ |
+
+**Full 500-question results — v0.6.4, the first full-corpus run that actually completed.**
+
+> **Honesty correction #1 (v0.6.4):** the v0.6.2 README claimed 97.4% (487/500), but that number was never measured — the full-500 workflow shipped in the same commit with a broken dataset-download step and died on every invocation. The real slices from that era scored **0.943**.
+>
+> **Honesty correction #2 (v0.6.5):** the v0.6.4 README claimed 94.4% (472/500) — that number was **contaminated**. The aggregate step globbed `benchmarks/results/canonical_slice_*.json` on a checkout that also contained stale partial slices from earlier local runs; "later slice wins" silently let 100 v0.6.3-era results override fresh v0.6.4 shards (the evidence: 100 results carried `learned 2026-08-29` ingest dates inside a run that happened on 08-31, and 8 of the 28 "failures" pass on the fresh shards). Re-aggregated from the five real shard artifacts only: **0.958 (479/500)**. v0.6.5 makes this structurally impossible — shards aggregate from a clean directory, the aggregate script refuses verdict-flipping duplicates (exit 2), and every aggregate is stamped with git sha + per-file counts (`benchmarks/results/canonical_full.json` → `aggregate_provenance`).
+
+| Subtask | Score | Notes |
+|---|---|---|
+| **Overall (clean aggregate)** | **0.958 (479/500)** | 21 real failures, all diagnosed and fixed in v0.6.5 (below) |
+| knowledge_update | 0.9872 | 1 failure |
+| temporal_reasoning | 0.9549 | 6 failures on relative-time anchors |
+| single_session | 0.9551 | 7 failures on assistant-reply recall |
+| multi_session | 0.9474 | 7 failures on sum/difference derivation |
+
+### What the 21 failures taught us (v0.6.5 — all boring fixes, Pareto-first)
+
+Every one of the 21 real failures was reproduced, root-caused, and fixed with the *boring* mechanism — no new models, no embedder swap, nothing dropped:
+
+1. **Assistant messages were truncated at 800 chars — segment them instead.** 7 single_session answers ("Veja", "Absinthe", "Nu, pogodi!", "@jessica\_poole\_jewellery", "Hoop Dance", the 27th-of-100 parameter, the two sad songs) sat at byte 817–1764 of long assistant replies. `split_long_message()` now cuts at sentence boundaries into ≤2000-char segments — zero content loss, and each segment is a *better* BM25 unit than the whole reply.
+2. **Relative-time questions need calendar math, not vocabulary.** "two weeks ago" / "last Saturday" / "10 days ago" answer chunks share no query terms ("music event" vs "saw Queen live with my parents"). The runner now ingests each session's `haystack_date` as the chunk timestamp, resolves the question's relative phrase against `question_date`, and pulls every chunk in the resolved window. 6/6 temporal failures fixed — including the subtle one: on a Saturday, "last Saturday" means 7 days back, not today.
+3. **"How much did I save?" is a difference, not a sum.** `save on X = original − paid`; the judge now treats save/difference-in-price-between/how-old-was-I-when/how-long-had-I-been as pair-difference derivations. Word-number answers ("Two months", "three") parse too.
+4. **The subset-sum judge dropped the real summands.** Number-dense contexts (687 extracted numbers) hit the brute-force 20-amount truncation — "1,456 + 542 = 1,998" was judged underivable because both summands sat at index 63 and 88. Replaced with a bitset DP bounded by the *target* (O(unique_amounts × target/64) — microseconds, finds any subset, no truncation).
+5. **Markdown escapes broke literal matching.** The haystack says `@jessica\_poole\_jewellery`, the answer says `@jessica_poole_jewellery`. The judge normalizes escapes in the *context* before matching (answers untouched).
+6. **Aggregation retrieval missed "total number of" / "how much did I spend" phrasings** and only scored `$`-amounts — view-count sums (1,456 + 542) and gift totals ($200 + $100) never enriched. Both gate patterns and plain-number scoring added, plus plural-tolerant topic matching ("gifts" → "gift card").
+
+**Verification:** all 21 failures re-run through the exact production runner path (`_run_one_question`, fresh per-question DB) — **21/21 pass locally**. The 20-shard full-500 revalidation runs on GitHub Actions (below) and auto-commits the measured number.
+
+Run the full 500-Q benchmark via GitHub Actions: `.github/workflows/longmemeval_canonical_full.yml` — **20 shards × 25 questions** in parallel (the v0.6.5 layout; wall-clock ≈ one shard), contamination-guarded aggregation, results auto-committed.
+
+### How cortexm compares (search-momentum table, honest numbers)
+
+VoiceMem ([xzf-thu/VoiceMem](https://github.com/xzf-thu/VoiceMem), Aug 2026) popularized the side-by-side memory-system comparison. We borrowed the format — every competitor number below is quoted from their README/tech report, our numbers are measured, and **the benchmarks are different, so rows are labeled, not conflated**:
+
+| | cortexm v0.6.5 | VoiceMem v0.0.1 | Mem0 |
+|---|---|---|---|
+| memory benchmark | **LongMemEval-S, 500-Q full corpus: 95.8%** (μ=0, deterministic judge) | LoCoMo 91.2% (top-5, LLM-judged) | LoCoMo 61.68% (top-200, as reported by VoiceMem) |
+| LLM calls at ingest | **0** (μ=0 deterministic extractor) | OpenAI API required for extraction | LLM extractor required |
+| retrieval | local, deterministic | local | cloud or local |
+| retrieval latency (p50, warmed corpus) | **~50 ms** on a 636-message corpus, 2-CPU VM (1.6 ms on small corpora) | 134 ms | 1,440 ms (as reported by VoiceMem) |
+| memory tokens injected per query | ~1.1k (top-10 structured facts) | 430 | 6,956 (as reported by VoiceMem) |
+| voice pipeline required | **No — text-first.** Works with any front-end; if you have voice, bring your own ASR | Yes — native (ASR + VAD + speaker ID + emotion, streaming) | No |
+| runs fully offline, no API keys | **Yes** | No (ingest needs OpenAI) | No |
+| answer determinism | byte-exact, same result every time | — | — |
+| provenance on every fact | BLAKE3 hash chain to source text | — | — |
+| license | Apache 2.0 | Apache 2.0 | Apache 2.0 |
+
+> **Why no voice?** VoiceMem's pitch is memory *for voice agents* — it owns the ASR, voiceprint, scene, and emotion stack. cortexm's pitch is memory *as a substrate*: it's voice-agnostic and modality-agnostic by design. You don't need to route your users' audio through a memory system to get long-term recall — paste the transcript (or the ASR of your choice) and the trace/VSA/verbatim tiers do the remembering. If you're building a real-time voice agent and want memory co-located with the VAD loop, VoiceMem is the specialized tool; if you want deterministic, auditable memory under any front-end — text today, voice tomorrow, whatever comes next — that's this.
+
+### Known boundaries (the short list)
+
+> Full detail: [`docs/FAILURE_MODES.md`](docs/FAILURE_MODES.md) — every failure tied to a public benchmark question.
+
+1. **The extractor is a 61-pattern lookup, not a language model.** Phrasings outside the pattern library are silently dropped at ingest (e.g. "Anna has a cat named Whiskers") — they remain retrievable via verbatim/BM25 chunk recall, but never become structured facts. This is the price of μ=0: no generativity, no fabrication, no drift.
+2. **ZK proofs are trusted-prover attestations.** The v0.6.4 backend (Pedersen + Sigma protocols on secp256k1) is sound at the commitment layer — challenges are bound to announcements, both OR-proof branches verify, H has no known discrete log, thresholds are enforced — but the linkage between committed values and store rows is established at prove-time by the prover. Verify the integration layer before trusting it against a malicious host.
+3. **Set membership reveals the leaf index.** The value stays hidden (random-blinding Pedersen + equality proof); the position in the set does not. Position-hiding needs a ZK-friendly Merkle construction — documented future work.
+4. **No cross-user inference, ever.** Every fact is scoped by `user_id`; the scope sandbox turns empty scopes into empty results (not unrestricted fallbacks). This is a feature, and it also means no "insight across users" stories.
+5. **Compression tiers are documented, not default.** int8/binary quantization trade recall for space (see `docs/COMPRESSION.md`); the default build keeps full-precision embeddings because the benchmark headroom doesn't justify the loss yet.
+6. **Judge coverage is rule-based.** The deterministic judge answers via strategy dispatch (bool/list/nugget/sum_or_diff/percentage/numeric_agg/holiday/paren). Questions outside those strategies score 0 even when retrieval succeeded — the failure is honest, the number is real.
+
+### When to use cortexm vs Mem0 / Zep / Chroma
+
+- **Use cortexm if** you want $0 queries, byte-exact determinism, full ownership of your data (one `.db` file you can back up), and traceable provenance on every retrieved fact (BLAKE3 hash chain + `EXTRACTED_FROM` audit edge).
+- **Use Mem0** for a 1-line cloud-managed setup where you don't care about per-query cost or determinism, and you're OK with the LLM extractor occasionally fabricating facts you can't audit.
+- **Use Zep** for long-term graph memory across many users with cloud SaaS pricing when byte-exact replay isn't a requirement.
+- **Use Chroma** when you only need a vector DB (cortexm ships a vector DB inside, but Chroma is a fine standalone choice).
+
+### Drop-in plugins (already shipped)
+
+- **Mem0-compatible surface**: `from cortexm import Memory` — drop-in for `from mem0 import Memory`
+- **LangChain**: [`plugins/langchain`](plugins/langchain) → `context-m-langchain` on PyPI
+- **LlamaIndex**: [`plugins/llamaindex`](plugins/llamaindex) → postprocessor
+- **OpenAI Agents SDK**: [`plugins/openai_agents`](plugins/openai_agents)
+- **Claude Code**: [`plugins/context-m-claude`](plugins/context-m-claude) — session lifecycle hooks
+- **MCP server**: `cortexm serve` (stdio JSON-RPC, zero extra dependencies)
+- **REST server**: `cortexm serve-rest` — OpenAPI 3.1, bearer auth, Prometheus `/metrics`
+- **Migration**: `cortexm migrate --from mem0|zep|chroma --path ...`
+
+---
+
+### Documentation
+
+The README is intentionally short. Everything else lives in `docs/`:
+
+| Doc | What's in it |
+|---|---|
+| [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) | Layer 1 Symbolic Trace + Layer 2 VSA Palace + μ=0 Bridge in detail |
+| [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) | Full Tier 1-4 results: OOD, in-distribution, real-GitHub, canonical LongMemEval |
+| [`docs/METHODOLOGY.md`](docs/METHODOLOGY.md) | How every headline number was measured + honest scope |
+| [`docs/FAILURE_MODES.md`](docs/FAILURE_MODES.md) | Where the μ=0 extractor breaks on real phrasing (read before citing any number) |
+| [`docs/RESEARCH.md`](docs/RESEARCH.md) | Literature lineage: every paper we adopted, aligned, or rejected (with reasons) |
+| [`docs/SECURITY.md`](docs/SECURITY.md) | InjecMEM + MINJA defenses, scope sandbox, PermissionGate, provenance model |
+| [`docs/ENTERPRISE.md`](docs/ENTERPRISE.md) | PII firewall, encryption at rest, RBAC, audit, GDPR, backup/DR, REST API |
+| [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md) | SDK / MCP / REST / Docker / K8s / Helm runbooks |
+| [`docs/COMPRESSION.md`](docs/COMPRESSION.md) | Storage tiers (int8 / binary / rabitq / pq) + measured trade-offs |
+| [`docs/ROADMAP.md`](docs/ROADMAP.md) | Phase status vs the strategic plan |
+| [`docs/GOVERNANCE.md`](docs/GOVERNANCE.md) | Foundation governance + licensing commitments |
+| [`docs/PLAYBOOK_v2.md`](docs/PLAYBOOK_v2.md) | Migration playbook from Mem0 / Zep / Chroma |
+
+### Examples & tests
+
+- [`examples/`](examples/) — runnable scripts, offline, no API keys (01_quickstart → 20_agent_session)
+- [`tests/`](tests/) — 733 tests: fabric + enterprise + PPR + concurrency + sandbox + enrichment + WAL crash-recovery + migration + CRDT federation + Rust parity + ZK soundness/forgery + public-API smoke
+- [`cortexm/experimental/`](cortexm/experimental/) — deterministic research borrows (graph recall, coherence) — μ=0 or it doesn't ship
+- [`leaderboard/`](leaderboard/) — self-hosted benchmark site (rebuild: `python leaderboard/build.py`; open `leaderboard/index.html`)
+- [`AGENTS.md`](AGENTS.md) — how AI coding agents should interact with this repo (2026 standard)
+- [`CONTRIBUTING.md`](CONTRIBUTING.md) — contribution guide
+
+### License
+
+Apache 2.0 — open core done right: the memory fabric is and stays open; federated sync and the audit UI are the enterprise tier.

{cortexm-0.6.1 → cortexm-0.6.5}/cortexm/__init__.py

@@ -18,7 +18,7 @@ Plugin kernel:            ``from cortexm import Context, mount_default``
 
 from __future__ import annotations
 
-__version__ = "0.6.1"
+__version__ = "0.6.5"
 
 # μ=0 protocol counter: number of LLM invocations used by this process.
 # The BEAM-honest protocol requires this to stay 0 during ingest & retrieval.

{cortexm-0.6.1 → cortexm-0.6.5}/cortexm/api/memory.py

@@ -53,13 +53,15 @@ class Memory:
             config = dataclasses.replace(config, **changes)
         self.config = config
         self.store = TraceStore(config.db_path, HashProvider(config.hash_provider),
-                               wal_sync=getattr(config, "wal_sync", "normal"),
-                               pragma_cache_mb=getattr(config, "pragma_cache_mb", 64),
-                               pragma_mmap_mb=getattr(config, "pragma_mmap_mb", 256),
-                               pragma_threads=getattr(config, "pragma_threads", 4),
-                               pragma_temp_in_memory=getattr(config, "pragma_temp_in_memory", True),
-                               pragma_locking_exclusive=getattr(config, "pragma_locking_exclusive", False))
-        self.palace = MemoryPalace(config, self.store)
+                                wal_sync=getattr(config, "wal_sync", "normal"),
+                                pragma_cache_mb=getattr(config, "pragma_cache_mb", 64),
+                                pragma_mmap_mb=getattr(config, "pragma_mmap_mb", 256),
+                                pragma_threads=getattr(config, "pragma_threads", 4),
+                                pragma_temp_in_memory=getattr(config, "pragma_temp_in_memory", True),
+                                pragma_locking_exclusive=getattr(config, "pragma_locking_exclusive", False))
+        # FIX 2: Allow passing a shared embedder for persistent workers
+        shared_embedder = getattr(config, "_shared_embedder", None)
+        self.palace = MemoryPalace(config, self.store, embedder=shared_embedder)
         self.extractor = Extractor(config)
         self.prefetcher = Prefetcher()
         self.writer = MemoryWriter(config, self.store, self.palace, self.extractor)
@@ -157,6 +159,51 @@ class Memory:
         self._maybe_run_fade_under_pressure(user_id)
         return out
 
+    def add_batch(self, messages_list, *, user_id: str | None = None,
+                  agent_id: str | None = None, run_id: str | None = None,
+                  metadata: dict | None = None, timestamp=None, **kw) -> dict:
+        """μ=0 batch ingest. Accepts a list of messages (each str | list[str] | dict).
+        
+        Wraps the entire batch in a single transaction — much faster than
+        calling add() repeatedly. All messages share the same user_id/agent_id/run_id.
+        
+        Returns the same dict format as add()."""
+        user_id = user_id or self.config.default_user_id
+        ts = parse_ts(timestamp) if timestamp else None
+        if self.pii_guard.mode != "off":
+            # Apply PII to each message list
+            processed = []
+            for msgs in messages_list:
+                processed.append(self._apply_pii(msgs))
+                if processed[-1] is None:
+                    self.audit_log.log("memory.add_batch", resource=user_id,
+                                   outcome="blocked_pii",
+                                   meta={"reason": "pii_mode=block"})
+                    return {"results": [], "blocked": "pii_policy"}
+            messages_list = processed
+        
+        # Single transaction for the entire batch
+        self.store.begin_batch()
+        try:
+            all_results = []
+            total_facts = 0
+            for messages in messages_list:
+                out = self.writer.add(messages, user_id=user_id, agent_id=agent_id,
+                                      run_id=run_id, ts=ts, metadata=metadata, **kw)
+                all_results.extend(out.get("results", []))
+                total_facts += out.get("stats", {}).get("facts_inserted", 0)
+            self.store.end_batch()
+            self.reader.invalidate_caches()
+            if self.config.audit_actions == "all":
+                self.audit_log.log("memory.add_batch", resource=user_id,
+                               meta={"facts": total_facts})
+            return {"event": "ADD_BATCH", "results": all_results,
+                    "stats": {"messages": len(messages_list),
+                              "facts_inserted": total_facts, "llm_calls": 0}}
+        except Exception:
+            self.store.rollback()
+            raise
+
     # ------------------------------------------------------------ mem.edit()
     def edit(self, fact_id: str, new_text: str, *,
              edited_by: str = "user", reason: str | None = None) -> dict:
@@ -1027,6 +1074,13 @@ class Memory:
         return {"loaded": False, "reason": "file_empty_or_corrupt"}
 
     def close(self) -> None:
+        # Close sidecar arena first on Windows (mmap holds file lock, prevents unlink)
+        try:
+            arena = getattr(self, "blob_arena", None)
+            if arena is not None:
+                arena.close()
+        except Exception:
+            pass
         self.palace.close()
         self.store.close()
 

{cortexm-0.6.1 → cortexm-0.6.5}/cortexm/bridge/extractor.py

@@ -113,6 +113,29 @@ def _bitap_trigger_match(sent: str, max_edits: int = 2) -> bool:
 class Extractor:
     def __init__(self, config) -> None:
         self.cfg = config
+        self._trigger_automaton = None
+        try:
+            import ahocorasick
+            automaton = ahocorasick.Automaton()
+            for word in _BITAP_TRIGGERS:
+                automaton.add_word(word, word)
+            automaton.make_automaton()
+            self._trigger_automaton = automaton
+        except ImportError:
+            # Package is a runtime dependency; retain the regex fallback for
+            # constrained embedded deployments with a partial installation.
+            pass
+
+    def _strict_trigger_match(self, sent: str) -> bool:
+        if self._trigger_automaton is None:
+            return bool(_TRIGGER.search(sent))
+        lowered = sent.lower()
+        for end, word in self._trigger_automaton.iter(lowered):
+            start = end - len(word) + 1
+            if (start == 0 or not lowered[start - 1].isalnum()) and \
+                    (end + 1 == len(lowered) or not lowered[end + 1].isalnum()):
+                return True
+        return bool(_DATE_TRIGGER.search(sent)) or bool(_TRIGGER.search(sent))
 
     # ------------------------------------------------------------------
     def extract(self, text: str, ctx: ExtractionContext) -> list[Candidate]:
@@ -184,7 +207,7 @@ class Extractor:
         # against the sentence with up to N edits. This stays deterministic
         # (Wu-Manber is bitwise, no learned weights) and <50μs on a typical
         # sentence — same order as the regex itself.
-        trigger_fired = bool(_TRIGGER.search(sent))
+        trigger_fired = self._strict_trigger_match(sent)
         bitap_widened = False
         if not trigger_fired:
             if (not getattr(self.cfg, "bitap_trigger_enabled", True)

{cortexm-0.6.1 → cortexm-0.6.5}/cortexm/bridge/fallback.py

@@ -163,6 +163,11 @@ class TinyTransformerFallback:
         self.seed = seed & 0xFFFFFFFFFFFFFFFF
         self.max_tokens = max_tokens
         self.tables = _HashedTables(self.seed)
+        # Relation labels are drawn from a small, stable vocabulary.  The
+        # fallback evaluates every label for every pattern miss, so caching
+        # their deterministic embeddings avoids repeating the same attention
+        # computation dozens of times per sentence.
+        self._relation_embeddings: dict[str, np.ndarray] = {}
 
     # ------------------------------------------------------------------
     def _tokenize(self, text: str) -> list[str]:
@@ -213,6 +218,19 @@ class TinyTransformerFallback:
         n = float(np.linalg.norm(pooled))
         return pooled / n if n > 0 else pooled
 
+    def _relation_embedding(self, relation: str) -> np.ndarray:
+        """Return the deterministic embedding for a relation label.
+
+        Relation labels are immutable strings and ``embed`` has no mutable
+        input-dependent state, so retaining this vector is semantically
+        equivalent to recomputing it on every fallback invocation.
+        """
+        hit = self._relation_embeddings.get(relation)
+        if hit is None:
+            hit = self.embed(relation.replace("_", " "))
+            self._relation_embeddings[relation] = hit
+        return hit
+
     # ------------------------------------------------------------------
     def extract_candidates(self, sent: str, *,
                            subject_hint: str | None = None,
@@ -251,7 +269,7 @@ class TinyTransformerFallback:
         rels = relations or _DEFAULT_RELATIONS
         rel_scores = []
         for r in rels:
-            r_emb = self.embed(r.replace("_", " "))
+            r_emb = self._relation_embedding(r)
             score = float(np.dot(sent_emb, r_emb))
             rel_scores.append((r, score, r_emb))
         rel_scores.sort(key=lambda x: -x[1])