enigma-memory 0.1.2 → 0.1.3

package/docs/benchmark-reproducibility.md

@@ -0,0 +1,113 @@
+# Benchmark reproducibility
+
+This guide explains how to reproduce the current local Enigma memory benchmark, save the public-safe JSON report, cite the external benchmark standards it is modeled after, and understand what is still required before publishing live third-party comparisons.
+
+## What is reproducible today
+
+The current package is `enigma-memory@0.1.3`. The local benchmark is available through the package script and the script file it wraps:
+
+```sh
+cd enigma
+npm run benchmark:memory-suite
+npm run benchmark:memory-suite -- --out benchmark-report.json
+node scripts/run-memory-benchmarks.mjs --out benchmark-report.json
+```
+
+The `--out` form writes the report to the requested path and prints only a small status object. Without `--out`, the command writes the full JSON report to stdout.
+
+The report schema is `enigma.memory_benchmark_suite.v1`. It is designed to be public-safe: it contains aggregate metrics, commitments, citations, cross-provider profile labels, and claim boundaries. It does not include raw fixture memory, private question text, private answer text, provider transcripts, credentials, account ids, or local absolute paths.
+
+## Reproduce and save JSON
+
+1. Use a clean checkout containing `enigma-memory@0.1.3`.
+2. From a repository root that contains `enigma/package.json`, enter the package directory:
+
+   ```sh
+   cd enigma
+   ```
+
+   If your checkout already has `package.json` for `enigma-memory` at the current directory, skip this `cd`.
+
+3. Install the package dependencies with the reviewed package command:
+
+   ```sh
+   npm install
+   ```
+
+4. Run the benchmark and save the public-safe JSON report:
+
+   ```sh
+   npm run benchmark:memory-suite -- --out benchmark-report.json
+   ```
+
+5. Preserve the JSON file with the command, package version, operating system/runtime, and review context that produced it.
+6. When sharing the result publicly, share the generated JSON report only after confirming it still has `public_safe: true` and `schema: "enigma.memory_benchmark_suite.v1"`.
+
+The local fixture measures Enigma-controlled operations only: vault remember/update, vault export/import, passport context-pack retrieval, optimizer token estimates and duplicate removal, bundle/context-pack verification, abstention behavior, exact-answer recall over the deterministic fixture, and p50/p95 operation latency from `performance.now`.
+
+## Local baseline rows in the report
+
+The report now includes `metrics.local_baseline_comparisons`, which compares deterministic local baselines over the same private fixture questions. These rows are local package evidence only: they do not call hosted providers, use provider APIs, or support invoice savings, ROI, compliance, model-forgetting, or benchmark-leadership claims.
+
+| Row | Local boundary |
+| --- | --- |
+| `full_context` | Supplies every active fixture memory without optimization or deduplication. |
+| `recency_last_n` | Supplies the three most recently updated active fixture memories. |
+| `keyword_filter` | Supplies active fixture memories whose content or tags match deterministic query terms. |
+| `enigma_context_pack` | Uses the Enigma passport context-pack compiler and optimizer boundary. |
+
+The report also includes `public_claims_allowed`; keep public copy within those local-fixture boundaries unless separate reviewed external evidence exists.
+
+## How to cite external benchmark standards
+
+Use these standards as citations and task-category references, not as claimed Enigma results unless the exact external benchmark has been run and reviewed:
+
+- LoCoMo: https://snap-research.github.io/locomo/ — cite for long-term conversational-memory QA, event summarization, and multimodal generation over long conversations.
+- LongMemEval: https://arxiv.org/abs/2410.10813 — cite for information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.
+
+The current local report mirrors some task categories from those benchmarks, but it does not download LoCoMo or LongMemEval data, run their official evaluation pipelines, or claim leaderboard-equivalent results.
+
+## Why live third-party comparisons are not claimed yet
+
+The current benchmark does not call external provider APIs, external SDKs, hosted memory services, ChatGPT native memory, Claude memory tooling, or third-party agent loops. Cross-provider rows in the report are profile labels that reuse the same Enigma context-pack boundary; they do not call or compare live provider models and are not live provider rankings.
+
+Real comparisons require fixed adapters, fixed datasets, fixed agent/tool loops, explicit provider terms review, and reviewed handling of secrets and raw benchmark data. Memory quality can change with the surrounding agent framework and tool loop, so a fair comparison must document more than the memory store.
+
+The current report must not be used as evidence of provider-side deletion, model forgetting, compliance certification, token ROI, provider invoice savings, benchmark leadership, hosted-cloud readiness, or “best in world” superiority.
+
+## External comparison requirements
+
+Use placeholder environment names only. Do not commit real tokens, API keys, account ids, provider transcripts, raw benchmark conversations, or private memory.
+
+The report field `external_competitor_adapters` is a requirements matrix, not a score table. External rows are expected to remain requirements-only until credentials, runtimes, and datasets are supplied and reviewed: `can_run_in_this_harness: false`, `scores_included: false`, and no recall, abstention, token, latency, or ranking scores.
+
+| Target | Runtime or SDK needed | Placeholder secrets and local inputs | Dataset requirement | Adapter boundary before results can be claimed |
+| --- | --- | --- | --- | --- |
+| Letta | Letta SDK/runtime; documented SDK packages include `@letta-ai/letta-client` and `letta-client`; API-key-backed service access may be required. | `LETTA_API_KEY`, `LETTA_BASE_URL`, `LETTA_PROJECT_ID`, `BENCHMARK_DATASET_PATH` | Local reviewed LoCoMo/LongMemEval split or another reviewed local dataset file with license, version, split, and checksum metadata. | Build a Letta adapter that fixes the agent loop, memory write/read policy, model settings, and scoring path. Results may describe that configured Letta run only, not generic provider deletion or model forgetting. |
+| LangGraph memory | LangGraph runtime with short-term checkpointer memory and long-term namespaced store. | `LANGGRAPH_CHECKPOINTER_URI`, `LANGGRAPH_STORE_URI`, `BENCHMARK_DATASET_PATH` | Same local dataset file and split used for Enigma and every competitor. | Fix graph topology, checkpoint scope, namespace policy, retrieval policy, model/tool loop, and scorer. Do not attribute graph/tool behavior solely to the memory store. |
+| Zep | Zep service/runtime positioned around temporal Context Graph and Context Lake retrieval. | `ZEP_API_KEY`, `ZEP_PROJECT_ID`, `ZEP_BASE_URL`, `BENCHMARK_DATASET_PATH` | Same local dataset file and split; include source checksum and whether any provider-side graph state is reused or reset. | Build a Zep adapter that records ingest, session, retrieval, reset, and scoring policy. Zep’s sub-200ms retrieval positioning is a vendor/source fact, not an Enigma-measured claim until measured in the same harness. |
+| Mem0 | Mem0 platform or open-source stack; positioned as a universal self-improving memory layer. | `MEM0_API_KEY`, `MEM0_BASE_URL`, `MEM0_PROJECT_ID`, `BENCHMARK_DATASET_PATH` | Same local dataset file and split; record Mem0 deployment flavor/version. | Build a Mem0 adapter with fixed extraction, update, retrieval, reset, and scorer behavior. Self-improving or platform behavior must be bounded to the configured run. |
+| OpenAI native ChatGPT memory | ChatGPT consumer-app/native memory environment. It is not directly available through a public API in this harness. | No usable harness secret; `OPENAI_API_KEY` alone is not sufficient to exercise ChatGPT native memory. | No fair automated dataset run until an approved interface can load/reset/query native memory reproducibly. | Do not claim live native ChatGPT memory comparison from this repository. A future adapter would need an approved public interface, reproducible memory reset/load semantics, and provider-policy review. |
+| Claude memory tool | Client-side/provider-specific memory tool environment. | `CLAUDE_MEMORY_TOOL_CONFIG`, `ANTHROPIC_API_KEY`, `BENCHMARK_DATASET_PATH` | Same local dataset file and split, plus reviewed tool-state reset/export rules. | Build an adapter around the exact client/tool environment, not generic Claude model behavior. Results can only cover that configured memory-tool setup. |
+
+## Source references for adapter planning
+
+- Letta MemGPT concepts: https://docs.letta.com/concepts/memgpt/
+- Zep documentation: https://help.getzep.com/
+- Mem0 documentation: https://docs.mem0.ai/
+- LangGraph memory documentation: https://docs.langchain.com/oss/python/langgraph/memory
+- OpenAI ChatGPT memory FAQ: https://help.openai.com/en/articles/8590148-memory-faq
+- Claude memory tool support article: https://support.anthropic.com/en/articles/11145838-using-claude-memory
+
+## Minimum evidence for a future live comparison
+
+Before publishing external comparison language, capture all of the following in the benchmark report or an adjacent reviewed evidence file:
+
+1. Package version, benchmark schema, command, timestamp, OS/runtime, and adapter version.
+2. Dataset name, source URL, license review status, local file checksum, split, and record count.
+3. Secret names used as placeholders, with confirmation that no secret values are printed or persisted.
+4. Adapter configuration: SDK/runtime version, model where applicable, memory write/read policy, reset policy, context limits, retry policy, and scoring code.
+5. Per-target raw scoring inputs retained privately when license permits, with public reports limited to safe aggregates.
+6. Explicit boundaries separating memory-store behavior, agent-loop behavior, model behavior, provider-hosted state, and Enigma receipt verification.
+
+Until that evidence exists, use only the local benchmark claim: Enigma can reproduce deterministic local memory-fixture operations and emit a public-safe `enigma.memory_benchmark_suite.v1` JSON report.

package/docs/developer-ecosystem.md

@@ -7,6 +7,7 @@ Enigma Memory is a local-first SDK, CLI, MCP server, and service-contract packag
 - SDK/API guide: [`docs/sdk-api.md`](./sdk-api.md)
 - Node example app: [`examples/node-basic-memory.mjs`](../examples/node-basic-memory.mjs)
 - GitHub Actions example: [`examples/ci/github-actions.yml`](../examples/ci/github-actions.yml)
+- Benchmark reproducibility guide: [`docs/benchmark-reproducibility.md`](./benchmark-reproducibility.md)
 - Generic MCP client template: [`templates/mcp-client-config.json`](../templates/mcp-client-config.json)
 
 ## Local SDK loop
@@ -24,16 +25,17 @@ The example app prints ids, counts, roots, and verification status only. It does
 
 ## CLI and CI loop
 
-The CI example installs Node 24, installs the published `enigma-memory` package, runs:
+The CI example installs Node 24, installs the published `enigma-memory@0.1.3` package, runs:
 
 ```sh
 npx enigma quickstart --overwrite
 npx enigma doctor
+npm run benchmark:memory-suite -- --out benchmark-report.json
 ```
 
-and then runs a small ESM import smoke. It does not require GitHub secrets, cloud provider credentials, npm tokens, private bundles, or local path assumptions.
+and then runs a small ESM import smoke. It does not require GitHub secrets, cloud provider credentials, npm tokens, private bundles, or local path assumptions. The benchmark step writes a public-safe local JSON report using schema `enigma.memory_benchmark_suite.v1`; see the benchmark reproducibility guide for claim boundaries and the requirements for any future live third-party comparison.
 
-Use the workflow as a template in a consumer repository. It is intentionally limited to install/import/doctor smoke coverage and local proof generation; it does not publish packages, deploy infrastructure, or contact hosted Enigma cloud.
+Use the workflow as a template in a consumer repository. It is intentionally limited to install/import/doctor smoke coverage, local proof generation, and deterministic local benchmark evidence; it does not publish packages, deploy infrastructure, contact hosted Enigma cloud, or call external memory providers.
 
 ## MCP client loop
 

package/docs/memory-benchmarks.md

@@ -29,15 +29,46 @@ The harness measures local Enigma operations only:
 - context-pack retrieval through the passport package;
 - optimizer plan token estimates and duplicate removal;
 - bundle and context-pack verification;
+- local baseline comparisons over the same deterministic fixture questions;
 - p50/p95 latency with `performance.now`.
 
-Reported metrics include exact-answer recall, abstention correctness, context-token reduction versus a full-context baseline, duplicate candidates removed, operation latency summaries, verification status, and same-boundary cross-provider profile rows.
+Reported metrics include exact-answer recall, abstention correctness, estimated prompt tokens, duplicate candidates removed where applicable, operation latency summaries, verification status, and same-boundary cross-provider profile rows.
+
+## Local baseline comparison
+
+`metrics.local_baseline_comparisons` compares Enigma against deterministic local baselines only. Every row scores the same private fixture questions and keeps raw memory, question text, and answer text out of the report.
+
+| Row | Boundary | Reported fields |
+| --- | --- | --- |
+| `full_context` | Supplies every active fixture memory without optimization or deduplication. | Recall, abstention correctness, estimated prompt tokens, selected memory count, p50/p95 local latency. |
+| `recency_last_n` | Supplies the three most recently updated active fixture memories. | Same fields; duplicate removal is marked not applicable. |
+| `keyword_filter` | Supplies active fixture memories whose content or tags match deterministic query terms. | Same fields; duplicate removal is marked not applicable. |
+| `enigma_context_pack` | Uses the Enigma passport context-pack compiler and optimizer boundary. | Same fields plus duplicate-removal counts from the Enigma optimizer plan. |
+
+These rows are local package evidence only. They do not compare hosted providers, do not use provider APIs, and do not support invoice savings, ROI, compliance, model-forgetting, or benchmark-leadership claims.
+
+## External competitor adapter requirements
+
+`external_competitor_adapters` is a requirements matrix, not a score table. Each row has `status: "not_run_requires_credentials_or_runtime"`, `can_run_in_this_harness: false`, `required_artifacts`, `official_doc`, an exact `boundary_reason`, and `scores_included: false`.
+
+| Adapter | Official source | Required artifacts before scoring | Boundary reason |
+| --- | --- | --- | --- |
+| Letta / MemGPT | https://docs.letta.com/concepts/memgpt/ | Letta API key or self-hosted runtime; pinned SDK such as `@letta-ai/letta-client` or `letta-client`; fixed agent/tools/model/memory config; operator-supplied dataset mapping. | This harness has no Letta credentials, SDK installation, runtime, fixed agent loop, or approved external dataset. |
+| LangGraph memory | https://docs.langchain.com/oss/python/langgraph/memory | Pinned LangGraph runtime; short-term checkpointer config; namespaced long-term store config; fixed graph/model/tools/dataset mapping. | This Node local package harness does not execute a LangGraph runtime, checkpointer, store, graph, model, or adapter. |
+| Zep | https://help.getzep.com/ | Zep credentials or endpoint; pinned client; temporal Context Graph or Context Lake config; ingestion and retrieval mapping. | This harness has no Zep credentials, client package, Context Graph/Context Lake runtime, ingestion job, or retrieval dataset. |
+| Mem0 | https://docs.mem0.ai/ | Mem0 platform credentials or open-source runtime; pinned SDK/package versions; extraction/update/retrieval/model/tool config; dataset mapping. | This harness has no Mem0 credentials, runtime, configured memory loop, model/tool environment, or external dataset adapter. |
+| OpenAI ChatGPT native memory | https://help.openai.com/en/articles/8590148-memory-faq | ChatGPT account/runtime with native memory enabled; account-safe evaluation protocol; dataset prompts; evidence capture that excludes personal data and credentials. | ChatGPT native memory is a consumer-app feature rather than a public API surface available to this local package harness. |
+| Claude memory tool | https://support.anthropic.com/en/articles/11145838-using-claude-memory | Claude/provider runtime with the memory tool available; client-side tool configuration; fixed model/tool-use policy/prompts/dataset mapping; safe evidence capture. | The memory tool is provider/client-side and requires a Claude runtime plus tool environment that this benchmark does not control. |
+
+No external adapter row contains recall, abstention, token, latency, or ranking scores. Third-party claims stay blocked until the required credentials, runtimes, and datasets are supplied and reviewed.
 
 ## Claim limits
 
 The benchmark report is evidence for this local deterministic fixture only. It is not provider deletion proof, model forgetting proof, compliance certification, ROI evidence, provider invoice savings evidence, benchmark leadership proof, hosted cloud readiness, or a substitute for external LoCoMo/LongMemEval evaluation.
 
-Cross-provider rows are profile labels using the same Enigma context-pack boundary. They do not call or rank live provider models.
+Cross-provider rows are profile labels using the same Enigma context-pack boundary. External competitor rows are adapter requirements only; they do not call, score, or rank live provider models.
+
+`public_claims_allowed` is intentionally narrow: deterministic local fixture execution, local recall/abstention metrics, local baseline comparison, local token estimates, duplicate removal where applicable, p50/p95 local latency, Enigma bundle/context-pack verification, and explicit withholding of third-party scores until the required external artifacts exist.
 
 ## Extending with external datasets later
 

package/docs/sdk-api.md

@@ -1,6 +1,6 @@
 # SDK and API guide
 
-This guide covers the public package imports for `enigma-memory@0.1.2`. The SDK runs locally by default: vaults, passports, context packs, receipts, relay/gateway demo state, storage contracts, metering artifacts, and settlement artifacts are package-level developer surfaces. They are not evidence of hosted Enigma cloud, provider-side deletion, provider model forgetting, token ROI, invoice savings, compliance certification, or benchmark leadership.
+This guide covers the public package imports for `enigma-memory@0.1.3`. The SDK runs locally by default: vaults, passports, context packs, receipts, relay/gateway demo state, storage contracts, metering artifacts, and settlement artifacts are package-level developer surfaces. They are not evidence of hosted Enigma cloud, provider-side deletion, provider model forgetting, token ROI, invoice savings, compliance certification, or benchmark leadership.
 
 ## Install and import style
 

package/examples/ci/github-actions.yml

@@ -22,7 +22,18 @@ jobs:
         run: npm init -y
 
       - name: Install Enigma Memory
-        run: npm install enigma-memory@0.1.1
+        run: npm install enigma-memory@0.1.3
+
+      - name: Expose local benchmark script
+        run: |
+          node --input-type=module <<'JS'
+          import { readFile, writeFile } from 'node:fs/promises';
+
+          const pkg = JSON.parse(await readFile('package.json', 'utf8'));
+          pkg.scripts ??= {};
+          pkg.scripts['benchmark:memory-suite'] = 'node ./node_modules/enigma-memory/scripts/run-memory-benchmarks.mjs';
+          await writeFile('package.json', `${JSON.stringify(pkg, null, 2)}\n`, 'utf8');
+          JS
 
       - name: Run Enigma quickstart
         run: npx enigma quickstart --overwrite
@@ -30,6 +41,23 @@ jobs:
       - name: Run Enigma doctor
         run: npx enigma doctor
 
+      - name: Run local memory benchmark
+        run: |
+          npm run benchmark:memory-suite -- --out benchmark-report.json
+          mkdir -p artifacts
+          cp benchmark-report.json artifacts/benchmark-report.json
+          echo "Saved public-safe benchmark report placeholder at artifacts/benchmark-report.json"
+
+      # Optional artifact upload placeholder:
+      # If your repository already permits actions/upload-artifact, uncomment a
+      # reviewed upload step like this. The example keeps the report saved in the
+      # workspace without requiring that action.
+      # - name: Upload benchmark report
+      #   uses: actions/upload-artifact@v4
+      #   with:
+      #     name: enigma-memory-benchmark-report
+      #     path: artifacts/benchmark-report.json
+
       - name: Smoke test package imports
         run: |
           node --input-type=module <<'JS'

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "enigma-memory",
-  "version": "0.1.2",
+  "version": "0.1.3",
   "type": "module",
   "description": "Provider-agnostic AI memory passport and offline-verifiable proof layer.",
   "license": "Apache-2.0",
@@ -112,6 +112,7 @@
     "docs/sdk-api.md",
     "docs/developer-ecosystem.md",
     "docs/memory-benchmarks.md",
+    "docs/benchmark-reproducibility.md",
     "docs/installers-and-desktop.md",
     "docs/browser-extension-install.md",
     "docs/hosted-cloud-product.md",

package/packages/mcp-server/src/index.js

@@ -17,7 +17,7 @@ import {
 const DEFAULT_BUNDLE = '.enigma/bundle.json';
 const JSONRPC_VERSION = '2.0';
 const MCP_PROTOCOL_VERSION = '2024-11-05';
-const SERVER_INFO = Object.freeze({ name: 'enigma-mcp-server', version: '0.1.2' });
+const SERVER_INFO = Object.freeze({ name: 'enigma-mcp-server', version: '0.1.3' });
 const JSON_RPC_ID_PATTERN = /^[A-Za-z0-9._:-]{1,128}$/;
 const JSON_RPC_ERROR = Object.freeze({
   INVALID_REQUEST: -32600,

package/scripts/build-installer-assets.mjs

@@ -6,7 +6,7 @@ import { fileURLToPath } from 'node:url';
 
 export const INSTALLER_ASSET_SCHEMA = 'enigma.installer_assets.v1';
 export const INSTALLER_ASSET_PACKAGE = 'enigma-memory';
-export const INSTALLER_ASSET_VERSION = '0.1.2';
+export const INSTALLER_ASSET_VERSION = '0.1.3';
 export const INSTALLER_ASSET_GENERATED_AT = '1970-01-01T00:00:00.000Z';
 
 const SCRIPT_PATH = fileURLToPath(import.meta.url);

package/scripts/run-memory-benchmarks.mjs

@@ -16,7 +16,7 @@ import {
   compileContextPack,
   verifyContextPack,
 } from '../packages/passport/src/index.js';
-import { createMemoryOptimizationPlan } from '../packages/optimizer/src/index.js';
+import { createMemoryOptimizationPlan, estimateTextTokens } from '../packages/optimizer/src/index.js';
 import { verifyBundle } from '../apps/verifier/bin/enigma-verify.mjs';
 
 export const MEMORY_BENCHMARK_SUITE_SCHEMA = 'enigma.memory_benchmark_suite.v1';
@@ -34,6 +34,131 @@ const CONTEXT_BOUNDARY = Object.freeze({
 
 const PROVIDER_PROFILES = Object.freeze(['chatgpt', 'claude', 'kimi', 'cursor', 'local-llm']);
 
+const LOCAL_BASELINES = Object.freeze([
+  {
+    id: 'full_context',
+    label: 'Full active context',
+    boundary: 'All active fixture memories are supplied without optimization or deduplication.',
+  },
+  {
+    id: 'recency_last_n',
+    label: 'Recency last N',
+    boundary: 'The three most recently updated active fixture memories are supplied.',
+  },
+  {
+    id: 'keyword_filter',
+    label: 'Keyword filter',
+    boundary: 'Active fixture memories are supplied when deterministic query terms match content or tags.',
+  },
+  {
+    id: 'enigma_context_pack',
+    label: 'Enigma context pack',
+    boundary: 'The Enigma passport context-pack compiler and optimizer boundary are used locally.',
+  },
+]);
+
+const EXTERNAL_COMPETITOR_ADAPTERS = Object.freeze([
+  {
+    id: 'letta_memgpt',
+    name: 'Letta / MemGPT',
+    status: 'not_run_requires_credentials_or_runtime',
+    required_artifacts: [
+      'Letta API key or self-hosted Letta runtime',
+      'Pinned Letta SDK package and version such as @letta-ai/letta-client or letta-client',
+      'Agent, tools, model, and memory configuration',
+      'Operator-supplied benchmark dataset or fixture mapping',
+    ],
+    official_doc: 'https://docs.letta.com/concepts/memgpt/',
+    can_run_in_this_harness: false,
+    boundary_reason: 'This local harness has no Letta credentials, SDK installation, hosted/self-hosted Letta runtime, fixed agent loop, or approved external dataset, so no Letta score is produced.',
+    scores_included: false,
+  },
+  {
+    id: 'langgraph_memory',
+    name: 'LangGraph memory',
+    status: 'not_run_requires_credentials_or_runtime',
+    required_artifacts: [
+      'Pinned LangGraph runtime and package versions',
+      'Checkpointer configuration for short-term memory',
+      'Namespaced long-term store configuration',
+      'Fixed graph, model, tools, and dataset mapping',
+    ],
+    official_doc: 'https://docs.langchain.com/oss/python/langgraph/memory',
+    can_run_in_this_harness: false,
+    boundary_reason: 'This Node local package harness does not install or execute a LangGraph runtime, checkpointer, namespaced store, graph, model, or dataset adapter, so no LangGraph score is produced.',
+    scores_included: false,
+  },
+  {
+    id: 'zep',
+    name: 'Zep',
+    status: 'not_run_requires_credentials_or_runtime',
+    required_artifacts: [
+      'Zep credentials or local/runtime endpoint',
+      'Pinned Zep client and version',
+      'Temporal Context Graph or Context Lake configuration',
+      'Dataset ingestion and retrieval mapping',
+    ],
+    official_doc: 'https://help.getzep.com/',
+    can_run_in_this_harness: false,
+    boundary_reason: 'This local harness has no Zep credentials, client package, Context Graph or Context Lake runtime, ingestion job, or provider-approved retrieval dataset, so no Zep score or latency claim is produced.',
+    scores_included: false,
+  },
+  {
+    id: 'mem0',
+    name: 'Mem0',
+    status: 'not_run_requires_credentials_or_runtime',
+    required_artifacts: [
+      'Mem0 platform credentials or open-source stack runtime',
+      'Pinned Mem0 SDK/package versions',
+      'Memory extraction, update, retrieval, model, and tool configuration',
+      'Dataset ingestion and scoring mapping',
+    ],
+    official_doc: 'https://docs.mem0.ai/',
+    can_run_in_this_harness: false,
+    boundary_reason: 'This local harness has no Mem0 credentials, SDK/runtime, configured extraction/retrieval loop, model/tool environment, or external dataset adapter, so no Mem0 score is produced.',
+    scores_included: false,
+  },
+  {
+    id: 'openai_native_memory',
+    name: 'OpenAI ChatGPT native memory',
+    status: 'not_run_requires_credentials_or_runtime',
+    required_artifacts: [
+      'Consumer ChatGPT account/runtime with native memory enabled',
+      'Account-safe evaluation protocol and export/review process',
+      'Dataset prompts and operator approval for non-public app interaction',
+      'Evidence capture that excludes personal data and credentials',
+    ],
+    official_doc: 'https://help.openai.com/en/articles/8590148-memory-faq',
+    can_run_in_this_harness: false,
+    boundary_reason: 'ChatGPT native memory is a consumer-app feature rather than a public API surface available to this local package harness, so no OpenAI native-memory score is produced.',
+    scores_included: false,
+  },
+  {
+    id: 'claude_memory_tool',
+    name: 'Claude memory tool',
+    status: 'not_run_requires_credentials_or_runtime',
+    required_artifacts: [
+      'Claude/provider runtime with the memory tool available',
+      'Client-side tool configuration and storage boundary',
+      'Fixed model, tool-use policy, prompts, and dataset mapping',
+      'Evidence capture that excludes personal data and credentials',
+    ],
+    official_doc: 'https://support.anthropic.com/en/articles/11145838-using-claude-memory',
+    can_run_in_this_harness: false,
+    boundary_reason: 'The Claude memory tool is provider/client-side and requires a Claude runtime plus tool environment that this local package benchmark does not control, so no Claude memory-tool score is produced.',
+    scores_included: false,
+  },
+]);
+
+const PUBLIC_CLAIMS_ALLOWED = Object.freeze([
+  'Runs a deterministic local Enigma memory fixture without external provider calls.',
+  'Reports local exact-answer recall and abstention correctness for the fixture questions.',
+  'Compares Enigma context packs with full-context, recency, and keyword local baselines on the same fixture.',
+  'Reports local estimated prompt tokens, duplicate-removal counts where applicable, and p50/p95 local latency.',
+  'Verifies Enigma-controlled bundles and context packs without exposing raw fixture memory, questions, or answers.',
+  'Lists external competitor adapter requirements and withholds third-party scores until credentials, runtimes, and datasets are supplied.',
+]);
+
 const BENCHMARK_CITATIONS = Object.freeze([
   {
     id: 'locomo',
@@ -402,6 +527,188 @@ function activeOptimizationPlan(vault, query) {
   });
 }
 
+function activeBenchmarkMemories(vault) {
+  const memories = [];
+  for (const memoryAddr of vault.activeAddresses ?? []) {
+    const record = vault.__getRecord(memoryAddr);
+    if (!record || record.state !== 'active') continue;
+    memories.push({
+      memory_addr: memoryAddr,
+      content: vault.__getPlaintext(memoryAddr),
+      updated_at: record.updated_at ?? record.created_at,
+      created_at: record.created_at,
+      purpose_tags: Array.isArray(record.purpose_tags) ? [...record.purpose_tags] : [],
+    });
+  }
+  return memories;
+}
+
+function estimatePromptTokens(query, memories) {
+  let tokens = estimateTextTokens(query);
+  for (const memory of memories) tokens += estimateTextTokens(memory.content);
+  return tokens;
+}
+
+function localLatencySummary(values) {
+  return {
+    samples: values.length,
+    p50_ms: Number(percentile(values, 0.5).toFixed(6)),
+    p95_ms: Number(percentile(values, 0.95).toFixed(6)),
+  };
+}
+
+function compareUpdatedDesc(left, right) {
+  const leftTime = Date.parse(left.updated_at ?? left.created_at ?? '');
+  const rightTime = Date.parse(right.updated_at ?? right.created_at ?? '');
+  if (leftTime !== rightTime) return rightTime - leftTime;
+  return String(left.memory_addr).localeCompare(String(right.memory_addr));
+}
+
+function queryTerms(query) {
+  const stopwords = new Set([
+    'after',
+    'should',
+    'the',
+    'use',
+    'what',
+    'which',
+    'who',
+  ]);
+  const terms = [];
+  for (const match of String(query).toLowerCase().matchAll(/[a-z0-9_-]{3,}/gu)) {
+    const term = match[0];
+    if (!stopwords.has(term)) terms.push(term);
+  }
+  return terms;
+}
+
+function keywordFilteredMemories(memories, query) {
+  const terms = queryTerms(query);
+  if (terms.length === 0) return [];
+  return memories.filter((memory) => {
+    const haystack = `${memory.content} ${(memory.purpose_tags ?? []).join(' ')}`.toLowerCase();
+    return terms.some((term) => haystack.includes(term));
+  });
+}
+
+function selectLocalBaselineMemories(baselineId, vault, passport, question) {
+  const memories = activeBenchmarkMemories(vault);
+  if (baselineId === 'full_context') {
+    return {
+      memories,
+      estimatedPromptTokens: estimatePromptTokens(question.query, memories),
+      duplicateCandidatesRemoved: 0,
+      duplicateRemovalApplicable: false,
+    };
+  }
+  if (baselineId === 'recency_last_n') {
+    const selected = [...memories].sort(compareUpdatedDesc).slice(0, 3);
+    return {
+      memories: selected,
+      estimatedPromptTokens: estimatePromptTokens(question.query, selected),
+      duplicateCandidatesRemoved: 0,
+      duplicateRemovalApplicable: false,
+    };
+  }
+  if (baselineId === 'keyword_filter') {
+    const selected = keywordFilteredMemories(memories, question.query);
+    return {
+      memories: selected,
+      estimatedPromptTokens: estimatePromptTokens(question.query, selected),
+      duplicateCandidatesRemoved: 0,
+      duplicateRemovalApplicable: false,
+    };
+  }
+  if (baselineId === 'enigma_context_pack') {
+    const fullPlan = activeOptimizationPlan(vault, question.query);
+    const pack = compileContextPack({
+      vault,
+      passport,
+      query: question.query,
+      context_pack_id: `ctx_benchmark_local_baseline_${question.id}`,
+      ...packOptions('local-llm'),
+    });
+    return {
+      memories: pack.memories,
+      estimatedPromptTokens: pack.optimization_plan?.optimized_prompt_tokens ?? estimatePromptTokens(question.query, pack.memories),
+      duplicateCandidatesRemoved: fullPlan.totals.duplicates_removed,
+      duplicateRemovalApplicable: true,
+    };
+  }
+  throw new Error(`unknown local baseline ${baselineId}`);
+}
+
+function compareLocalBaselines(vault, passport) {
+  const rows = [];
+  for (const baseline of LOCAL_BASELINES) {
+    let exactTotal = 0;
+    let exactCorrect = 0;
+    let abstainTotal = 0;
+    let abstainCorrect = 0;
+    let totalEstimatedPromptTokens = 0;
+    let selectedMemoryTotal = 0;
+    let maxDuplicateCandidatesRemoved = 0;
+    let totalDuplicateCandidatesRemoved = 0;
+    let duplicateRemovalApplicable = false;
+    const latencies = [];
+
+    for (const question of PRIVATE_FIXTURE.questions) {
+      const start = performance.now();
+      const selected = selectLocalBaselineMemories(baseline.id, vault, passport, question);
+      const result = chooseAnswer(question, { memories: selected.memories });
+      latencies.push(Math.max(0, performance.now() - start));
+
+      totalEstimatedPromptTokens += selected.estimatedPromptTokens;
+      selectedMemoryTotal += selected.memories.length;
+      maxDuplicateCandidatesRemoved = Math.max(maxDuplicateCandidatesRemoved, selected.duplicateCandidatesRemoved);
+      totalDuplicateCandidatesRemoved += selected.duplicateCandidatesRemoved;
+      duplicateRemovalApplicable = duplicateRemovalApplicable || selected.duplicateRemovalApplicable;
+
+      if (question.category === 'abstention') {
+        abstainTotal += 1;
+        if (result.abstained && result.correct) abstainCorrect += 1;
+      } else {
+        exactTotal += 1;
+        if (!result.abstained && result.correct) exactCorrect += 1;
+      }
+    }
+
+    rows.push({
+      id: baseline.id,
+      label: baseline.label,
+      boundary: baseline.boundary,
+      local_fixture_only: true,
+      external_provider_called: false,
+      deterministic_fixture: true,
+      question_count: PRIVATE_FIXTURE.questions.length,
+      exact_answer_questions: exactTotal,
+      exact_answer_correct: exactCorrect,
+      exact_answer_recall: exactTotal === 0 ? 0 : Number((exactCorrect / exactTotal).toFixed(6)),
+      abstention_questions: abstainTotal,
+      abstention_correct: abstainCorrect,
+      abstention_correctness: abstainTotal === 0 ? 0 : Number((abstainCorrect / abstainTotal).toFixed(6)),
+      estimated_prompt_tokens: {
+        total: totalEstimatedPromptTokens,
+        mean_per_question: Number((totalEstimatedPromptTokens / PRIVATE_FIXTURE.questions.length).toFixed(6)),
+        estimator: 'estimateTextTokens deterministic local estimator',
+      },
+      selected_memory_count: {
+        total: selectedMemoryTotal,
+        mean_per_question: Number((selectedMemoryTotal / PRIVATE_FIXTURE.questions.length).toFixed(6)),
+      },
+      duplicate_removal: {
+        applicable: duplicateRemovalApplicable,
+        max_duplicate_candidates_removed: maxDuplicateCandidatesRemoved,
+        total_duplicate_candidates_removed: totalDuplicateCandidatesRemoved,
+      },
+      latency: localLatencySummary(latencies),
+      public_question_text_included: false,
+      public_answer_text_included: false,
+    });
+  }
+  return rows;
+}
+
 function summarizeContextReduction(plan) {
   const baseline = plan?.baseline_prompt_tokens ?? 0;
   const optimized = plan?.optimized_prompt_tokens ?? 0;
@@ -489,6 +796,7 @@ export function runMemoryBenchmarkSuite(options = {}) {
   const passport = createPassport({ vault: importedVault, now: generatedAt });
 
   const scored = scoreQuestions(importedVault, passport, samples);
+  const localBaselineRows = compareLocalBaselines(importedVault, passport);
   const fullOptimizationPlan = activeOptimizationPlan(importedVault, 'Use the benchmark memory boundary for recall and abstention evaluation.');
   const providerRows = compareProviders(importedVault, passport, samples);
   const verificationBundleResults = [];
@@ -522,7 +830,7 @@ export function runMemoryBenchmarkSuite(options = {}) {
       benchmark_leadership_claim: false,
       claim_boundary: [
         'This suite measures deterministic local Enigma fixture operations only.',
-        'Cross-provider rows use profile labels and the same Enigma context-pack boundary; they do not call or compare live provider models.',
+        'Cross-provider profile rows remain same-boundary Enigma labels; external competitor adapter rows are requirements only and contain no third-party scores.',
         'Token reduction is a local estimator result against this fixture, not a provider invoice, ROI, guaranteed savings, or benchmark-leadership claim.',
         'Verification proves Enigma-controlled receipts/bundles/context packs only; it is not provider deletion, provider forgetting, model forgetting, or compliance certification evidence.',
       ],
@@ -536,6 +844,7 @@ export function runMemoryBenchmarkSuite(options = {}) {
       optimizer_plan_token_estimates: true,
       bundle_verification: true,
       context_pack_verification: true,
+      local_baseline_comparison: true,
       latency_clock: 'performance.now',
     },
     metrics: {
@@ -546,6 +855,7 @@ export function runMemoryBenchmarkSuite(options = {}) {
         deduped_candidates: fullOptimizationPlan.totals.deduped_candidates,
         duplicate_candidates_removed: fullOptimizationPlan.totals.duplicates_removed,
       },
+      local_baseline_comparisons: localBaselineRows,
       latency: latencySummary(samples),
       verification: {
         bundle_verify_runs: verificationBundleResults.length,
@@ -554,6 +864,8 @@ export function runMemoryBenchmarkSuite(options = {}) {
       },
     },
     cross_provider_profiles: providerRows,
+    external_competitor_adapters: EXTERNAL_COMPETITOR_ADAPTERS,
+    public_claims_allowed: PUBLIC_CLAIMS_ALLOWED,
   };
 
   return assertNoRawFixtureLeak(report);