enigma-memory 0.1.3 → 0.1.5

package/docs/benchmark-reproducibility.md

@@ -1,25 +1,70 @@
 # Benchmark reproducibility
 
-This guide explains how to reproduce the current local Enigma memory benchmark, save the public-safe JSON report, cite the external benchmark standards it is modeled after, and understand what is still required before publishing live third-party comparisons.
+This guide explains how to reproduce the current local Enigma memory benchmark, run official-dataset retrieval/evidence proxy benchmarks against LoCoMo and LongMemEval inputs, save public-safe JSON reports, cite source datasets honestly, and understand what is still required before publishing live LLM answer-accuracy or competitor comparisons.
 
 ## What is reproducible today
 
-The current package is `enigma-memory@0.1.3`. The local benchmark is available through the package script and the script file it wraps:
+The current package is `enigma-memory@0.1.4`. Two benchmark paths are reproducible without provider credentials:
+
+1. The local deterministic memory suite, available through the package script and the script file it wraps:
+
+   ```sh
+   cd enigma
+   npm run benchmark:memory-suite
+   npm run benchmark:memory-suite -- --out benchmark-report.json
+   node scripts/run-memory-benchmarks.mjs --out benchmark-report.json
+   ```
+
+   The `--out` form writes the report to the requested path and prints only a small status object. Without `--out`, the command writes the full JSON report to stdout. The report schema is `enigma.memory_benchmark_suite.v1`.
+
+2. The official-dataset standard runner, which consumes locally downloaded LoCoMo and/or LongMemEval JSON files:
+
+   ```sh
+   node scripts/run-standard-memory-benchmarks.mjs --locomo .enigma/benchmarks/datasets/locomo10.json --longmemeval .enigma/benchmarks/datasets/longmemeval_s_cleaned.json --max-locomo-qa 25 --max-longmemeval-items 25 --top-k 5 --out .enigma/standard-memory-benchmark-sample.json
+   ```
+
+   The standard report schema is `enigma.standard_memory_benchmark_suite.v1`. It scores retrieval/evidence coverage over official dataset records with local deterministic methods only. It does not call LLM providers, generate final answers, grade natural-language answer correctness, call competitor SDKs, or create provider/competitor scores.
+
+   In that standard report, `keyword_filter` is intentionally the simpler lexical baseline. `enigma_relevance` is the deterministic Enigma retrieval approximation: it uses deterministic query expansion, term normalization and stemming, task/category and temporal/date hints, role/session metadata, phrase/proximity scoring, and final reranking for evidence diversity. It does not use raw answer text, evidence labels, or `has_answer` flags to choose records. It must be interpreted only as retrieval/evidence proxy scoring over the local dataset file named in the report, not as LLM answer accuracy, provider performance, competitor performance, or leaderboard standing.
+
+   | Standard-runner row | Retrieval boundary |
+   | --- | --- |
+   | `full_context` | Scores every parsed local memory record for the dataset item without retrieval filtering. |
+   | `recency_last_n` | Scores the most recent parsed records as a deterministic recency baseline. |
+   | `keyword_filter` | Scores direct normalized query/content term overlap only. |
+   | `enigma_relevance` | Scores deterministic Enigma-style retrieval signals before `--top-k`: query expansion, stemming, role/session metadata, temporal hints, phrase/proximity matches, and evidence-diversity reranking. |
+
+Both report families are designed to be public-safe: they contain aggregate metrics, commitments, citations, profile labels, source metadata, and claim boundaries. They do not include raw fixture memory, raw dataset conversation text, private question text, private answer text, provider transcripts, credentials, account ids, or local absolute paths.
+
+## Official dataset download runbook
+
+Use `scripts/download-standard-benchmarks.mjs` to stage official LoCoMo and LongMemEval files for future standard benchmark runs without adding raw data to the repository. The default mode is a public-safe dry run:
 
 ```sh
 cd enigma
-npm run benchmark:memory-suite
-npm run benchmark:memory-suite -- --out benchmark-report.json
-node scripts/run-memory-benchmarks.mjs --out benchmark-report.json
+node scripts/download-standard-benchmarks.mjs --dry-run
 ```
 
-The `--out` form writes the report to the requested path and prints only a small status object. Without `--out`, the command writes the full JSON report to stdout.
+The dry-run output lists planned fetches only: dataset ids, source URLs, licenses or upstream license-review notes, usage boundaries, expected output files under `.enigma/benchmarks/datasets`, and the manifest path. It does not fetch data, print raw dataset snippets, include credentials, or emit local absolute paths when the default relative paths are used.
+
+To download all supported datasets and capture hashes/sizes, opt in explicitly:
+
+```sh
+node scripts/download-standard-benchmarks.mjs --execute --dataset all --out-dir .enigma/benchmarks/datasets --manifest .enigma/benchmarks/dataset-manifest.json
+```
+
+For a single dataset, use `--dataset locomo`, `--dataset longmemeval-oracle`, `--dataset longmemeval-s`, or `--dataset longmemeval-m`. The manifest schema is `enigma.standard_benchmark_dataset_manifest.v1`; it records source URLs, output file names, byte sizes, SHA-256 hashes, licenses/usage boundaries, and `raw_dataset_content_included: false`.
+
+Expected official source facts:
 
-The report schema is `enigma.memory_benchmark_suite.v1`. It is designed to be public-safe: it contains aggregate metrics, commitments, citations, cross-provider profile labels, and claim boundaries. It does not include raw fixture memory, private question text, private answer text, provider transcripts, credentials, account ids, or local absolute paths.
+- LoCoMo data source: `https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json`; license: CC BY-NC 4.0.
+- LongMemEval cleaned source files: `https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_oracle.json`, `https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json`, and `https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_m_cleaned.json`. LongMemEval covers information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.
 
-## Reproduce and save JSON
+Do not commit downloaded files or raw benchmark conversations. The package `.gitignore` excludes `.enigma/`; keep `.enigma/benchmarks/datasets` and the manifest as local/review artifacts unless a separate publication review approves what can be shared. LoCoMo is licensed CC BY-NC 4.0. LongMemEval cleaned files are hosted by the upstream Hugging Face dataset/repository; review the upstream terms before use or redistribution. Downloading these files enables retrieval/evidence-coverage or other reviewed benchmark scoring, not provider deletion proof, model forgetting proof, ROI/savings claims, compliance certification, live competitor scores, or benchmark-leadership claims.
 
-1. Use a clean checkout containing `enigma-memory@0.1.3`.
+## Reproduce and save local fixture JSON
+
+1. Use a clean checkout containing `enigma-memory@0.1.4`.
 2. From a repository root that contains `enigma/package.json`, enter the package directory:
 
    ```sh
@@ -40,10 +85,48 @@ The report schema is `enigma.memory_benchmark_suite.v1`. It is designed to be pu
    npm run benchmark:memory-suite -- --out benchmark-report.json
    ```
 
-5. Preserve the JSON file with the command, package version, operating system/runtime, and review context that produced it.
+5. Preserve the JSON file with the command, package version, operating system/runtime, hardware class when relevant, and review context that produced it.
 6. When sharing the result publicly, share the generated JSON report only after confirming it still has `public_safe: true` and `schema: "enigma.memory_benchmark_suite.v1"`.
 
-The local fixture measures Enigma-controlled operations only: vault remember/update, vault export/import, passport context-pack retrieval, optimizer token estimates and duplicate removal, bundle/context-pack verification, abstention behavior, exact-answer recall over the deterministic fixture, and p50/p95 operation latency from `performance.now`.
+The local fixture measures Enigma-controlled operations only: vault remember/update, vault export/import, passport context-pack retrieval, deterministic local relevance filtering before optimizer tiering, optimizer token estimates and duplicate removal, bundle/context-pack verification, abstention behavior, exact-answer recall over the deterministic fixture, and p50/p95 operation latency from `performance.now`.
+
+Interpret improvements as local fixture behavior. Enigma reduces context-pack estimated prompt tokens by selecting the deterministic query/purpose/address-relevant local memories before optimizer tiering and deduplication; it does not measure provider invoice savings, token ROI, live model quality, or third-party memory superiority. Token estimates and p50/p95 timings can change across hardware, Node/runtime versions, script revisions, and fixture updates.
+
+## Run the official-dataset standard benchmark
+
+The standard runner reads local dataset files produced by the downloader and writes a public-safe proxy report to the path supplied with `--out`. A bounded sample is the safest first run:
+
+```sh
+node scripts/run-standard-memory-benchmarks.mjs --locomo .enigma/benchmarks/datasets/locomo10.json --longmemeval .enigma/benchmarks/datasets/longmemeval_s_cleaned.json --max-locomo-qa 25 --max-longmemeval-items 25 --top-k 5 --out .enigma/standard-memory-benchmark-sample.json
+```
+
+Useful runner options:
+
+- `--locomo <path>` supplies a local LoCoMo JSON file.
+- `--longmemeval <path>` supplies a local LongMemEval JSON file. Use one cleaned split per run when you want split-specific evidence.
+- `--max-locomo-qa <n>` and `--max-longmemeval-items <n>` bound the sample size.
+- `--top-k <n>` controls retrieval depth; the default is `5`.
+- `--out <path>` writes public-safe JSON to that path. Without `--out`, the report is printed to stdout.
+
+If only `--locomo` or only `--longmemeval` is supplied, the runner scores only that dataset.
+
+For a full local proxy run, remove the sample caps:
+
+```sh
+node scripts/run-standard-memory-benchmarks.mjs --locomo .enigma/benchmarks/datasets/locomo10.json --longmemeval .enigma/benchmarks/datasets/longmemeval_s_cleaned.json --top-k 5 --out .enigma/standard-memory-benchmark.json
+```
+
+Full runs may take materially longer, may produce larger JSON reports, and may change with Node/runtime, hardware, script revision, dataset split, retrieval depth, and any future parsing fixes. They still remain retrieval/evidence proxy runs: no LLM answer generation, no provider APIs, no hosted memory services, and no competitor adapters are exercised. If the generated report shows `enigma_relevance` ahead of `keyword_filter`, describe the improvement as a local deterministic retrieval/evidence proxy result produced by that report, not as a hard-coded final score or any provider/model/competitor claim.
+
+When preserving or publishing official-dataset benchmark artifacts, keep the benchmark report and dataset manifest together. The report path is chosen with `--out`; the dataset hash/size capture is the manifest path passed to `--manifest`, usually `.enigma/benchmarks/dataset-manifest.json`.
+
+Public sharing should include the generated benchmark report JSON and generated dataset manifest JSON, not raw dataset files or raw conversations. Before publishing generated JSON, verify:
+
+1. The report schema is `enigma.standard_memory_benchmark_suite.v1`.
+2. The report does not contain raw conversation text, raw questions, raw answers, secrets, provider transcripts, account ids, or local absolute paths.
+3. The companion manifest schema is `enigma.standard_benchmark_dataset_manifest.v1`.
+4. The manifest includes source URLs, byte sizes, SHA-256 hashes, license/usage boundaries, and local file names for the exact dataset files used.
+5. Any public claim says "retrieval/evidence coverage proxy", quotes scores only from the generated report for the exact dataset hash/top-k/sample bounds, and avoids provider/model/competitor implications unless a separate reviewed provider answer-accuracy run exists.
 
 ## Local baseline rows in the report
 
@@ -54,40 +137,54 @@ The report now includes `metrics.local_baseline_comparisons`, which compares det
 | `full_context` | Supplies every active fixture memory without optimization or deduplication. |
 | `recency_last_n` | Supplies the three most recently updated active fixture memories. |
 | `keyword_filter` | Supplies active fixture memories whose content or tags match deterministic query terms. |
-| `enigma_context_pack` | Uses the Enigma passport context-pack compiler and optimizer boundary. |
+| `enigma_context_pack` | Uses the Enigma passport context-pack compiler with deterministic local relevance filtering before optimizer tiering and deduplication. |
 
 The report also includes `public_claims_allowed`; keep public copy within those local-fixture boundaries unless separate reviewed external evidence exists.
 
 ## How to cite external benchmark standards
 
-Use these standards as citations and task-category references, not as claimed Enigma results unless the exact external benchmark has been run and reviewed:
+Use these standards as dataset sources, citations, and task-category references, not as claimed Enigma leaderboard-equivalent results unless the exact external benchmark, scoring setup, and source-data hashes have been run and reviewed:
+
+- LoCoMo: https://snap-research.github.io/locomo/ and `https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json` — cite for long-term conversational-memory QA, event summarization, and multimodal generation over long conversations. The LoCoMo dataset license is CC BY-NC 4.0.
+- LongMemEval: https://arxiv.org/abs/2410.10813 and cleaned HuggingFace JSON files `https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_oracle.json`, `https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json`, and `https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_m_cleaned.json` — cite for information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.
+
+The local report mirrors some task categories from those benchmarks but does not download or score official records. The standard runner consumes official local dataset files and scores retrieval/evidence coverage; it does not run the original papers' full LLM evaluation pipelines or claim leaderboard-equivalent answer accuracy.
+
+## Future provider answer-accuracy runs
+
+A real answer-accuracy run is a different benchmark from the current standard runner. It would need all of the following before any answer-correctness or model-quality claim is published:
 
-- LoCoMo: https://snap-research.github.io/locomo/ — cite for long-term conversational-memory QA, event summarization, and multimodal generation over long conversations.
-- LongMemEval: https://arxiv.org/abs/2410.10813 — cite for information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.
+1. Provider API keys supplied at run time through reviewed environment names only, with secret values never printed, persisted, or copied into reports.
+2. Frozen model ids for generator and, if used, evaluator models. Model aliases are not enough for reproducibility.
+3. Budget caps before execution: maximum records, maximum generated tokens, maximum retries, timeout policy, and maximum provider spend.
+4. Frozen prompts for memory ingestion, retrieval, answer generation, abstention, evaluator grading, and any tool-use instructions.
+5. A fixed evaluator choice: exact-match/structured checks where the dataset supports them, human review where required, or a separately versioned LLM-as-judge prompt/model with known limitations.
+6. Dataset manifest hashes, split names, record counts, source licenses, and any excluded-record policy.
+7. Raw provider inputs/outputs retained only in private reviewed storage when license and policy permit; public reports should expose safe aggregates and hashes, not raw conversations.
 
-The current local report mirrors some task categories from those benchmarks, but it does not download LoCoMo or LongMemEval data, run their official evaluation pipelines, or claim leaderboard-equivalent results.
+The current standard runner is intentionally retrieval/evidence proxy only because it can run without provider keys, prompt variance, evaluator-model drift, provider billing risk, or provider transcript handling. It can say whether the local retrieval/evidence path surfaced expected supporting material, including whether deterministic Enigma relevance outperformed the simpler keyword row in the generated report. It cannot say whether an LLM would answer correctly, abstain correctly, forget something, comply with a deletion request, or outperform a provider/native memory product.
 
 ## Why live third-party comparisons are not claimed yet
 
-The current benchmark does not call external provider APIs, external SDKs, hosted memory services, ChatGPT native memory, Claude memory tooling, or third-party agent loops. Cross-provider rows in the report are profile labels that reuse the same Enigma context-pack boundary; they do not call or compare live provider models and are not live provider rankings.
+The current benchmarks do not call external provider APIs, external SDKs, hosted memory services, ChatGPT native memory, Claude memory tooling, or third-party agent loops. Cross-provider rows in the local report are profile labels that reuse the same Enigma context-pack boundary; they do not call or compare live provider models and are not live provider rankings. The official-dataset standard report is likewise local retrieval/evidence scoring only.
 
-Real comparisons require fixed adapters, fixed datasets, fixed agent/tool loops, explicit provider terms review, and reviewed handling of secrets and raw benchmark data. Memory quality can change with the surrounding agent framework and tool loop, so a fair comparison must document more than the memory store.
+Real comparisons require fixed adapters, fixed datasets, fixed agent/tool loops, explicit provider terms review, reviewed handling of secrets and raw benchmark data, and a no-score-without-run rule. Memory quality can change with the surrounding agent framework and tool loop, so a fair comparison must document more than the memory store. Until those inputs exist and the adapter is actually run in the same harness, external competitor rows stay requirements-only and must not carry recall, abstention, token, latency, answer-accuracy, cost, or ranking scores.
 
-The current report must not be used as evidence of provider-side deletion, model forgetting, compliance certification, token ROI, provider invoice savings, benchmark leadership, hosted-cloud readiness, or “best in world” superiority.
+The current reports must not be used as evidence of provider-side deletion, model forgetting, compliance certification, token ROI, provider invoice savings, benchmark leadership, hosted-cloud readiness, or “best in world” superiority.
 
-## External comparison requirements
+## Competitor comparison plan and no-score-without-run rule
 
-Use placeholder environment names only. Do not commit real tokens, API keys, account ids, provider transcripts, raw benchmark conversations, or private memory.
+Use placeholder environment names only. Do not commit real tokens, API keys, account ids, provider transcripts, raw benchmark conversations, raw provider answers, or private memory.
 
-The report field `external_competitor_adapters` is a requirements matrix, not a score table. External rows are expected to remain requirements-only until credentials, runtimes, and datasets are supplied and reviewed: `can_run_in_this_harness: false`, `scores_included: false`, and no recall, abstention, token, latency, or ranking scores.
+The report field `external_competitor_adapters` is a requirements matrix, not a score table. External rows are expected to remain requirements-only until credentials, runtimes, datasets, fixed prompts, fixed model ids, budget caps, reset policies, and scoring code are supplied and reviewed. A competitor row must have `can_run_in_this_harness: false`, `scores_included: false`, and no recall, abstention, token, latency, answer-accuracy, cost, or ranking score unless that exact adapter was run over the same dataset manifest in the same harness.
 
 | Target | Runtime or SDK needed | Placeholder secrets and local inputs | Dataset requirement | Adapter boundary before results can be claimed |
 | --- | --- | --- | --- | --- |
-| Letta | Letta SDK/runtime; documented SDK packages include `@letta-ai/letta-client` and `letta-client`; API-key-backed service access may be required. | `LETTA_API_KEY`, `LETTA_BASE_URL`, `LETTA_PROJECT_ID`, `BENCHMARK_DATASET_PATH` | Local reviewed LoCoMo/LongMemEval split or another reviewed local dataset file with license, version, split, and checksum metadata. | Build a Letta adapter that fixes the agent loop, memory write/read policy, model settings, and scoring path. Results may describe that configured Letta run only, not generic provider deletion or model forgetting. |
-| LangGraph memory | LangGraph runtime with short-term checkpointer memory and long-term namespaced store. | `LANGGRAPH_CHECKPOINTER_URI`, `LANGGRAPH_STORE_URI`, `BENCHMARK_DATASET_PATH` | Same local dataset file and split used for Enigma and every competitor. | Fix graph topology, checkpoint scope, namespace policy, retrieval policy, model/tool loop, and scorer. Do not attribute graph/tool behavior solely to the memory store. |
+| Letta/MemGPT | Letta SDK/runtime and MemGPT-style memory agent configuration; documented SDK packages include `@letta-ai/letta-client` and `letta-client`; API-key-backed service access may be required. | `LETTA_API_KEY`, `LETTA_BASE_URL`, `LETTA_PROJECT_ID`, `BENCHMARK_DATASET_PATH` | Local reviewed LoCoMo/LongMemEval split or another reviewed local dataset file with license, version, split, and checksum metadata. | Build a Letta adapter that fixes the agent loop, memory write/read policy, model settings, and scoring path. Results may describe that configured Letta/MemGPT run only, not generic provider deletion or model forgetting. |
+| LangGraph | LangGraph runtime with short-term checkpointer memory and long-term namespaced store. | `LANGGRAPH_CHECKPOINTER_URI`, `LANGGRAPH_STORE_URI`, `BENCHMARK_DATASET_PATH` | Same local dataset file and split used for Enigma and every competitor. | Fix graph topology, checkpoint scope, namespace policy, retrieval policy, model/tool loop, and scorer. Do not attribute graph/tool behavior solely to the memory store. |
 | Zep | Zep service/runtime positioned around temporal Context Graph and Context Lake retrieval. | `ZEP_API_KEY`, `ZEP_PROJECT_ID`, `ZEP_BASE_URL`, `BENCHMARK_DATASET_PATH` | Same local dataset file and split; include source checksum and whether any provider-side graph state is reused or reset. | Build a Zep adapter that records ingest, session, retrieval, reset, and scoring policy. Zep’s sub-200ms retrieval positioning is a vendor/source fact, not an Enigma-measured claim until measured in the same harness. |
 | Mem0 | Mem0 platform or open-source stack; positioned as a universal self-improving memory layer. | `MEM0_API_KEY`, `MEM0_BASE_URL`, `MEM0_PROJECT_ID`, `BENCHMARK_DATASET_PATH` | Same local dataset file and split; record Mem0 deployment flavor/version. | Build a Mem0 adapter with fixed extraction, update, retrieval, reset, and scorer behavior. Self-improving or platform behavior must be bounded to the configured run. |
-| OpenAI native ChatGPT memory | ChatGPT consumer-app/native memory environment. It is not directly available through a public API in this harness. | No usable harness secret; `OPENAI_API_KEY` alone is not sufficient to exercise ChatGPT native memory. | No fair automated dataset run until an approved interface can load/reset/query native memory reproducibly. | Do not claim live native ChatGPT memory comparison from this repository. A future adapter would need an approved public interface, reproducible memory reset/load semantics, and provider-policy review. |
+| OpenAI native memory (ChatGPT memory) | ChatGPT consumer-app/native memory environment. It is not directly available through a public API in this harness. | No usable harness secret; `OPENAI_API_KEY` alone is not sufficient to exercise ChatGPT native memory. | No fair automated dataset run until an approved interface can load/reset/query native memory reproducibly. | Do not claim live native ChatGPT memory comparison from this repository. A future adapter would need an approved public interface, reproducible memory reset/load semantics, and provider-policy review. |
 | Claude memory tool | Client-side/provider-specific memory tool environment. | `CLAUDE_MEMORY_TOOL_CONFIG`, `ANTHROPIC_API_KEY`, `BENCHMARK_DATASET_PATH` | Same local dataset file and split, plus reviewed tool-state reset/export rules. | Build an adapter around the exact client/tool environment, not generic Claude model behavior. Results can only cover that configured memory-tool setup. |
 
 ## Source references for adapter planning
@@ -104,10 +201,11 @@ The report field `external_competitor_adapters` is a requirements matrix, not a
 Before publishing external comparison language, capture all of the following in the benchmark report or an adjacent reviewed evidence file:
 
 1. Package version, benchmark schema, command, timestamp, OS/runtime, and adapter version.
-2. Dataset name, source URL, license review status, local file checksum, split, and record count.
+2. Dataset name, source URL, license review status, local file checksum, split, record count, and manifest schema/hash.
 3. Secret names used as placeholders, with confirmation that no secret values are printed or persisted.
-4. Adapter configuration: SDK/runtime version, model where applicable, memory write/read policy, reset policy, context limits, retry policy, and scoring code.
-5. Per-target raw scoring inputs retained privately when license permits, with public reports limited to safe aggregates.
-6. Explicit boundaries separating memory-store behavior, agent-loop behavior, model behavior, provider-hosted state, and Enigma receipt verification.
+4. Adapter configuration: SDK/runtime version, model ids where applicable, memory write/read policy, reset policy, context limits, retry policy, budget caps, frozen prompts, and scoring code.
+5. Evaluator choice and version: exact deterministic scorer, human rubric/version, or LLM-as-judge model id and frozen prompt.
+6. Per-target raw scoring inputs retained privately when license permits, with public reports limited to safe aggregates and hashes.
+7. Explicit boundaries separating memory-store behavior, agent-loop behavior, model behavior, provider-hosted state, and Enigma receipt verification.
 
-Until that evidence exists, use only the local benchmark claim: Enigma can reproduce deterministic local memory-fixture operations and emit a public-safe `enigma.memory_benchmark_suite.v1` JSON report.
+Until that evidence exists, use only the supported benchmark claims: Enigma can reproduce deterministic local memory-fixture operations with `enigma.memory_benchmark_suite.v1`, and Enigma can run official-dataset retrieval/evidence proxy scoring with `enigma.standard_memory_benchmark_suite.v1` when the local dataset files and manifest are supplied.

package/docs/developer-ecosystem.md

@@ -25,7 +25,7 @@ The example app prints ids, counts, roots, and verification status only. It does
 
 ## CLI and CI loop
 
-The CI example installs Node 24, installs the published `enigma-memory@0.1.3` package, runs:
+The CI example installs Node 24, installs the published `enigma-memory@0.1.4` package, runs:
 
 ```sh
 npx enigma quickstart --overwrite
@@ -33,9 +33,19 @@ npx enigma doctor
 npm run benchmark:memory-suite -- --out benchmark-report.json
 ```
 
-and then runs a small ESM import smoke. It does not require GitHub secrets, cloud provider credentials, npm tokens, private bundles, or local path assumptions. The benchmark step writes a public-safe local JSON report using schema `enigma.memory_benchmark_suite.v1`; see the benchmark reproducibility guide for claim boundaries and the requirements for any future live third-party comparison.
+and then runs a small ESM import smoke. It does not require GitHub secrets, cloud provider credentials, npm tokens, private bundles, local path assumptions, or official dataset network downloads in normal CI. The benchmark step writes a public-safe local JSON report using schema `enigma.memory_benchmark_suite.v1`; see the benchmark reproducibility guide for claim boundaries and the requirements for any future live third-party comparison.
 
-Use the workflow as a template in a consumer repository. It is intentionally limited to install/import/doctor smoke coverage, local proof generation, and deterministic local benchmark evidence; it does not publish packages, deploy infrastructure, contact hosted Enigma cloud, or call external memory providers.
+The workflow also includes optional official-dataset benchmark preparation steps gated behind the manual `workflow_dispatch` input `run_standard_benchmark: true`. Normal `push` and `pull_request` runs skip them, so official dataset downloads are not required in normal CI. Enable the manual path only after the repository has reviewed network use and dataset-license handling:
+
+```sh
+node ./node_modules/enigma-memory/scripts/download-standard-benchmarks.mjs --dry-run
+node ./node_modules/enigma-memory/scripts/download-standard-benchmarks.mjs --execute --dataset all --out-dir .enigma/benchmarks/datasets --manifest .enigma/benchmarks/dataset-manifest.json
+node ./node_modules/enigma-memory/scripts/run-standard-memory-benchmarks.mjs --locomo .enigma/benchmarks/datasets/locomo10.json --longmemeval .enigma/benchmarks/datasets/longmemeval_s_cleaned.json --max-locomo-qa 25 --max-longmemeval-items 25 --top-k 5 --out .enigma/standard-memory-benchmark-sample.json
+```
+
+Those commands produce a dataset manifest with source URLs, byte sizes, and SHA-256 hashes plus a standard benchmark report using schema `enigma.standard_memory_benchmark_suite.v1`. The standard runner is retrieval/evidence proxy scoring only: it does not call providers, grade generated answers, or produce competitor scores.
+
+Use the workflow as a template in a consumer repository. It is intentionally limited to install/import/doctor smoke coverage, local proof generation, and deterministic local benchmark evidence by default; it does not publish packages, deploy infrastructure, contact hosted Enigma cloud, call external memory providers, or download official benchmark datasets unless you intentionally enable `run_standard_benchmark` for a manual workflow run.
 
 ## MCP client loop
 

package/docs/memory-benchmarks.md

@@ -8,15 +8,27 @@ node scripts/run-memory-benchmarks.mjs
 node scripts/run-memory-benchmarks.mjs --out ./.enigma/memory-benchmark.json
 ```
 
+For official-dataset retrieval/evidence proxy scoring over operator-downloaded files:
+
+```sh
+cd enigma
+node scripts/run-standard-memory-benchmarks.mjs --locomo ./data/locomo10.json --out ./.enigma/locomo-standard-memory-benchmark.json
+node scripts/run-standard-memory-benchmarks.mjs --longmemeval ./data/longmemeval_s_cleaned.json --top-k 5 --out ./.enigma/longmemeval-standard-memory-benchmark.json
+node scripts/run-standard-memory-benchmarks.mjs --locomo ./data/locomo10.json --longmemeval ./data/longmemeval_s_cleaned.json --max-locomo-qa 100 --max-longmemeval-items 100 --out ./.enigma/standard-memory-benchmark.json
+```
+
+The standard report schema is `enigma.standard_memory_benchmark_suite.v1`. It reads local official dataset JSON files only, scores deterministic retrieval/evidence coverage proxies, and still excludes raw question, answer, and conversation text from reports.
+
 The report schema is `enigma.memory_benchmark_suite.v1`. It is public-safe by design: aggregate metrics, commitments, citations, boundaries, and cross-provider profile labels are emitted, but raw fixture memory, question text, and answer text are not included.
 
 ## External standards and boundaries
 
 - LoCoMo is the relevant long-term conversational-memory standard for multi-session QA, event summarization, and multimodal generation over long conversations. See https://snap-research.github.io/locomo/.
 - LongMemEval is the relevant standard for information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. See https://arxiv.org/abs/2410.10813.
+- Official local inputs for the standard runner are LoCoMo `locomo10.json` from `https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json` and LongMemEval cleaned files `longmemeval_oracle.json`, `longmemeval_s_cleaned.json`, or `longmemeval_m_cleaned.json` from the upstream Hugging Face dataset repository.
 - Letta's benchmark discussion is a useful boundary reminder: measured memory quality depends on the agent/framework/tool loop as well as memory-store mechanics. See https://www.letta.com/blog/benchmarking-ai-agent-memory/.
 
-This repository harness does not download or run LoCoMo, LongMemEval, provider APIs, or third-party agents. It mirrors their task categories with a deterministic local fixture so Enigma can make narrow package claims about local vault operations, context-pack generation, optimizer token estimates, export/import, verification, deduplication, abstention behavior, and latency.
+The fixture harness does not download or run LoCoMo, LongMemEval, provider APIs, or third-party agents. The standard harness (`run-standard-memory-benchmarks.mjs`) can score operator-supplied local LoCoMo and LongMemEval JSON files without credentials, provider APIs, or third-party SDKs.
 
 ## What the harness measures
 
@@ -26,30 +38,55 @@ The harness measures local Enigma operations only:
 
 - vault remember/update operations;
 - vault export and import;
-- context-pack retrieval through the passport package;
-- optimizer plan token estimates and duplicate removal;
+- context-pack retrieval through the passport package, including deterministic local relevance filtering before optimizer tiering;
+- optimizer plan token estimates, tiering, and duplicate removal;
 - bundle and context-pack verification;
 - local baseline comparisons over the same deterministic fixture questions;
 - p50/p95 latency with `performance.now`.
 
 Reported metrics include exact-answer recall, abstention correctness, estimated prompt tokens, duplicate candidates removed where applicable, operation latency summaries, verification status, and same-boundary cross-provider profile rows.
 
+Enigma context-pack token improvements are achieved by local selection, not external-provider behavior: the passport compiler narrows active local memories to the deterministic query/purpose/address-relevant set before optimizer tiering and deduplication. This lowers estimated prompt tokens by excluding locally irrelevant fixture memories while preserving the same recall and abstention scoring boundary.
+
+Numeric token and latency values in generated reports are local fixture measurements. They can change with hardware, Node/runtime version, script version, and fixture contents, so copy should cite the command and report artifact rather than treating one run as a universal score.
+
+## Official dataset retrieval/proxy runner
+
+`scripts/run-standard-memory-benchmarks.mjs` is the dependency-free runner for official dataset files already present on disk. It supports `--locomo <path>`, `--longmemeval <path>`, `--max-locomo-qa <n>`, `--max-longmemeval-items <n>`, `--top-k <n>` (default `5`), and optional `--out <path>`. Supplying only one dataset path scores only that dataset.
+
+Reports include only the input file name plus the input SHA-256, not the full local path, so operator usernames or workstation directories are not persisted.
+
+The runner parses LoCoMo `conversation` session turns as memory records and maps evidence labels such as `D1:3` or `D8:6; D9:17` to dialog IDs. It parses LongMemEval `haystack_sessions` turns as memory records, uses `has_answer: true` turns plus `answer_session_ids` as gold evidence, and treats `_abs` question IDs as abstention cases.
+
+Rows are local methods only: `full_context`, `recency_last_n`, `keyword_filter`, and `enigma_relevance`. `keyword_filter` remains a simple deterministic lexical baseline: it selects records whose public-safe normalized terms overlap the query. `enigma_relevance` is the more production-like local Enigma approximation: it uses deterministic query expansion, term normalization and stemming, task/category and temporal/date hints, role/session metadata, phrase/proximity scoring, and final reranking for evidence diversity. It does not use LLMs, provider APIs, competitor SDKs, hosted services, raw answers, or `has_answer` evidence flags to select records.
+
+| Standard-runner row | Retrieval boundary |
+| --- | --- |
+| `full_context` | Scores every parsed local memory record for the dataset item without retrieval filtering. |
+| `recency_last_n` | Scores the most recent parsed records as a deterministic recency baseline. |
+| `keyword_filter` | Scores direct normalized query/content term overlap only, so it remains intentionally easy to audit. |
+| `enigma_relevance` | Scores deterministic Enigma-style retrieval signals before `--top-k`: query expansion, stemming, role/session metadata, temporal hints, phrase/proximity matches, and evidence-diversity reranking. |
+
+Because `enigma_relevance` can match normalized variants, session/role cues, temporal wording, nearby phrases, and diverse evidence-bearing turns that a direct keyword overlap can miss, the benchmark report may show it improving over `keyword_filter` on retrieval/evidence proxy metrics. Those results are whatever the generated report records for the operator-supplied files; do not hard-code unreviewed scores or restate them as LLM answer accuracy, provider quality, competitor ranking, or benchmark-leadership evidence.
+
+The standard runner reports retrieval/evidence proxy metrics: LoCoMo evidence-hit@k and exact evidence coverage; LongMemEval turn evidence-hit@k, session evidence-hit@k, exact coverage, and abstention correctness; plus estimated prompt tokens, selected memory counts, and local latency. These are not LLM-generated answer-accuracy scores and must not be described as provider, competitor, or benchmark-leadership results.
+
 ## Local baseline comparison
 
-`metrics.local_baseline_comparisons` compares Enigma against deterministic local baselines only. Every row scores the same private fixture questions and keeps raw memory, question text, and answer text out of the report.
+`local_baseline_comparisons` (also mirrored at `metrics.local_baseline_comparisons`) compares Enigma against deterministic local baselines only. Every row scores the same private fixture questions and keeps raw memory, question text, and answer text out of the report.
 
 | Row | Boundary | Reported fields |
 | --- | --- | --- |
 | `full_context` | Supplies every active fixture memory without optimization or deduplication. | Recall, abstention correctness, estimated prompt tokens, selected memory count, p50/p95 local latency. |
 | `recency_last_n` | Supplies the three most recently updated active fixture memories. | Same fields; duplicate removal is marked not applicable. |
 | `keyword_filter` | Supplies active fixture memories whose content or tags match deterministic query terms. | Same fields; duplicate removal is marked not applicable. |
-| `enigma_context_pack` | Uses the Enigma passport context-pack compiler and optimizer boundary. | Same fields plus duplicate-removal counts from the Enigma optimizer plan. |
+| `enigma_context_pack` | Uses the Enigma passport context-pack compiler with deterministic local relevance filtering before optimizer tiering and deduplication. | Same fields plus duplicate-removal counts from the Enigma optimizer plan. |
 
 These rows are local package evidence only. They do not compare hosted providers, do not use provider APIs, and do not support invoice savings, ROI, compliance, model-forgetting, or benchmark-leadership claims.
 
 ## External competitor adapter requirements
 
-`external_competitor_adapters` is a requirements matrix, not a score table. Each row has `status: "not_run_requires_credentials_or_runtime"`, `can_run_in_this_harness: false`, `required_artifacts`, `official_doc`, an exact `boundary_reason`, and `scores_included: false`.
+`external_competitor_adapters` is a requirements matrix, not a score table. Each row has `status: "not_run_requires_credentials_or_runtime"`, `can_run_in_this_harness: false`, `required_artifacts`, `official_doc`, `official_positioning`, an exact `boundary_reason`, and `scores_included: false`.
 
 | Adapter | Official source | Required artifacts before scoring | Boundary reason |
 | --- | --- | --- | --- |
@@ -60,23 +97,24 @@ These rows are local package evidence only. They do not compare hosted providers
 | OpenAI ChatGPT native memory | https://help.openai.com/en/articles/8590148-memory-faq | ChatGPT account/runtime with native memory enabled; account-safe evaluation protocol; dataset prompts; evidence capture that excludes personal data and credentials. | ChatGPT native memory is a consumer-app feature rather than a public API surface available to this local package harness. |
 | Claude memory tool | https://support.anthropic.com/en/articles/11145838-using-claude-memory | Claude/provider runtime with the memory tool available; client-side tool configuration; fixed model/tool-use policy/prompts/dataset mapping; safe evidence capture. | The memory tool is provider/client-side and requires a Claude runtime plus tool environment that this benchmark does not control. |
 
-No external adapter row contains recall, abstention, token, latency, or ranking scores. Third-party claims stay blocked until the required credentials, runtimes, and datasets are supplied and reviewed.
+No external adapter row contains recall, abstention, token, latency, or ranking scores. Third-party rows remain requirements-only until the required credentials, runtimes, fixed agent/tool loops, and reviewed datasets are supplied and reviewed.
+
+The `official_positioning` field records only source-attributed context needed to build a future adapter: Letta/MemGPT runtime and SDK/API-key requirements; LangGraph short-term checkpointer and long-term namespaced store memory; Zep temporal Context Graph/Context Lake positioning and retrieval-latency claim; Mem0 platform/open-source memory stack; OpenAI native consumer-app memory; and Claude provider/client-side memory tooling. None of those facts are scored or verified by this local run.
 
 ## Claim limits
 
-The benchmark report is evidence for this local deterministic fixture only. It is not provider deletion proof, model forgetting proof, compliance certification, ROI evidence, provider invoice savings evidence, benchmark leadership proof, hosted cloud readiness, or a substitute for external LoCoMo/LongMemEval evaluation.
+The fixture benchmark report is evidence for the local deterministic fixture only. The standard benchmark report is evidence for retrieval/evidence proxy scoring over the operator-supplied LoCoMo or LongMemEval file only. Reported `enigma_relevance` improvements mean the local deterministic retrieval method surfaced evidence more effectively than the simpler keyword row for that run's questions, top-k, parser, and dataset file; they are not provider deletion proof, model forgetting proof, compliance certification, ROI evidence, provider invoice savings evidence, benchmark leadership proof, hosted cloud readiness, or LLM answer-accuracy evidence.
 
 Cross-provider rows are profile labels using the same Enigma context-pack boundary. External competitor rows are adapter requirements only; they do not call, score, or rank live provider models.
 
 `public_claims_allowed` is intentionally narrow: deterministic local fixture execution, local recall/abstention metrics, local baseline comparison, local token estimates, duplicate removal where applicable, p50/p95 local latency, Enigma bundle/context-pack verification, and explicit withholding of third-party scores until the required external artifacts exist.
 
-## Extending with external datasets later
+## Running official datasets safely
 
-To extend this harness with real external benchmark datasets without weakening claim boundaries:
+To run real LoCoMo or LongMemEval retrieval/proxy scores without weakening claim boundaries:
 
-1. Add an explicit dataset loader that reads a local file supplied by the operator; do not add network downloads to the benchmark command.
-2. Preserve source license, version, split, and checksum metadata in the report.
+1. Download the official dataset files separately and pass local paths to `run-standard-memory-benchmarks.mjs`; the benchmark command itself does not fetch network resources.
+2. Preserve source URL, license, split/file name, and checksum metadata beside private run artifacts when publishing internally.
 3. Keep raw conversations, private memory, questions, and answers out of public reports unless the dataset license and review process explicitly allow publication.
-4. Route every candidate through the same Enigma vault, context-pack, optimizer, export, and verify operations measured here.
-5. Add separate agent/model evaluation only when the evaluated agent loop is fixed and documented; do not attribute agent/tool behavior solely to the memory store.
-6. Keep release notes bounded to the observed command, dataset, timestamp, and review approval.
+4. Treat standard-runner metrics as retrieval/evidence proxy scores only. Add separate LLM answer-accuracy evaluation only when the evaluated agent/model loop is fixed, documented, and credentialed by the operator.
+5. Keep release notes bounded to the observed command, dataset file name, input SHA-256, timestamp, top-k, max-item limits, and review approval.

package/docs/sdk-api.md

@@ -1,6 +1,6 @@
 # SDK and API guide
 
-This guide covers the public package imports for `enigma-memory@0.1.3`. The SDK runs locally by default: vaults, passports, context packs, receipts, relay/gateway demo state, storage contracts, metering artifacts, and settlement artifacts are package-level developer surfaces. They are not evidence of hosted Enigma cloud, provider-side deletion, provider model forgetting, token ROI, invoice savings, compliance certification, or benchmark leadership.
+This guide covers the public package imports for `enigma-memory@0.1.5`. The SDK runs locally by default: vaults, passports, context packs, receipts, relay/gateway demo state, storage contracts, metering artifacts, and settlement artifacts are package-level developer surfaces. They are not evidence of hosted Enigma cloud, provider-side deletion, provider model forgetting, token ROI, invoice savings, compliance certification, or benchmark leadership.
 
 ## Install and import style
 

package/examples/ci/github-actions.yml

@@ -2,6 +2,15 @@ name: Enigma Memory smoke
 
 on:
   workflow_dispatch:
+    inputs:
+      run_standard_benchmark:
+        description: 'Download official LoCoMo/LongMemEval data and run the bounded standard benchmark sample'
+        required: false
+        default: 'false'
+        type: choice
+        options:
+          - 'false'
+          - 'true'
   pull_request:
   push:
     branches:
@@ -22,7 +31,7 @@ jobs:
         run: npm init -y
 
       - name: Install Enigma Memory
-        run: npm install enigma-memory@0.1.3
+        run: npm install enigma-memory@0.1.4
 
       - name: Expose local benchmark script
         run: |
@@ -48,6 +57,23 @@ jobs:
           cp benchmark-report.json artifacts/benchmark-report.json
           echo "Saved public-safe benchmark report placeholder at artifacts/benchmark-report.json"
 
+      - name: Dry-run standard benchmark dataset download
+        if: ${{ github.event_name == 'workflow_dispatch' && inputs.run_standard_benchmark == 'true' }}
+        run: node ./node_modules/enigma-memory/scripts/download-standard-benchmarks.mjs --dry-run
+
+      - name: Download official benchmark datasets
+        if: ${{ github.event_name == 'workflow_dispatch' && inputs.run_standard_benchmark == 'true' }}
+        run: |
+          node ./node_modules/enigma-memory/scripts/download-standard-benchmarks.mjs --execute --dataset all --out-dir .enigma/benchmarks/datasets --manifest .enigma/benchmarks/dataset-manifest.json
+
+      - name: Run standard benchmark sample
+        if: ${{ github.event_name == 'workflow_dispatch' && inputs.run_standard_benchmark == 'true' }}
+        run: |
+          node ./node_modules/enigma-memory/scripts/run-standard-memory-benchmarks.mjs --locomo .enigma/benchmarks/datasets/locomo10.json --longmemeval .enigma/benchmarks/datasets/longmemeval_s_cleaned.json --max-locomo-qa 25 --max-longmemeval-items 25 --top-k 5 --out .enigma/standard-memory-benchmark-sample.json
+          mkdir -p artifacts
+          cp .enigma/benchmarks/dataset-manifest.json artifacts/dataset-manifest.json
+          cp .enigma/standard-memory-benchmark-sample.json artifacts/standard-memory-benchmark-sample.json
+
       # Optional artifact upload placeholder:
       # If your repository already permits actions/upload-artifact, uncomment a
       # reviewed upload step like this. The example keeps the report saved in the

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "enigma-memory",
-  "version": "0.1.3",
+  "version": "0.1.5",
   "type": "module",
   "description": "Provider-agnostic AI memory passport and offline-verifiable proof layer.",
   "license": "Apache-2.0",
@@ -61,6 +61,8 @@
     "scripts/install-enigma-local.mjs",
     "scripts/verify-registry-install.mjs",
     "scripts/run-memory-benchmarks.mjs",
+    "scripts/download-standard-benchmarks.mjs",
+    "scripts/run-standard-memory-benchmarks.mjs",
     "scripts/build-installer-assets.mjs",
     "scripts/package-browser-extension.mjs",
     "scripts/build-production-readiness-manifest.mjs",
@@ -145,6 +147,8 @@
     "infrastructure:readiness": "node scripts/infrastructure-readiness.mjs",
     "memory:benchmark": "node scripts/memory-optimization-benchmark.mjs",
     "benchmark:memory-suite": "node scripts/run-memory-benchmarks.mjs",
+    "benchmark:datasets": "node scripts/download-standard-benchmarks.mjs",
+    "benchmark:standard": "node scripts/run-standard-memory-benchmarks.mjs",
     "installer:assets": "node scripts/build-installer-assets.mjs",
     "browser:extension:package": "node scripts/package-browser-extension.mjs",
     "production:manifest": "node scripts/build-production-readiness-manifest.mjs",

package/packages/mcp-server/src/index.js

@@ -17,7 +17,7 @@ import {
 const DEFAULT_BUNDLE = '.enigma/bundle.json';
 const JSONRPC_VERSION = '2.0';
 const MCP_PROTOCOL_VERSION = '2024-11-05';
-const SERVER_INFO = Object.freeze({ name: 'enigma-mcp-server', version: '0.1.3' });
+const SERVER_INFO = Object.freeze({ name: 'enigma-mcp-server', version: '0.1.5' });
 const JSON_RPC_ID_PATTERN = /^[A-Za-z0-9._:-]{1,128}$/;
 const JSON_RPC_ERROR = Object.freeze({
   INVALID_REQUEST: -32600,