stratagate-dsh 0.3.7 → 0.3.8

package/docs/EVALUATION.md

@@ -1,189 +1,189 @@
-# Evaluation record and reporting rules
-
-This document separates completed development experiments from benchmark claims. Every public score should state the dataset scope, question categories, extraction model, answer model, Judge model, prompt, repetitions, and whether the memory state was created inside the same run.
-
-## Final round-eight `conv-26` result
-
-The latest completed run covers one LoCoMo conversation, `conv-26`: 419 messages, 35 sessions, and 152 category 1–4 questions. StrataGate completed all 152 answers and all 1,520 Judge decisions with no unrecovered question failure.
-
-> [!IMPORTANT]
-> This is a single-conversation directional comparison, not a full LoCoMo score or a cross-project leaderboard. StrataGate and Mem0 used the same questions, answer model, Judge, prompt, parser, and repetitions, but different memory-construction, retrieval, embedding, and answer-context pipelines. The comparison does not isolate memory architecture as one causal variable.
-
-### Overall result
-
-| Metric | StrataGate final R8 | Mem0 base | StrataGate - Mem0 |
-| --- | ---: | ---: | ---: |
-| Ten-Judge mean accuracy | **80.4606%** | 63.2237% | **+17.2369 pp** |
-| Standard deviation across Judge runs | 0.5138 pp | 0.9045 pp | -0.3907 pp |
-| Judge-run range | 79.6053%–81.5789% | 61.8421%–65.1316% | — |
-| Majority-correct questions | **121 / 152** | 96 / 152 | **+25** |
-| Majority accuracy | **79.61%** | 63.1579% | **+16.45 pp** |
-
-The ± values describe repeated Judge variation on one fixed answer set. They are not confidence intervals across LoCoMo conversations.
-
-### Mean accuracy by category
-
-| Category | Questions | StrataGate final R8 | Mem0 base | StrataGate - Mem0 |
-| --- | ---: | ---: | ---: | ---: |
-| Multi-hop | 32 | **66.5625%** | 61.5625% | +5.0000 pp |
-| Temporal | 37 | **74.8649%** | 34.5946% | **+40.2703 pp** |
-| Open-domain | 13 | 83.0769% | **84.6154%** | -1.5385 pp |
-| Single-hop | 70 | **89.2857%** | 75.1429% | **+14.1428 pp** |
-
-The largest observed difference is temporal. Mem0's local base run often anchored relative dates from the 2023 conversation to the 2026 experiment date. StrataGate retained source timestamps and could fall back to the original messages. This mechanism is consistent with the gap, but the paired result is not an ablation and does not prove that timestamps or raw fallback caused the full difference.
-
-### Paired majority outcomes
-
-| Outcome | Questions |
-| --- | ---: |
-| Both correct | 80 |
-| StrataGate correct, Mem0 wrong | 41 |
-| StrataGate wrong, Mem0 correct | 16 |
-| Both wrong | 15 |
-
-### Protocol matrix
-
-| Field | StrataGate final R8 | Mem0 base |
-| --- | --- | --- |
-| Dataset scope | `conv-26`, categories 1–4, 152 questions | Same IDs, order, text, gold answers, and categories |
-| Memory construction | Fresh build from 419 timestamped messages with `gpt-5.6-sol`; 17 blocks, 97 events, 4 elements | Fresh build with `gpt-5.6-sol`; 173 final memories |
-| Retrieval and assessment | BM25/RRF event and fact-level element tools, raw-message fallback, four-round budget, five-field evidence gate | Two speaker searches, top-30 each; no graph |
-| Answer model | `gpt-5.6-sol`, `reasoning_effort=low` | Same |
-| Judge | `gpt-5.6-sol`, ten repeats, concurrency 1 | Same model, prompt, parser, and repeats |
-| Judge prompt SHA-256 | `44fb3d8f7a1f37b2430772cf90518a32172e4056b7a0dec085402763fd179b9f` | Same |
-| Answer context and prompt | StrataGate-specific | Mem0-specific |
-| Embedding | No vector retrieval in the evaluated event/element search | `text-embedding-3-small`, 1536 dimensions |
-
-Mem0 used the local base SDK version 0.1.97 pinned to commit `2b58775c17eb1c1b7532242b7154af6744102280`, with Graph, Cloud, and Platform v3 disabled. It processed 419 source messages through two speaker views and completed all 428 write units.
-
-The StrataGate run recorded 2,065 successful model responses and 14 bodyless socket failures during extraction. All 14 failures were recovered from checkpoints, no completed question was replayed, and every successful response reported `gpt-5.6-sol`. The Mem0 run retained 187 historical failed-attempt traces, all recovered; its generated summary reports `passed=false` because it treats those bodyless attempts as missing `response.model`, even though successful responses returned the expected model. Completion and clean-transport acceptance are reported separately.
-
-The public aggregates are:
-
-- [`benchmarks/locomo-conv26-r8-final.json`](../benchmarks/locomo-conv26-r8-final.json) for final R8, its prior-round comparisons, and remaining-error stages;
-- [`benchmarks/locomo-conv26-sol-mem0-paired.json`](../benchmarks/locomo-conv26-sol-mem0-paired.json) for the earlier round-seven/Mem0 pairing retained as historical evidence;
-- [`benchmarks/locomo-conv26-development.json`](../benchmarks/locomo-conv26-development.json) for R1–R5.
-
-Raw requests and per-question traces are not copied into this public repository.
-
-| Source artifact | SHA-256 |
-| --- | --- |
-| Final R8 `summary.json` | `3e29d985fcdd88a385c106838c3046808c476affb63679c2e02601b5a2006656` |
-| Final R8 `checkpoint.json` | `82a2f5be362369c72aff39c7d6eaf9b5b6904ccba15a186aef1e7ff770ff79db` |
-| Final R8 `source-snapshot.patch` | `5eed8bd5fd7b316ad2aaf4b142992bac31c427ea81cbb1f7ab1e81449e08e581` |
-| Mem0 `summary.json` | `88f39b729546c6f343e51a11ad8f80bc1eea06ba831f33008fad779b04962927` |
-| Mem0 `paired-comparison.json` | `27b604692364f24600d9513d8d2b91da9b52245a4574f581afb0afc6f00ee7fc` |
-| Mem0 `protocol-audit.json` | `c1e39714eb08a775f9cbe12d0a43de6d825c142991840eab329398d1b9d744eb` |
-
-## Development sequence
-
-All recorded rounds used the same `conv-26` development slice, but the model and scoring protocols changed. R1–R5 used `gpt-4o-mini` extraction with a GPT-4o answerer/Judge setup. R7 and R8 used ten `gpt-5.6-sol` Judge decisions per question. R6 stopped at a 123-question snapshot and has no final score.
-
-| Run | Main intervention | Reported result | Protocol note |
-| --- | --- | ---: | --- |
-| R1 | Initial layered blocks and event cards | 67 / 152 (44.08%) | Temporal accuracy 18.92% |
-| R2 | Multiple event cards per block and explicit occurrence time | 77 / 152 (50.66%) | Temporal accuracy rose to 45.95% |
-| R3 | Extraction, read tools, and per-batch assessment changed together | 116 / 152 (76.32%) | One adoption-rule violation; strict result 115 / 152 (75.66%) |
-| R4 | Bounded five-field assessment context | 118 / 152 (77.63%) | Zero recorded adoption violations and about 10.35% fewer QA tokens than R3 |
-| R5 | Larger structured retrieval scratchpad | 97 / 152 (63.82%) | Reproducible regression led to restoring the smaller gate |
-| R6 | Mini-model sensitivity run | incomplete | 123-question snapshot only; no final score |
-| R7 | Sol retrieval, assessment, answer, and Judge | 71.9737% mean; 111 / 152 majority | Reused a state extracted with `gpt-4o-mini`; not end-to-end Sol |
-| R8 initial | Fresh end-to-end Sol extraction and hybrid event/element retrieval | 70.3289% mean; 107 / 152 majority | Repeated card searches frequently exhausted the budget |
-| **R8 final** | Fresh end-to-end Sol state with raw fallback and retrieval fixes | **80.4606% mean; 121 / 152 majority** | Best completed result on this slice |
-
-The sequence supports several engineering conclusions, not a single cumulative causal curve:
-
-1. Explicit occurrence time improved temporal questions on the fixed early-round protocol.
-2. A small, enforceable evidence gate performed better than a larger structured scratchpad.
-3. Model and Judge changes materially affect scores; at least one fixed answer moved from 10 / 10 correct under the mini Judge to 0 / 10 under the Sol Judge.
-4. Retrieval strategy matters more than raw search count. Repeating the same card search can be worse than changing channel and checking the original source.
-5. Element-card value remains unresolved. Final R8 used `search_elements` on only four questions; all four were correct, but this usage-conditioned subset is too small and selected to support a causal claim.
-
-## Initial R8 versus final R8
-
-| Metric | Initial R8 | Final R8 | Change |
-| --- | ---: | ---: | ---: |
-| Ten-Judge mean | 70.3289% | **80.4606%** | **+10.1317 pp** |
-| Majority correct | 107 / 152 | **121 / 152** | **+14** |
-| Retrieval rounds | 215 | **146** | -69 (-32.1%) |
-| Assessment calls | 237 | **146** | -91 (-38.4%) |
-| Total tokens | 6,692,417 | **4,088,324** | -2,604,093 (-38.9%) |
-| Official-equivalent cost | $33.907499 | **$21.406145** | -$12.501354 (-36.9%) |
-
-Question-level majority transitions were: 95 both correct, 26 final-R8-only correct, 12 initial-R8-only correct, and 19 both wrong.
-
-The initial run had 19 questions with at least three `search_events` calls; only 2 were majority-correct. Final R8 answered 15 of those same questions correctly, with 12 using raw-message fallback. A representative temporal question asked when Caroline gave a school speech: the initial run repeatedly searched event cards and abstained, while final R8 fell back to the source message dated 2023-06-09, resolved “last week,” and received 10 / 10 correct Judge votes.
-
-This is strong diagnostic evidence that the old repeated-search path was defective. It is not a clean ablation: retrieval policy, filters, element result shape, and freshly extracted memory contents all changed.
-
-## Remaining-error analysis
-
-Final R8 has 31 majority-wrong questions. Their observable failure stage is:
-
-| Failure stage | Questions | Multi-hop | Temporal | Open-domain | Single-hop |
-| --- | ---: | ---: | ---: | ---: | ---: |
-| Direct answer without retrieval | 15 | 5 | 3 | 2 | 5 |
-| Retrieval marked `sufficient`, final answer wrong | 14 | 6 | 6 | 0 | 2 |
-| Retrieval remained `partial` at the limit | 2 | 0 | 1 | 0 | 1 |
-| **Total** | **31** | **11** | **10** | **2** | **8** |
-
-The two largest problems are therefore not a lack of retrieval rounds:
-
-- **Direct-answer risk:** 15 wrong answers bypassed retrieval entirely. These include missed list facts, wrong event details, and unanchored relative dates.
-- **False sufficiency:** 14 answers passed the evidence gate but were still wrong. The selected material was often related but belonged to an adjacent event, or it supported only part of a requested list.
-
-Recurring patterns include:
-
-- choosing the wrong research topic or the wrong pottery event;
-- resolving a relative date against the wrong message or returning “next month” without an absolute anchor;
-- omitting one item from books, instruments, purchases, or attended events;
-- using a generally plausible reason instead of the event-specific reason requested;
-- open-domain Judge disagreements where the memory evidence supports more than one cautious inference.
-
-The next diagnostic should inject the gold-evidence raw block while keeping the answer and Judge protocol fixed. That oracle split will distinguish retrieval/selection failure from answer reasoning and temporal-calculation failure. Before increasing the retrieval budget, the implementation should also gate direct temporal/multi-hop answers, tighten `sufficient` against adjacent events, and add completeness checks for list questions.
-
-## Reporting checklist
-
-Before treating two scores as comparable, freeze and report:
-
-- dataset version and conversation/question scope;
-- included categories;
-- extraction code and model;
-- retrieval code, tools, budgets, and prompts;
-- answer model, temperature, and reasoning configuration;
-- Judge prompt, model, temperature, parser, and repetitions;
-- provider and returned-model audit;
-- memory-state provenance;
-- retries, missing responses, and checkpoint completion;
-- exact numerator and denominator, not only a rounded percentage.
-
-## What the current evidence supports
-
-The current evidence supports these narrow claims:
-
-- on `conv-26` and the tested configurations, final R8 scored 121 / 152 by majority vote and Mem0 base scored 96 / 152;
-- the largest observed paired category difference was temporal, while Mem0 remained slightly higher on open-domain questions;
-- the final R8 retrieval policy avoided a measured repeated-event-search failure mode while using fewer rounds, tokens, and cost;
-- explicit occurrence time, an enforceable evidence gate, and raw-source access are useful components on this development slice;
-- remaining errors are split mainly between direct answers that skipped retrieval and retrieved evidence incorrectly marked sufficient;
-- completed per-question checkpoints and request traces are necessary to distinguish model behavior from transport failure.
-
-It does not yet support these broader claims:
-
-- state of the art on LoCoMo;
-- generalization across the full LoCoMo dataset;
-- architectural superiority under a shared end-to-end retrieval and context protocol;
-- a single-component causal explanation for the final R8 gain;
-- a causal benefit from element cards without disabled/forced/fixed-state ablations.
-
-The next credible milestone is a fixed-state component ablation, followed by the same frozen end-to-end protocol across more conversations and then the full dataset.
-
-## Follow-up validation plan
-
-The following are proposed experiments, not completed results:
-
-1. Hold the model and memory state fixed and ablate short-term display decay and individual retrieval mechanisms. Compare answer accuracy, historical context size, background model input, and total call costs separately.
-2. Give the answering model known-correct source evidence directly. This separates failures to retrieve evidence from reasoning failures that remain when evidence is available.
-3. Repeat the frozen protocol across additional conversations before reporting a full LoCoMo result.
-
-These experiments would help distinguish the contribution of each component from improvements observed between versions that changed multiple components together.
+# Evaluation record and reporting rules
+
+This document separates completed development experiments from benchmark claims. Every public score should state the dataset scope, question categories, extraction model, answer model, Judge model, prompt, repetitions, and whether the memory state was created inside the same run.
+
+## Final round-eight `conv-26` result
+
+The latest completed run covers one LoCoMo conversation, `conv-26`: 419 messages, 35 sessions, and 152 category 1–4 questions. StrataGate completed all 152 answers and all 1,520 Judge decisions with no unrecovered question failure.
+
+> [!IMPORTANT]
+> This is a single-conversation directional comparison, not a full LoCoMo score or a cross-project leaderboard. StrataGate and Mem0 used the same questions, answer model, Judge, prompt, parser, and repetitions, but different memory-construction, retrieval, embedding, and answer-context pipelines. The comparison does not isolate memory architecture as one causal variable.
+
+### Overall result
+
+| Metric | StrataGate final R8 | Mem0 base | StrataGate - Mem0 |
+| --- | ---: | ---: | ---: |
+| Ten-Judge mean accuracy | **80.4606%** | 63.2237% | **+17.2369 pp** |
+| Standard deviation across Judge runs | 0.5138 pp | 0.9045 pp | -0.3907 pp |
+| Judge-run range | 79.6053%–81.5789% | 61.8421%–65.1316% | — |
+| Majority-correct questions | **121 / 152** | 96 / 152 | **+25** |
+| Majority accuracy | **79.61%** | 63.1579% | **+16.45 pp** |
+
+The ± values describe repeated Judge variation on one fixed answer set. They are not confidence intervals across LoCoMo conversations.
+
+### Mean accuracy by category
+
+| Category | Questions | StrataGate final R8 | Mem0 base | StrataGate - Mem0 |
+| --- | ---: | ---: | ---: | ---: |
+| Multi-hop | 32 | **66.5625%** | 61.5625% | +5.0000 pp |
+| Temporal | 37 | **74.8649%** | 34.5946% | **+40.2703 pp** |
+| Open-domain | 13 | 83.0769% | **84.6154%** | -1.5385 pp |
+| Single-hop | 70 | **89.2857%** | 75.1429% | **+14.1428 pp** |
+
+The largest observed difference is temporal. Mem0's local base run often anchored relative dates from the 2023 conversation to the 2026 experiment date. StrataGate retained source timestamps and could fall back to the original messages. This mechanism is consistent with the gap, but the paired result is not an ablation and does not prove that timestamps or raw fallback caused the full difference.
+
+### Paired majority outcomes
+
+| Outcome | Questions |
+| --- | ---: |
+| Both correct | 80 |
+| StrataGate correct, Mem0 wrong | 41 |
+| StrataGate wrong, Mem0 correct | 16 |
+| Both wrong | 15 |
+
+### Protocol matrix
+
+| Field | StrataGate final R8 | Mem0 base |
+| --- | --- | --- |
+| Dataset scope | `conv-26`, categories 1–4, 152 questions | Same IDs, order, text, gold answers, and categories |
+| Memory construction | Fresh build from 419 timestamped messages with `gpt-5.6-sol`; 17 blocks, 97 events, 4 elements | Fresh build with `gpt-5.6-sol`; 173 final memories |
+| Retrieval and assessment | BM25/RRF event and fact-level element tools, raw-message fallback, four-round budget, five-field evidence gate | Two speaker searches, top-30 each; no graph |
+| Answer model | `gpt-5.6-sol`, `reasoning_effort=low` | Same |
+| Judge | `gpt-5.6-sol`, ten repeats, concurrency 1 | Same model, prompt, parser, and repeats |
+| Judge prompt SHA-256 | `44fb3d8f7a1f37b2430772cf90518a32172e4056b7a0dec085402763fd179b9f` | Same |
+| Answer context and prompt | StrataGate-specific | Mem0-specific |
+| Embedding | No vector retrieval in the evaluated event/element search | `text-embedding-3-small`, 1536 dimensions |
+
+Mem0 used the local base SDK version 0.1.97 pinned to commit `2b58775c17eb1c1b7532242b7154af6744102280`, with Graph, Cloud, and Platform v3 disabled. It processed 419 source messages through two speaker views and completed all 428 write units.
+
+The StrataGate run recorded 2,065 successful model responses and 14 bodyless socket failures during extraction. All 14 failures were recovered from checkpoints, no completed question was replayed, and every successful response reported `gpt-5.6-sol`. The Mem0 run retained 187 historical failed-attempt traces, all recovered; its generated summary reports `passed=false` because it treats those bodyless attempts as missing `response.model`, even though successful responses returned the expected model. Completion and clean-transport acceptance are reported separately.
+
+The public aggregates are:
+
+- [`benchmarks/locomo-conv26-r8-final.json`](../benchmarks/locomo-conv26-r8-final.json) for final R8, its prior-round comparisons, and remaining-error stages;
+- [`benchmarks/locomo-conv26-sol-mem0-paired.json`](../benchmarks/locomo-conv26-sol-mem0-paired.json) for the earlier round-seven/Mem0 pairing retained as historical evidence;
+- [`benchmarks/locomo-conv26-development.json`](../benchmarks/locomo-conv26-development.json) for R1–R5.
+
+Raw requests and per-question traces are not copied into this public repository.
+
+| Source artifact | SHA-256 |
+| --- | --- |
+| Final R8 `summary.json` | `3e29d985fcdd88a385c106838c3046808c476affb63679c2e02601b5a2006656` |
+| Final R8 `checkpoint.json` | `82a2f5be362369c72aff39c7d6eaf9b5b6904ccba15a186aef1e7ff770ff79db` |
+| Final R8 `source-snapshot.patch` | `5eed8bd5fd7b316ad2aaf4b142992bac31c427ea81cbb1f7ab1e81449e08e581` |
+| Mem0 `summary.json` | `88f39b729546c6f343e51a11ad8f80bc1eea06ba831f33008fad779b04962927` |
+| Mem0 `paired-comparison.json` | `27b604692364f24600d9513d8d2b91da9b52245a4574f581afb0afc6f00ee7fc` |
+| Mem0 `protocol-audit.json` | `c1e39714eb08a775f9cbe12d0a43de6d825c142991840eab329398d1b9d744eb` |
+
+## Development sequence
+
+All recorded rounds used the same `conv-26` development slice, but the model and scoring protocols changed. R1–R5 used `gpt-4o-mini` extraction with a GPT-4o answerer/Judge setup. R7 and R8 used ten `gpt-5.6-sol` Judge decisions per question. R6 stopped at a 123-question snapshot and has no final score.
+
+| Run | Main intervention | Reported result | Protocol note |
+| --- | --- | ---: | --- |
+| R1 | Initial layered blocks and event cards | 67 / 152 (44.08%) | Temporal accuracy 18.92% |
+| R2 | Multiple event cards per block and explicit occurrence time | 77 / 152 (50.66%) | Temporal accuracy rose to 45.95% |
+| R3 | Extraction, read tools, and per-batch assessment changed together | 116 / 152 (76.32%) | One adoption-rule violation; strict result 115 / 152 (75.66%) |
+| R4 | Bounded five-field assessment context | 118 / 152 (77.63%) | Zero recorded adoption violations and about 10.35% fewer QA tokens than R3 |
+| R5 | Larger structured retrieval scratchpad | 97 / 152 (63.82%) | Reproducible regression led to restoring the smaller gate |
+| R6 | Mini-model sensitivity run | incomplete | 123-question snapshot only; no final score |
+| R7 | Sol retrieval, assessment, answer, and Judge | 71.9737% mean; 111 / 152 majority | Reused a state extracted with `gpt-4o-mini`; not end-to-end Sol |
+| R8 initial | Fresh end-to-end Sol extraction and hybrid event/element retrieval | 70.3289% mean; 107 / 152 majority | Repeated card searches frequently exhausted the budget |
+| **R8 final** | Fresh end-to-end Sol state with raw fallback and retrieval fixes | **80.4606% mean; 121 / 152 majority** | Best completed result on this slice |
+
+The sequence supports several engineering conclusions, not a single cumulative causal curve:
+
+1. Explicit occurrence time improved temporal questions on the fixed early-round protocol.
+2. A small, enforceable evidence gate performed better than a larger structured scratchpad.
+3. Model and Judge changes materially affect scores; at least one fixed answer moved from 10 / 10 correct under the mini Judge to 0 / 10 under the Sol Judge.
+4. Retrieval strategy matters more than raw search count. Repeating the same card search can be worse than changing channel and checking the original source.
+5. Element-card value remains unresolved. Final R8 used `search_elements` on only four questions; all four were correct, but this usage-conditioned subset is too small and selected to support a causal claim.
+
+## Initial R8 versus final R8
+
+| Metric | Initial R8 | Final R8 | Change |
+| --- | ---: | ---: | ---: |
+| Ten-Judge mean | 70.3289% | **80.4606%** | **+10.1317 pp** |
+| Majority correct | 107 / 152 | **121 / 152** | **+14** |
+| Retrieval rounds | 215 | **146** | -69 (-32.1%) |
+| Assessment calls | 237 | **146** | -91 (-38.4%) |
+| Total tokens | 6,692,417 | **4,088,324** | -2,604,093 (-38.9%) |
+| Official-equivalent cost | $33.907499 | **$21.406145** | -$12.501354 (-36.9%) |
+
+Question-level majority transitions were: 95 both correct, 26 final-R8-only correct, 12 initial-R8-only correct, and 19 both wrong.
+
+The initial run had 19 questions with at least three `search_events` calls; only 2 were majority-correct. Final R8 answered 15 of those same questions correctly, with 12 using raw-message fallback. A representative temporal question asked when Caroline gave a school speech: the initial run repeatedly searched event cards and abstained, while final R8 fell back to the source message dated 2023-06-09, resolved “last week,” and received 10 / 10 correct Judge votes.
+
+This is strong diagnostic evidence that the old repeated-search path was defective. It is not a clean ablation: retrieval policy, filters, element result shape, and freshly extracted memory contents all changed.
+
+## Remaining-error analysis
+
+Final R8 has 31 majority-wrong questions. Their observable failure stage is:
+
+| Failure stage | Questions | Multi-hop | Temporal | Open-domain | Single-hop |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Direct answer without retrieval | 15 | 5 | 3 | 2 | 5 |
+| Retrieval marked `sufficient`, final answer wrong | 14 | 6 | 6 | 0 | 2 |
+| Retrieval remained `partial` at the limit | 2 | 0 | 1 | 0 | 1 |
+| **Total** | **31** | **11** | **10** | **2** | **8** |
+
+The two largest problems are therefore not a lack of retrieval rounds:
+
+- **Direct-answer risk:** 15 wrong answers bypassed retrieval entirely. These include missed list facts, wrong event details, and unanchored relative dates.
+- **False sufficiency:** 14 answers passed the evidence gate but were still wrong. The selected material was often related but belonged to an adjacent event, or it supported only part of a requested list.
+
+Recurring patterns include:
+
+- choosing the wrong research topic or the wrong pottery event;
+- resolving a relative date against the wrong message or returning “next month” without an absolute anchor;
+- omitting one item from books, instruments, purchases, or attended events;
+- using a generally plausible reason instead of the event-specific reason requested;
+- open-domain Judge disagreements where the memory evidence supports more than one cautious inference.
+
+The next diagnostic should inject the gold-evidence raw block while keeping the answer and Judge protocol fixed. That oracle split will distinguish retrieval/selection failure from answer reasoning and temporal-calculation failure. Before increasing the retrieval budget, the implementation should also gate direct temporal/multi-hop answers, tighten `sufficient` against adjacent events, and add completeness checks for list questions.
+
+## Reporting checklist
+
+Before treating two scores as comparable, freeze and report:
+
+- dataset version and conversation/question scope;
+- included categories;
+- extraction code and model;
+- retrieval code, tools, budgets, and prompts;
+- answer model, temperature, and reasoning configuration;
+- Judge prompt, model, temperature, parser, and repetitions;
+- provider and returned-model audit;
+- memory-state provenance;
+- retries, missing responses, and checkpoint completion;
+- exact numerator and denominator, not only a rounded percentage.
+
+## What the current evidence supports
+
+The current evidence supports these narrow claims:
+
+- on `conv-26` and the tested configurations, final R8 scored 121 / 152 by majority vote and Mem0 base scored 96 / 152;
+- the largest observed paired category difference was temporal, while Mem0 remained slightly higher on open-domain questions;
+- the final R8 retrieval policy avoided a measured repeated-event-search failure mode while using fewer rounds, tokens, and cost;
+- explicit occurrence time, an enforceable evidence gate, and raw-source access are useful components on this development slice;
+- remaining errors are split mainly between direct answers that skipped retrieval and retrieved evidence incorrectly marked sufficient;
+- completed per-question checkpoints and request traces are necessary to distinguish model behavior from transport failure.
+
+It does not yet support these broader claims:
+
+- state of the art on LoCoMo;
+- generalization across the full LoCoMo dataset;
+- architectural superiority under a shared end-to-end retrieval and context protocol;
+- a single-component causal explanation for the final R8 gain;
+- a causal benefit from element cards without disabled/forced/fixed-state ablations.
+
+The next credible milestone is a fixed-state component ablation, followed by the same frozen end-to-end protocol across more conversations and then the full dataset.
+
+## Follow-up validation plan
+
+The following are proposed experiments, not completed results:
+
+1. Hold the model and memory state fixed and ablate short-term display decay and individual retrieval mechanisms. Compare answer accuracy, historical context size, background model input, and total call costs separately.
+2. Give the answering model known-correct source evidence directly. This separates failures to retrieve evidence from reasoning failures that remain when evidence is available.
+3. Repeat the frozen protocol across additional conversations before reporting a full LoCoMo result.
+
+These experiments would help distinguish the contribution of each component from improvements observed between versions that changed multiple components together.

package/docs/EXTERNAL_MEMORY_IMPORT.zh-CN.md

@@ -1,99 +1,99 @@
-# 外部 AI 记忆导入
-
-StrataGate 提供 `importExternalMemory()` 编排外部记忆迁移：
-
-```text
-外部 AI JSON
-    ↓ extractor（候选 Event）
-每个候选 Event → searchEvents()（确定性 BM25，Top-K）
-    ↓ decider（ADD / MERGE / SUPERSEDE / CONFLICT / IGNORE）
-写入新的规范 Event，保留来源和 supersedes/conflicts 关系
-    ↓
-只为新 Event 创建元素/知识图谱投影任务
-```
-
-## 最小接入
-
-```ts
-import {
-  EXTERNAL_MEMORY_EXPORT_PROMPT_ZH_CN,
-  StrataGate,
-  externalMemoryJsonExtractor,
-} from '@diqier/stratagate';
-
-// 把 EXTERNAL_MEMORY_EXPORT_PROMPT_ZH_CN 交给外部 AI，并把它返回的 JSON 放入 text。
-const result = await memory.importExternalMemory({
-  text,
-  extractor: externalMemoryJsonExtractor,
-  topK: 5,
-  decider: async ({ candidate, matches }) => {
-    // 这里通常调用你的 LLM；它只能从 matches 中选择 existingEventIds。
-    // 返回的 MERGE/SUPERSEDE 会创建新 Event，不会覆盖旧 Event。
-    return {
-      action: matches.length === 0 ? 'ADD' : 'MERGE',
-      existingEventIds: matches.slice(0, 1).map(({ event }) => event.id),
-      mergedCandidate: candidate,
-    };
-  },
-});
-```
-
-`decider` 的 `matches` 已经被限制为 `topK` 条；即使模型返回其它事件 ID，库也会丢弃这些越界引用。`IGNORE` 只留下审计记录，不会写入 Event。`CONFLICT` 会在新旧事件两侧建立对称的 `conflictsWithEventIds`。
-
-## 给外部 AI 的提示词
-
-直接使用导出的 `EXTERNAL_MEMORY_EXPORT_PROMPT_ZH_CN`。它要求外部 AI 只输出 `stratagate.external-memory.v2` JSON，并区分 `memoryKind`（instruction/preference/fact/event）与 `category`，同时特别约束时间：
-
-- `mentionedAt`（被提及时间）与 `happenedStart/happenedEnd`（实际发生/计划时间）分开；
-- 没有明确时间或可靠参照时，不填写日期，不把当前时间、导出时间或聊天顺序当作事件时间；
-- 保留 `originalText`，用 `precision` 和 `basis` 标记粒度与依据；
-- “上周”等相对时间只有在能依据已知消息时间唯一换算时才转换，否则保持 `unknown`。
-
-如果外部 AI 仍然返回 Markdown 代码块，`parseExternalMemoryExport()` 会自动去除围栏；其它非 JSON 文本会被拒绝，避免把模型解释误写进记忆。
-
-用于第二阶段裁决的系统提示词可使用 `EXTERNAL_MEMORY_DECIDER_PROMPT_ZH_CN`。它明确规定了五种写入动作的边界，并要求模型只能引用本次 Top-K 结果中的事件 ID。
-
-## 导出常量、格式与解析器
-
-| 入口 | 用途 |
-| --- | --- |
-| `EXTERNAL_MEMORY_EXPORT_PROMPT_ZH_CN` | 交给外部 AI，要求生成 `stratagate.external-memory.v2` JSON |
-| `parseExternalMemoryExport()` | 严格解析导出内容，允许去除 Markdown 代码围栏 |
-| `externalMemoryJsonExtractor` | 将严格 JSON 解析包装为可直接传给核心导入 API 的 extractor |
-| `EXTERNAL_MEMORY_DECIDER_PROMPT_ZH_CN` | 指导裁决模型从 ADD、MERGE、SUPERSEDE、CONFLICT、IGNORE 中选择动作 |
-
-v2 将记忆性质 `memoryKind` 与内容分类 `category` 分开。决定如何导入时，应同时保留原始说法、来源和时间依据；未知日期不应由导出日期补齐。最小接入示例中的 `memory` 是已经打开的 StrataGate 实例，`text` 是外部 AI 返回的内容。
-
-示例里的“有匹配就 MERGE”只展示回调结构，不应直接作为实际裁决策略。检索相关不代表两条记忆属于同一件事；实际接入应结合候选内容和匹配结果决定动作。
-
-## DSH 界面：预览、恢复与确认
-
-DSH 管理界面使用可持久化的导入任务，而不是只调用一次严格解析器：
-
-1. **创建任务并解析。** 保存导入文本，尝试解析候选记忆。合法 JSON 进入逐条处理；解析失败的任务进入 `extracting` 状态，由运行时调用模型尝试恢复候选。
-2. **逐条比较。** 每条候选只与有限数量的本地 Event 比较。核心任务默认 `topK=5`，限制在 1–20；与已有有效记忆或本批次较早候选完全重复的内容会确定性地选择 IGNORE。
-3. **生成处理建议。** 模型决定新增、合并、取代、冲突或忽略。引用仅限本次匹配范围；需要既有目标却没有有效目标的建议会降低置信度。
-4. **确认与提交。** 正常解析的候选在置信度达到当前阈值 0.85 时无需逐项确认；低置信度项需要确认。由不合格 JSON 恢复出来的候选会强制进入确认流程，不能仅凭高置信度自动放行。
-
-严格的 `parseExternalMemoryExport()` / `externalMemoryJsonExtractor` 本身不会调用模型修复 JSON。模型兜底属于 DSH 运行时的任务编排；恢复不出候选时，任务失败并记录错误。
-
-## 进度保存与失败重试
-
-任务保存原始输入、候选列表、已处理数量、逐条决定及错误信息。关闭后重新打开页面可以继续查看任务，不需要重新粘贴原始内容。持久化使用需要通过 `StrataGate.open()` 打开数据库；内存模式不提供进程退出后的保留保证。
-
-核心任务接口分别处理：
-
-- `createExternalMemoryImportJob()`：创建任务并尝试严格解析。
-- `completeExternalMemoryFallback()`：保存运行时模型恢复出的候选。
-- `prepareNextExternalMemoryImport()` / `completeNextExternalMemoryImport()`：准备和保存单条裁决；模型调用在事务外执行。
-- `retryExternalMemoryImportJob()`：将失败任务恢复到解析恢复或逐条处理阶段，保留已保存进度。
-
-这些接口管理任务状态；自定义接入方仍需驱动模型调用和后续提交。重复提交已经完成的候选索引不会再推进一次进度。
-
-## 来源保留与批次撤销
-
-正常导入把原文保存为来源 Block，新 Event 引用该来源。MERGE / SUPERSEDE 创建新的 Event，并维护旧事件状态和关系，不直接覆盖旧事件正文。只为新写入的 Event 安排后续投影任务。
-
-显式撤销是独立操作：`undoExternalMemoryImport(sourceBlockId)` 会移除该批次导入的 Event，清理相应来源关系与派生数据，并按剩余替代关系恢复受影响的旧事件状态。它不等同于权重衰减，也不能理解为仅隐藏导入结果。DSH 界面提供提交后的批次撤销入口。
-
-实现入口：[核心导入与任务状态](../packages/core/src/store.ts)、[格式与提示词](../packages/core/src/external-memory.ts)、[DSH 运行时](../src/runtime.ts)。
+# 外部 AI 记忆导入
+
+StrataGate 提供 `importExternalMemory()` 编排外部记忆迁移：
+
+```text
+外部 AI JSON
+    ↓ extractor（候选 Event）
+每个候选 Event → searchEvents()（确定性 BM25，Top-K）
+    ↓ decider（ADD / MERGE / SUPERSEDE / CONFLICT / IGNORE）
+写入新的规范 Event，保留来源和 supersedes/conflicts 关系
+    ↓
+只为新 Event 创建元素/知识图谱投影任务
+```
+
+## 最小接入
+
+```ts
+import {
+  EXTERNAL_MEMORY_EXPORT_PROMPT_ZH_CN,
+  StrataGate,
+  externalMemoryJsonExtractor,
+} from '@diqier/stratagate';
+
+// 把 EXTERNAL_MEMORY_EXPORT_PROMPT_ZH_CN 交给外部 AI，并把它返回的 JSON 放入 text。
+const result = await memory.importExternalMemory({
+  text,
+  extractor: externalMemoryJsonExtractor,
+  topK: 5,
+  decider: async ({ candidate, matches }) => {
+    // 这里通常调用你的 LLM；它只能从 matches 中选择 existingEventIds。
+    // 返回的 MERGE/SUPERSEDE 会创建新 Event，不会覆盖旧 Event。
+    return {
+      action: matches.length === 0 ? 'ADD' : 'MERGE',
+      existingEventIds: matches.slice(0, 1).map(({ event }) => event.id),
+      mergedCandidate: candidate,
+    };
+  },
+});
+```
+
+`decider` 的 `matches` 已经被限制为 `topK` 条；即使模型返回其它事件 ID，库也会丢弃这些越界引用。`IGNORE` 只留下审计记录，不会写入 Event。`CONFLICT` 会在新旧事件两侧建立对称的 `conflictsWithEventIds`。
+
+## 给外部 AI 的提示词
+
+直接使用导出的 `EXTERNAL_MEMORY_EXPORT_PROMPT_ZH_CN`。它要求外部 AI 只输出 `stratagate.external-memory.v2` JSON，并区分 `memoryKind`（instruction/preference/fact/event）与 `category`，同时特别约束时间：
+
+- `mentionedAt`（被提及时间）与 `happenedStart/happenedEnd`（实际发生/计划时间）分开；
+- 没有明确时间或可靠参照时，不填写日期，不把当前时间、导出时间或聊天顺序当作事件时间；
+- 保留 `originalText`，用 `precision` 和 `basis` 标记粒度与依据；
+- “上周”等相对时间只有在能依据已知消息时间唯一换算时才转换，否则保持 `unknown`。
+
+如果外部 AI 仍然返回 Markdown 代码块，`parseExternalMemoryExport()` 会自动去除围栏；其它非 JSON 文本会被拒绝，避免把模型解释误写进记忆。
+
+用于第二阶段裁决的系统提示词可使用 `EXTERNAL_MEMORY_DECIDER_PROMPT_ZH_CN`。它明确规定了五种写入动作的边界，并要求模型只能引用本次 Top-K 结果中的事件 ID。
+
+## 导出常量、格式与解析器
+
+| 入口 | 用途 |
+| --- | --- |
+| `EXTERNAL_MEMORY_EXPORT_PROMPT_ZH_CN` | 交给外部 AI，要求生成 `stratagate.external-memory.v2` JSON |
+| `parseExternalMemoryExport()` | 严格解析导出内容，允许去除 Markdown 代码围栏 |
+| `externalMemoryJsonExtractor` | 将严格 JSON 解析包装为可直接传给核心导入 API 的 extractor |
+| `EXTERNAL_MEMORY_DECIDER_PROMPT_ZH_CN` | 指导裁决模型从 ADD、MERGE、SUPERSEDE、CONFLICT、IGNORE 中选择动作 |
+
+v2 将记忆性质 `memoryKind` 与内容分类 `category` 分开。决定如何导入时，应同时保留原始说法、来源和时间依据；未知日期不应由导出日期补齐。最小接入示例中的 `memory` 是已经打开的 StrataGate 实例，`text` 是外部 AI 返回的内容。
+
+示例里的“有匹配就 MERGE”只展示回调结构，不应直接作为实际裁决策略。检索相关不代表两条记忆属于同一件事；实际接入应结合候选内容和匹配结果决定动作。
+
+## DSH 界面：预览、恢复与确认
+
+DSH 管理界面使用可持久化的导入任务，而不是只调用一次严格解析器：
+
+1. **创建任务并解析。** 保存导入文本，尝试解析候选记忆。合法 JSON 进入逐条处理；解析失败的任务进入 `extracting` 状态，由运行时调用模型尝试恢复候选。
+2. **逐条比较。** 每条候选只与有限数量的本地 Event 比较。核心任务默认 `topK=5`，限制在 1–20；与已有有效记忆或本批次较早候选完全重复的内容会确定性地选择 IGNORE。
+3. **生成处理建议。** 模型决定新增、合并、取代、冲突或忽略。引用仅限本次匹配范围；需要既有目标却没有有效目标的建议会降低置信度。
+4. **确认与提交。** 正常解析的候选在置信度达到当前阈值 0.85 时无需逐项确认；低置信度项需要确认。由不合格 JSON 恢复出来的候选会强制进入确认流程，不能仅凭高置信度自动放行。
+
+严格的 `parseExternalMemoryExport()` / `externalMemoryJsonExtractor` 本身不会调用模型修复 JSON。模型兜底属于 DSH 运行时的任务编排；恢复不出候选时，任务失败并记录错误。
+
+## 进度保存与失败重试
+
+任务保存原始输入、候选列表、已处理数量、逐条决定及错误信息。关闭后重新打开页面可以继续查看任务，不需要重新粘贴原始内容。持久化使用需要通过 `StrataGate.open()` 打开数据库；内存模式不提供进程退出后的保留保证。
+
+核心任务接口分别处理：
+
+- `createExternalMemoryImportJob()`：创建任务并尝试严格解析。
+- `completeExternalMemoryFallback()`：保存运行时模型恢复出的候选。
+- `prepareNextExternalMemoryImport()` / `completeNextExternalMemoryImport()`：准备和保存单条裁决；模型调用在事务外执行。
+- `retryExternalMemoryImportJob()`：将失败任务恢复到解析恢复或逐条处理阶段，保留已保存进度。
+
+这些接口管理任务状态；自定义接入方仍需驱动模型调用和后续提交。重复提交已经完成的候选索引不会再推进一次进度。
+
+## 来源保留与批次撤销
+
+正常导入把原文保存为来源 Block，新 Event 引用该来源。MERGE / SUPERSEDE 创建新的 Event，并维护旧事件状态和关系，不直接覆盖旧事件正文。只为新写入的 Event 安排后续投影任务。
+
+显式撤销是独立操作：`undoExternalMemoryImport(sourceBlockId)` 会移除该批次导入的 Event，清理相应来源关系与派生数据，并按剩余替代关系恢复受影响的旧事件状态。它不等同于权重衰减，也不能理解为仅隐藏导入结果。DSH 界面提供提交后的批次撤销入口。
+
+实现入口：[核心导入与任务状态](../packages/core/src/store.ts)、[格式与提示词](../packages/core/src/external-memory.ts)、[DSH 运行时](../src/runtime.ts)。