@cairnkeep/cli 2.22.5 → 2.23.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (36) hide show
  1. package/CHANGELOG.md +21 -0
  2. package/claude/commands/recall.md +8 -1
  3. package/docs/README.md +3 -0
  4. package/docs/agents.md +9 -0
  5. package/docs/benchmarks/memory-retrieval-v1.json +115 -0
  6. package/docs/learning/CURRICULUM-MAP.md +1 -1
  7. package/docs/learning/FEATURE-GUIDE.md +7 -1
  8. package/docs/learning/lessons/L00-why-cairnkeep.md +1 -1
  9. package/docs/learning/lessons/L01-safe-trial.md +3 -3
  10. package/docs/learning/lessons/L02-installation.md +2 -2
  11. package/docs/learning/lessons/L03-first-project.md +1 -1
  12. package/docs/learning/lessons/L04-memory-fundamentals.md +20 -0
  13. package/docs/learning/lessons/L19-validated-skill-improvement.md +1 -1
  14. package/docs/learning/lessons/L20-mcp-tool-profiles.md +1 -1
  15. package/docs/learning/lessons/L21-context-packs.md +1 -1
  16. package/docs/learning/lessons/L22-native-windows.md +2 -2
  17. package/docs/learning/lessons/L23-guided-setup.md +1 -1
  18. package/docs/learning/lessons/L24-okf-exchange.md +1 -1
  19. package/docs/learning/lessons/L25-playbooks.md +1 -1
  20. package/docs/learning/lessons/L26-context-intelligence.md +1 -1
  21. package/docs/learning/lessons/L27-ecosystem-evidence.md +1 -1
  22. package/docs/learning/lessons/L28-security-assurance.md +2 -2
  23. package/docs/learning/video-scripts/V04-memory-retrieval-supplement.md +34 -0
  24. package/docs/memory-retrieval.md +174 -0
  25. package/docs/operating.md +5 -0
  26. package/docs/releasing.md +7 -0
  27. package/docs/storage.md +5 -0
  28. package/examples/eval/bundled-fake.json +1 -1
  29. package/mcp-memory-server/dist/index.js +43 -2
  30. package/mcp-memory-server/dist/memory-retrieval.js +128 -0
  31. package/mcp-memory-server/package.json +2 -0
  32. package/opencode/command/recall.md +8 -1
  33. package/package.json +1 -1
  34. package/scripts/test-memory-retrieval-docs.sh +19 -0
  35. package/scripts/test-release-sbom.sh +32 -0
  36. package/templates/playbook-agent-instructions.md.template +5 -0
package/CHANGELOG.md CHANGED
@@ -5,6 +5,27 @@ All notable user-facing changes are documented here. This project follows
5
5
 
6
6
  ## [Unreleased]
7
7
 
8
+ ## [2.23.0] - 2026-10-21
9
+
10
+ ### Memory retrieval
11
+
12
+ - Add opt-in `memory_search` hybrid lexical/identifier admission, bounded query
13
+ aliases, explicit semantic-only thresholds, deterministic ranking, duplicate
14
+ source locators and optional conflict/match explanations. Preserve existing
15
+ default responses, history, scope filters and storage layouts. Fall back to
16
+ lexical admission when optional embedding results are invalid or unavailable.
17
+ - Add a frozen continuity study against real curated Markdown files, including
18
+ the same-policy Markdown parity control and 0/100/1,000 distractors. Document
19
+ synthetic limitations and unsuccessful live-provider evaluation; update the
20
+ agent guidance, learning lesson and recording supplement.
21
+
22
+ ### CI
23
+
24
+ - Allow a retried, fully gated CI job to replace its run-local tree-addressed
25
+ release candidate instead of failing on the existing artifact name. Keep all
26
+ platform/runtime dependencies and immutable published versions unchanged; add
27
+ parsed-workflow positive and mutation controls.
28
+
8
29
  ## [2.22.5] - 2026-10-08
9
30
 
10
31
  ### Operations
@@ -19,7 +19,14 @@ Read order (canonical sources still win over derived layers):
19
19
 
20
20
  <process>
21
21
  ## 1. AgentFS
22
- `memory_search` scope `project` with $ARGUMENTS. If nothing relevant, retry scope `all`.
22
+ `memory_search` scope `project` with $ARGUMENTS. When the discovered schema
23
+ supports it, use `strategy: "hybrid"` for identifier-sensitive or mixed-wording
24
+ queries and `explain: true` to inspect matches. Supply only aliases already known
25
+ from the task. Semantic-only hybrid matches require an explicitly chosen
26
+ positive `min_score`; retain the existing semantic strategy for paraphrases
27
+ when no model-specific threshold is available. Empty results are valid.
28
+ Use broader configured scopes only when the question genuinely spans them;
29
+ do not automatically widen a search merely to force a hit.
23
30
 
24
31
  ## 2. Wiki
25
32
  Read `.planning/wiki/index.md` if present and follow links relevant to $ARGUMENTS.
package/docs/README.md CHANGED
@@ -31,6 +31,9 @@ linked guides rather than the project landing page.
31
31
 
32
32
  ## Context intelligence
33
33
 
34
+ - [Memory retrieval and the Markdown baseline](memory-retrieval.md) explains
35
+ optional hybrid admission, aliases, relevance diagnostics and reproducible
36
+ continuity evidence against searchable Markdown.
34
37
  - [Context intelligence](context-intelligence.md) connects progressive context-pack
35
38
  retrieval, the frozen benchmark, usage receipts, review-gated proposals, and
36
39
  the optional read-only OpenViking provider.
package/docs/agents.md CHANGED
@@ -27,6 +27,15 @@ constraints, recurring failures, or prior work:
27
27
  This protocol improves continuity without allowing stale memory to outrank
28
28
  code, tests, current documentation, or the user's request.
29
29
 
30
+ When the discovered tool schema supports it, `strategy: "hybrid"` is useful for
31
+ identifier-sensitive and mixed-wording queries. `explain: true` exposes selection
32
+ reasons; explicit `query_aliases` can express names already known from the task.
33
+ Do not invent aliases or broaden scopes to obtain a match. Semantic-only
34
+ paraphrase retrieval needs an explicitly chosen positive `min_score` in hybrid
35
+ mode; without a model-specific threshold, the existing semantic strategy remains
36
+ available. Empty results are valid, and ranking scores are not factual confidence.
37
+ See [memory retrieval and evidence](memory-retrieval.md).
38
+
30
39
  ## What setup installs
31
40
 
32
41
  `cairn setup` reconciles a delimited Cairnkeep block in the project's
@@ -0,0 +1,115 @@
1
+ {
2
+ "schema_version": 1,
3
+ "study": "synthetic-memory-continuity-v1",
4
+ "fixture_digest": "051a1ee175fc20d9e38c43d838b234c759133805ab335f74ef5603ee0871fce8",
5
+ "embedding": {
6
+ "mode": "none"
7
+ },
8
+ "limitations": [
9
+ "Synthetic retrieval scenarios, not autonomous task-quality or human-usability evidence.",
10
+ "Markdown baseline shares BM25/tokenization and receives identical alias information.",
11
+ "Markdown with the same admission policy is included to distinguish retrieval-policy gains from storage format.",
12
+ "Offline cases do not evaluate learned semantic paraphrase retrieval.",
13
+ "Explicit live threshold is an example, not universal calibration or a confidence probability.",
14
+ "Current-state fixture excludes archived history; actual supersession is covered by the MCP regression.",
15
+ "No automatic contradiction reconciliation, human consent or concurrent local-store access is claimed."
16
+ ],
17
+ "measurements": [
18
+ {
19
+ "adapter": "markdown-bm25",
20
+ "distractor_count": 0,
21
+ "hit_at_1": 0.8888888888888888,
22
+ "recall_at_5": 0.8888888888888888,
23
+ "negative_false_positive_rate": 0.75,
24
+ "irrelevant_bytes": 794
25
+ },
26
+ {
27
+ "adapter": "markdown-hybrid",
28
+ "distractor_count": 0,
29
+ "hit_at_1": 0.8888888888888888,
30
+ "recall_at_5": 0.8888888888888888,
31
+ "negative_false_positive_rate": 0,
32
+ "irrelevant_bytes": 0
33
+ },
34
+ {
35
+ "adapter": "legacy-substring",
36
+ "distractor_count": 0,
37
+ "hit_at_1": 0.3333333333333333,
38
+ "recall_at_5": 0.3333333333333333,
39
+ "negative_false_positive_rate": 0,
40
+ "irrelevant_bytes": 0
41
+ },
42
+ {
43
+ "adapter": "hybrid-lexical",
44
+ "distractor_count": 0,
45
+ "hit_at_1": 0.8888888888888888,
46
+ "recall_at_5": 0.8888888888888888,
47
+ "negative_false_positive_rate": 0,
48
+ "irrelevant_bytes": 0
49
+ },
50
+ {
51
+ "adapter": "markdown-bm25",
52
+ "distractor_count": 100,
53
+ "hit_at_1": 0.8888888888888888,
54
+ "recall_at_5": 0.8888888888888888,
55
+ "negative_false_positive_rate": 0.75,
56
+ "irrelevant_bytes": 3359
57
+ },
58
+ {
59
+ "adapter": "markdown-hybrid",
60
+ "distractor_count": 100,
61
+ "hit_at_1": 0.8888888888888888,
62
+ "recall_at_5": 0.8888888888888888,
63
+ "negative_false_positive_rate": 0,
64
+ "irrelevant_bytes": 0
65
+ },
66
+ {
67
+ "adapter": "legacy-substring",
68
+ "distractor_count": 100,
69
+ "hit_at_1": 0.3333333333333333,
70
+ "recall_at_5": 0.3333333333333333,
71
+ "negative_false_positive_rate": 0,
72
+ "irrelevant_bytes": 0
73
+ },
74
+ {
75
+ "adapter": "hybrid-lexical",
76
+ "distractor_count": 100,
77
+ "hit_at_1": 0.8888888888888888,
78
+ "recall_at_5": 0.8888888888888888,
79
+ "negative_false_positive_rate": 0,
80
+ "irrelevant_bytes": 0
81
+ },
82
+ {
83
+ "adapter": "markdown-bm25",
84
+ "distractor_count": 1000,
85
+ "hit_at_1": 0.8888888888888888,
86
+ "recall_at_5": 0.8888888888888888,
87
+ "negative_false_positive_rate": 0.75,
88
+ "irrelevant_bytes": 3359
89
+ },
90
+ {
91
+ "adapter": "markdown-hybrid",
92
+ "distractor_count": 1000,
93
+ "hit_at_1": 0.8888888888888888,
94
+ "recall_at_5": 0.8888888888888888,
95
+ "negative_false_positive_rate": 0,
96
+ "irrelevant_bytes": 0
97
+ },
98
+ {
99
+ "adapter": "legacy-substring",
100
+ "distractor_count": 1000,
101
+ "hit_at_1": 0.3333333333333333,
102
+ "recall_at_5": 0.3333333333333333,
103
+ "negative_false_positive_rate": 0,
104
+ "irrelevant_bytes": 0
105
+ },
106
+ {
107
+ "adapter": "hybrid-lexical",
108
+ "distractor_count": 1000,
109
+ "hit_at_1": 0.8888888888888888,
110
+ "recall_at_5": 0.8888888888888888,
111
+ "negative_false_positive_rate": 0,
112
+ "irrelevant_bytes": 0
113
+ }
114
+ ]
115
+ }
@@ -1,6 +1,6 @@
1
1
  # Curriculum Coverage Map
2
2
 
3
- **Baseline:** Cairnkeep 2.22.5
3
+ **Baseline:** Cairnkeep 2.23.0
4
4
  **Last reviewed:** 2026-10-07
5
5
 
6
6
  This map prevents the course from drifting behind the product. It assigns each
@@ -1,6 +1,6 @@
1
1
  # Cairnkeep feature guide
2
2
 
3
- **Baseline:** Cairnkeep 2.22.5
3
+ **Baseline:** Cairnkeep 2.23.0
4
4
 
5
5
  This is the step-back map for choosing and operating Cairnkeep features. The
6
6
  [course labs](https://github.com/cairnkeep/cairnkeep-course-labs) provide one
@@ -36,6 +36,12 @@ explicit native Codex exec exports. See the L16 memory-protocol supplement;
36
36
  protocol observations and independently verified task correctness remain
37
37
  separate evidence, not a claim of model improvement.
38
38
 
39
+ Version 2.23 adds opt-in hybrid memory retrieval, explicit query aliases and
40
+ match/source/conflict explanations without changing legacy defaults. See
41
+ [memory retrieval](../memory-retrieval.md), [L04](lessons/L04-memory-fundamentals.md)
42
+ and its recording supplement. The synthetic Markdown comparison measures
43
+ retrieval policy, not general task quality or a storage-format advantage.
44
+
39
45
  ## The three-layer model
40
46
 
41
47
  1. **Memory server:** an MCP process that owns scoped durable storage. The
@@ -3,7 +3,7 @@
3
3
  **Status:** Ready
4
4
  **Track:** Quickstart
5
5
  **Time:** 10 minutes
6
- **Tested with:** Cairnkeep 2.22.5
6
+ **Tested with:** Cairnkeep 2.23.0
7
7
 
8
8
  ## Outcome
9
9
 
@@ -3,7 +3,7 @@
3
3
  **Status:** Ready
4
4
  **Track:** Quickstart
5
5
  **Time:** 15 minutes
6
- **Tested with:** Cairnkeep 2.22.5
6
+ **Tested with:** Cairnkeep 2.23.0
7
7
 
8
8
  ## Outcome
9
9
 
@@ -26,7 +26,7 @@ ephemeral; the named course volume is persistent.
26
26
  1. Pull the public image:
27
27
 
28
28
  ```bash
29
- podman pull ghcr.io/cairnkeep/cairnkeep:2.22.5
29
+ podman pull ghcr.io/cairnkeep/cairnkeep:2.23.0
30
30
  ```
31
31
 
32
32
  2. Start the stdio server with a read-only root filesystem and no Linux
@@ -38,7 +38,7 @@ ephemeral; the named course volume is persistent.
38
38
  --read-only --cap-drop=all --security-opt=no-new-privileges \
39
39
  --tmpfs=/tmp:rw,noexec,nosuid,size=64m,mode=1777 \
40
40
  --volume cairnkeep-course-data:/data:Z,U \
41
- ghcr.io/cairnkeep/cairnkeep:2.22.5 stdio
41
+ ghcr.io/cairnkeep/cairnkeep:2.23.0 stdio
42
42
  ```
43
43
 
44
44
  3. The process waits quietly for MCP input. Press `Ctrl-C` after confirming it
@@ -3,7 +3,7 @@
3
3
  **Status:** Ready
4
4
  **Track:** Quickstart
5
5
  **Time:** 25 minutes
6
- **Tested with:** Cairnkeep 2.22.5 and Node.js 22 or newer
6
+ **Tested with:** Cairnkeep 2.23.0 and Node.js 22 or newer
7
7
 
8
8
  ## Outcome
9
9
 
@@ -47,7 +47,7 @@ sync. Codex project trust remains an explicit operator decision.
47
47
  2. Install Cairnkeep:
48
48
 
49
49
  ```bash
50
- npm install --global @cairnkeep/cli@2.22.5
50
+ npm install --global @cairnkeep/cli@2.23.0
51
51
  cairn --version
52
52
  ```
53
53
 
@@ -3,7 +3,7 @@
3
3
  **Status:** Ready
4
4
  **Track:** Quickstart
5
5
  **Time:** 25 minutes
6
- **Tested with:** Cairnkeep 2.22.5
6
+ **Tested with:** Cairnkeep 2.23.0
7
7
 
8
8
  ## Outcome
9
9
 
@@ -15,6 +15,10 @@ review extracted candidates before they become durable memory.
15
15
  - Key families: `decisions/`, `pitfalls/`, `patterns/`, `bugs/`,
16
16
  `constraints/`, and `conventions/`.
17
17
  - Exact read versus semantic search and substring fallback.
18
+ - Optional hybrid lexical/identifier admission, explicit aliases, match reasons
19
+ and valid empty results; similarity scores are not factual confidence.
20
+ - Curated Markdown as a credible baseline; indexing policy and storage format
21
+ are separate decisions.
18
22
  - Superseding a fact without losing history.
19
23
  - Session-end candidate extraction and `/memory-review` as the acceptance gate.
20
24
  - Typed node metadata, tags, and hard filters as an independent opt-in.
@@ -27,10 +31,24 @@ Store three synthetic facts in different categories, retrieve them by an
27
31
  imprecise query, supersede one, inspect its history, and reject one staged
28
32
  candidate.
29
33
 
34
+ From a source checkout, run:
35
+
36
+ ```sh
37
+ npm --prefix mcp-memory-server run study:memory-retrieval
38
+ ```
39
+
40
+ Compare the 0/100/1,000-distractor results and explain
41
+ why Markdown with the same admission policy matches Cairnkeep. Inspect the
42
+ semantic-only case that lexical modes miss; do not describe the synthetic
43
+ report as general accuracy. Repeat retrieval after superseding a synthetic
44
+ fact and starting a fresh client, verifying the new value and explicit history.
45
+
30
46
  ## Acceptance criteria
31
47
 
32
48
  - The learner can justify the scope and category for each fact.
33
49
  - Search finds a fact without requiring its exact key.
50
+ - A nearby but incorrect identifier and an unrelated query return no hybrid
51
+ match; aliases never widen scope or grant authority.
34
52
  - Superseding preserves the prior value in history.
35
53
  - No extracted candidate becomes durable without an explicit review action.
36
54
  - Enabling typed nodes changes discovery structure, not the authority of a fact.
@@ -38,5 +56,7 @@ candidate.
38
56
  ## Source material
39
57
 
40
58
  - [Operating guide](../../operating.md#the-workflow)
59
+ - [Memory retrieval and the Markdown baseline](../../memory-retrieval.md)
60
+ - [Recording supplement](../video-scripts/V04-memory-retrieval-supplement.md)
41
61
  - [Storage and deployment](../../storage.md)
42
62
  - [L14 - Typed memory and controlled import](L14-typed-memory.md)
@@ -3,7 +3,7 @@
3
3
  **Status:** Ready
4
4
  **Track:** Evidence and Evaluation
5
5
  **Time:** 50 minutes
6
- **Tested with:** Cairnkeep 2.22.5
6
+ **Tested with:** Cairnkeep 2.23.0
7
7
 
8
8
  ## Outcome
9
9
 
@@ -1,7 +1,7 @@
1
1
  # L20 - Least-authority MCP tool profiles
2
2
 
3
3
  **Status:** Ready
4
- **Tested with:** Cairnkeep 2.22.5 and Node.js 22 or newer
4
+ **Tested with:** Cairnkeep 2.23.0 and Node.js 22 or newer
5
5
  **Time:** 25 minutes
6
6
 
7
7
  ## Outcome
@@ -1,7 +1,7 @@
1
1
  # L21 - Immutable context packs and skill approval
2
2
 
3
3
  **Status:** Ready
4
- **Tested with:** Cairnkeep 2.22.5 and Node.js 22 or newer
4
+ **Tested with:** Cairnkeep 2.23.0 and Node.js 22 or newer
5
5
  **Time:** 40 minutes
6
6
 
7
7
  ## Outcome
@@ -3,7 +3,7 @@
3
3
  **Status:** Ready
4
4
  **Track:** Operator
5
5
  **Time:** 30 minutes
6
- **Tested with:** Cairnkeep 2.22.5, Windows x64, and Node.js 22 or newer
6
+ **Tested with:** Cairnkeep 2.23.0, Windows x64, and Node.js 22 or newer
7
7
 
8
8
  ## Outcome
9
9
 
@@ -16,7 +16,7 @@ perform a reversible uninstall without relying on WSL or Git Bash.
16
16
  From PowerShell:
17
17
 
18
18
  ```powershell
19
- npm install --global @cairnkeep/cli@2.22.5
19
+ npm install --global @cairnkeep/cli@2.23.0
20
20
  cairn version
21
21
 
22
22
  $Lab = Join-Path $env:TEMP 'Cairnkeep Windows Lab'
@@ -3,7 +3,7 @@
3
3
  **Status:** Ready
4
4
  **Track:** Quickstart and Operator
5
5
  **Time:** 35 minutes
6
- **Tested with:** Cairnkeep 2.22.5; Pi 0.84.1; Node.js 22, 24, and 26
6
+ **Tested with:** Cairnkeep 2.23.0; Pi 0.84.1; Node.js 22, 24, and 26
7
7
 
8
8
  ## Outcome
9
9
 
@@ -1,7 +1,7 @@
1
1
  # L24 - Reviewed knowledge exchange
2
2
 
3
3
  **Status:** Ready
4
- **Tested with:** Cairnkeep 2.22.5 and Node.js 22 or newer
4
+ **Tested with:** Cairnkeep 2.23.0 and Node.js 22 or newer
5
5
  **Time:** 35 minutes
6
6
 
7
7
  ## Outcome
@@ -1,7 +1,7 @@
1
1
  # L25 - Bounded workflow playbooks
2
2
 
3
3
  **Status:** Ready
4
- **Tested with:** Cairnkeep 2.22.5 and Node.js 22 or newer
4
+ **Tested with:** Cairnkeep 2.23.0 and Node.js 22 or newer
5
5
  **Time:** 35 minutes
6
6
 
7
7
  ## Outcome
@@ -1,7 +1,7 @@
1
1
  # L26 - Context intelligence with explicit authority
2
2
 
3
3
  **Status:** Ready
4
- **Tested with:** Cairnkeep 2.22.5 and Node.js 22 or newer
4
+ **Tested with:** Cairnkeep 2.23.0 and Node.js 22 or newer
5
5
 
6
6
  ## Outcome
7
7
 
@@ -1,6 +1,6 @@
1
1
  # L27 - Evaluate context companions without moving authority
2
2
 
3
- **Tested with:** Cairnkeep 2.22.5 and Node.js 22 or newer
3
+ **Tested with:** Cairnkeep 2.23.0 and Node.js 22 or newer
4
4
 
5
5
  ## Outcome
6
6
 
@@ -1,7 +1,7 @@
1
1
  # L28 - Verify Cairnkeep's security posture
2
2
 
3
3
  **Status:** Ready
4
- **Tested with:** Cairnkeep 2.22.5 and Node.js 22 or newer
4
+ **Tested with:** Cairnkeep 2.23.0 and Node.js 22 or newer
5
5
 
6
6
  ## Outcome
7
7
 
@@ -11,7 +11,7 @@ and choose least authority for a network-facing MCP client.
11
11
 
12
12
  ## Prerequisites
13
13
 
14
- - Install Cairnkeep 2.22.5 or use a disposable source checkout.
14
+ - Install Cairnkeep 2.23.0 or use a disposable source checkout.
15
15
  - Complete [L20 - Least-authority MCP tool profiles](L20-mcp-tool-profiles.md).
16
16
  - Use synthetic values only; never paste a real bearer token into a recording,
17
17
  terminal transcript, or issue.
@@ -0,0 +1,34 @@
1
+ # V04 supplement — Finding the current fact without flooding context
2
+
3
+ Technical recording brief; no video is claimed as recorded. Use the source
4
+ checkout containing hybrid memory search and only invented fixture content.
5
+
6
+ ## Real terminal demonstration
7
+
8
+ Type at readable human speed, leaving pauses after each command:
9
+
10
+ ```sh
11
+ npm --prefix mcp-memory-server run build
12
+ npm --prefix mcp-memory-server run check:memory-retrieval
13
+ npm --prefix mcp-memory-server run study:memory-retrieval
14
+ ```
15
+
16
+ Zoom into the real MCP test's new-process handoff, superseded-history exclusion,
17
+ wrong-identifier refusal and provider-failure result. Demonstrate an MCP call
18
+ with `strategy: "hybrid"` and `explain: true`; show matched terms and source
19
+ locators, then show an unrelated query returning zero results. Explain that
20
+ agents still verify maintained sources and do not automatically correct memory.
21
+
22
+ ## Evidence and pacing
23
+
24
+ Show the frozen 1,000-distractor table: 3/9 first hits for substring, 8/9 for
25
+ BM25 and hybrid. Pause on the difference in irrelevant bytes and negative hits.
26
+ Then highlight that Markdown with the same admission policy also achieves 8/9
27
+ and zero negative hits. State that storage format did not cause the gain.
28
+
29
+ Finish with the semantic-only paraphrase that lexical modes miss. Explain that
30
+ provider-specific thresholds need their own evidence, that a ranking score is
31
+ not confidence, and that the attempted live study failed with HTTP 502. Do not
32
+ invent a successful live result or claim general task-quality improvements.
33
+ Use short terminal zooms and side-by-side result comparisons instead of static
34
+ slides; keep credentials and real memory off screen.
@@ -0,0 +1,174 @@
1
+ # Memory retrieval and the Markdown baseline
2
+
3
+ Status: available since Cairnkeep 2.23.0 as an opt-in strategy. Existing search
4
+ defaults are unchanged; the developer study commands require a source checkout.
5
+
6
+ Cairnkeep keeps authoritative memory text readable. Ordinary memory lives in
7
+ AgentFS-backed SQLite; derived note address spaces and wiki pages use files.
8
+ Optional embeddings are a rebuildable search aid, not fine-tuning, encrypted
9
+ storage, or a replacement for the original text. Semantic search computes
10
+ missing vectors lazily and caches them outside the authoritative records.
11
+ Configured providers receive memory text and queries; an endpoint can run
12
+ locally or remotely. See [storage](storage.md).
13
+
14
+ ## Opt-in hybrid search
15
+
16
+ Existing `memory_search` calls retain legacy behavior and response shapes.
17
+ Clients can explicitly select the new strategy through MCP:
18
+
19
+ ```json
20
+ {
21
+ "scope": "project",
22
+ "query": "office tunnel access",
23
+ "strategy": "hybrid",
24
+ "top_k": 5,
25
+ "explain": true
26
+ }
27
+ ```
28
+
29
+ The result has `mode: "hybrid"`, ordinary original-text `results`, and a
30
+ `retrieval` diagnostic. Empty results are valid: `empty_reason` distinguishes
31
+ an empty scope from `no_relevant_memory`. `embedding_status` distinguishes
32
+ unconfigured, available, failed and not-used providers. Ranking scores are
33
+ labelled `ranking-not-confidence`; neither a cosine nor a lexical score is a
34
+ probability that a statement is true.
35
+
36
+ The new strategy currently supports the `memory` address space. File-backed
37
+ note searches retain their existing exact behavior; requesting hybrid there
38
+ fails explicitly. Existing `memory_read` key/query semantics are unchanged.
39
+
40
+ ### Selection and ranking
41
+
42
+ 1. Resolve explicit scopes and apply enabled typed-node hard filters before
43
+ ranking or embedding. Neither aliases nor semantic similarity widens scope.
44
+ 2. Tokenize Unicode words, ignore a small source-defined set of common
45
+ English/Italian stop words, and rank with BM25 (key terms appear twice).
46
+ Query terms are capped at 64; tokenization is capped at 4,096 terms per field.
47
+ 3. Admit lexical matches covering at least 60% of significant query terms,
48
+ with at least two matched terms for multi-term queries or one for a single
49
+ term. An exact key is admitted directly. Compound identifiers and numeric
50
+ identifiers must match exactly within at least one explicit query variant.
51
+ Match complete identifiers, not prefixes: `ember-42` does not match
52
+ `ember-42-backup`, and `4.2.1` does not match `4.2.1-rc1`. Sentence-ending
53
+ punctuation is not part of an identifier.
54
+ 4. With embeddings available, semantic scores contribute to ranking. A match
55
+ based only on semantics additionally requires an **explicit positive**
56
+ `min_score`. Omitted or zero `min_score` never admits semantic-only matches.
57
+ Choose that threshold using examples from the configured model and corpus;
58
+ no universal cosine cutoff is supplied or claimed as calibrated confidence.
59
+ 5. Put exact keys first, then sort by 80% normalized BM25 contribution plus
60
+ 20% nonnegative cosine contribution. Break ties by scope/key deterministically.
61
+ Collapse identical key/value pairs; keep different values separate.
62
+
63
+ `top_k` is a maximum, not a quota to fill. Provider failure, incomplete vectors,
64
+ zero vectors or invalid dimensions fall back to lexical admission. Failure
65
+ diagnostics include no provider response bodies. This does not change the
66
+ legacy strategy's existing fallback semantics.
67
+
68
+ With `explain: true`, each result includes `match.reasons`, matched terms,
69
+ lexical coverage, the optional semantic score, and source scope/key locators.
70
+ Identical records retain all source locators. `possible_conflicts` identifies
71
+ admitted records sharing a key but having different live values across scopes.
72
+ That is a review aid, not a finding that one value is false or an instruction
73
+ to merge them. Explanations exist only in the response; no new recall log is kept.
74
+
75
+ ### Explicit aliases
76
+
77
+ Aliases can be curated into ordinary memory text, or supplied as bounded query
78
+ alternatives:
79
+
80
+ ```json
81
+ {
82
+ "scope": "project",
83
+ "query": "home connection",
84
+ "strategy": "hybrid",
85
+ "query_aliases": ["office tunnel access", "relaybox"],
86
+ "explain": true
87
+ }
88
+ ```
89
+
90
+ At most eight aliases of 256 UTF-8 bytes each are accepted. The original hybrid
91
+ query is bounded to 4 KiB. Invalid bounds fail before opening scopes or making
92
+ provider requests. Aliases affect lexical selection; the original query is
93
+ embedded once. They are caller-supplied hints and may explicitly substitute an
94
+ identifier, not automatically inferred identities or additional authority.
95
+ `query_aliases` and `explain` require `strategy: "hybrid"`.
96
+
97
+ ## Current values, history and authority
98
+
99
+ Search reads live records and excludes hidden supersession history. Updating a
100
+ fact through the existing supersession workflow makes the new value current
101
+ and preserves the displaced value for explicit history inspection. A new MCP
102
+ process can retrieve the live value after the previous local process exits.
103
+ Simultaneous clients should use one shared server: independent direct local
104
+ AgentFS processes can contend for native database locks.
105
+
106
+ Current means current in the store, not independently verified or still true
107
+ in the real world. No automatic expiry, contradiction resolution, history
108
+ deletion, knowledge promotion or durable memory mutation is introduced. Verify
109
+ maintained sources and supersede stale facts only through an authorized workflow.
110
+ Personal scopes in a trusted deployment are not a multi-user ACL mechanism.
111
+ The separate team preview is unchanged.
112
+
113
+ ## A fair Markdown comparison
114
+
115
+ A curated Markdown folder provides readable durable knowledge, human editing,
116
+ navigation and Git history. Adding the same retrieval policy can reproduce the
117
+ same retrieval results. Cairnkeep provides a maintained shared MCP interface,
118
+ scope routing, lifecycle operations and harness integrations; SQLite itself is
119
+ not evidence of better recall.
120
+
121
+ The reproducible study builds real temporary Markdown files and equivalent
122
+ memory candidates with identical keys, values, scopes and explicit aliases.
123
+ It compares a searchable-folder BM25 baseline, the legacy substring policy,
124
+ the new admission policy and the same new policy applied to Markdown. Both
125
+ folder variants receive the same tokenizer/BM25 implementation; the baseline
126
+ is deliberately strong, and the Markdown parity control distinguishes search
127
+ policy gains from storage format. The substring comparator unions explicit
128
+ query variants, equivalent to separate legacy requests.
129
+
130
+ Thirteen declared scenarios include a changed tunnel configuration, aliases,
131
+ Italian queries, continuation conventions, release policy, unrelated queries,
132
+ nearby/obsolete identifiers and wrong-project content, with 0/100/1,000
133
+ distractors. The [frozen report](benchmarks/memory-retrieval-v1.json) shows:
134
+
135
+ | Adapter, 1,000 distractors | Correct first result, 9 positive cases | Negative queries returning results, 4 cases | Irrelevant value bytes |
136
+ |---|---:|---:|---:|
137
+ | Legacy substring | 3/9 | 0/4 | 0 |
138
+ | Markdown + BM25 | 8/9 | 3/4 | 3,359 |
139
+ | Hybrid lexical admission | 8/9 | 0/4 | 0 |
140
+ | Markdown + same admission | 8/9 | 0/4 | 0 |
141
+
142
+ These are synthetic regression cases, not general retrieval accuracy, an
143
+ independent holdout, autonomous task quality or proof of human consent. The
144
+ semantic-only paraphrase is deliberately missed by lexical modes; the report
145
+ does not claim perfect recall. A real MCP control separately verifies actual
146
+ supersession, new-process handoff, scope filtering before provider requests and
147
+ provider-failure behavior. No live user memory is read by these checks.
148
+
149
+ From a source checkout:
150
+
151
+ ```sh
152
+ npm --prefix mcp-memory-server run build
153
+ npm --prefix mcp-memory-server run check:memory-retrieval
154
+ npm --prefix mcp-memory-server run study:memory-retrieval
155
+ # Explicit provider evaluation, using synthetic fixtures only:
156
+ node mcp-memory-server/scripts/run-memory-retrieval-study.mjs --live --summary
157
+ ```
158
+
159
+ The live study reports a model-ID digest, not credentials or endpoint addresses.
160
+ Its sample threshold of 0.75 is an evaluation arm, not a recommended default.
161
+ An attempted live run during development accepted short probes but failed with
162
+ HTTP 502 on some fixture inputs. No successful live-model result is claimed.
163
+ The offline suite never requires a model, API key, paid service or network beyond
164
+ its own loopback fixture. An optional learned reranker remains deferred until
165
+ provider-specific evidence demonstrates a benefit beyond this simpler policy.
166
+
167
+ ## Research connection
168
+
169
+ [Alex Wang's associative-memory article](https://alexwang.ai/posts/associative-memory/)
170
+ provides a useful perspective on recall interference. Its neural-layer capacity
171
+ experiments do not establish limits for a text database. This implementation
172
+ applies selective retrieval at the application layer without changing model
173
+ weights. Merely applying softmax to an existing score list preserves ordering
174
+ and would not improve fixed-top-k selection.
package/docs/operating.md CHANGED
@@ -32,6 +32,11 @@ sync command but never runs it automatically. This guide covers all three.
32
32
  - Optional: an OpenAI-compatible LLM endpoint for memory extraction and
33
33
  embedding-ranked search. Without it, memory search degrades to substring
34
34
  matching — everything else still works.
35
+ Explicit `memory_search` calls with `strategy: "hybrid"` additionally support
36
+ lexical/identifier admission without a provider, bounded `query_aliases`,
37
+ optional explanations and empty results. Semantic-only admission requires an
38
+ explicit positive `min_score`; see [memory retrieval](memory-retrieval.md)
39
+ for model-specific threshold limits and the Markdown comparison.
35
40
  - Optional: rootless Podman for the containerized memory server and isolated
36
41
  workspace base described in [Containers](containers.md).
37
42
 
package/docs/releasing.md CHANGED
@@ -66,6 +66,13 @@ or moved. If the tree-addressed candidate has expired, rerun CI for the exact
66
66
  release commit before retrying publication; the workflow never substitutes a
67
67
  different version or tree.
68
68
 
69
+ CI job retries can also regenerate the same tree-addressed candidate. Its
70
+ upload explicitly replaces an existing artifact of that name within the same
71
+ workflow run, only after all platform/runtime dependencies succeed. This avoids
72
+ an artifact-name conflict after a completed job is retried; it does not move
73
+ versioned image tags, republish npm versions or replace GitHub Release assets.
74
+ Candidate checksums and tree/version binding are verified as usual.
75
+
69
76
  ## Repository configuration
70
77
 
71
78
  ### Repeatable post-publication verification
package/docs/storage.md CHANGED
@@ -48,6 +48,11 @@ services used to process/search memory. They do not change where the SQLite
48
48
  databases are stored. Git-provider and routing configuration do not change
49
49
  storage either.
50
50
 
51
+ `memory_search` can optionally select [hybrid retrieval](memory-retrieval.md).
52
+ This changes read-time ranking only: aliases are request hints, explanations
53
+ are returned to the caller without a new log, and original memory text remains
54
+ authoritative. No new store, migration or automatic memory capture is introduced.
55
+
51
56
  Trajectory storage is deliberately separate from MCP memory. The harness hook
52
57
  writes `.agentfs/trajectory.db` on the client machine where Claude Code,
53
58
  OpenCode, or Pi is running; it is never redirected to the remote HTTP memory server.
@@ -1 +1 @@
1
- {"identifier":"cairn-offline-fake-v1","package_version":"2.22.5","schema_version":1,"task_set_digest":"13e95379f35839593047ec34521a65a8fa3ab92bd3993bca9dc8bf5c4a9ff36f"}
1
+ {"identifier":"cairn-offline-fake-v1","package_version":"2.23.0","schema_version":1,"task_set_digest":"13e95379f35839593047ec34521a65a8fa3ab92bd3993bca9dc8bf5c4a9ff36f"}
@@ -25,6 +25,7 @@ import { profileAllowsTool, resolveMcpToolProfile } from "./mcp-tool-profile.js"
25
25
  import { isWorkEvidenceEnabled } from "./work-evidence-schema.js";
26
26
  import { extractMemoryCandidates as extractMemoryCandidatesFromProvider } from "./memory-extraction.js";
27
27
  import { applyReviewedMemory, invalidateReviewedMemory } from "./reviewed-memory-store.js";
28
+ import { rankHybridMemory } from "./memory-retrieval.js";
28
29
  const MINIMUM_NODE_MAJOR = 22;
29
30
  const nodeMajor = Number.parseInt(process.versions.node, 10);
30
31
  if (!Number.isInteger(nodeMajor) || nodeMajor < MINIMUM_NODE_MAJOR) {
@@ -377,7 +378,14 @@ async function semanticSearch(scope, query, topK, minScore, config = getMemoryCo
377
378
  entries: filterEntries(await listEntries(candidate, "", { projectId, typed: options.typed }), options.filters ?? {}),
378
379
  })));
379
380
  const allEntries = perScopeEntries.flatMap((group) => group.entries);
381
+ const hybrid = (embeddingStatus, semanticScores) => ({
382
+ ...rankHybridMemory(allEntries, query, { topK, minScore, aliases: options.aliases, explain: options.explain, embeddingStatus, semanticScores }),
383
+ mode: "hybrid",
384
+ ...(embeddingConfig ? { model: embeddingConfig.model } : {}),
385
+ });
380
386
  if (allEntries.length === 0) {
387
+ if (options.strategy === "hybrid")
388
+ return hybrid(embeddingConfig ? "not_used" : "unconfigured");
381
389
  return { results: [], mode: embeddingConfig ? "semantic" : "substring" };
382
390
  }
383
391
  const substringFallback = () => {
@@ -390,6 +398,8 @@ async function semanticSearch(scope, query, topK, minScore, config = getMemoryCo
390
398
  };
391
399
  };
392
400
  if (!embeddingConfig) {
401
+ if (options.strategy === "hybrid")
402
+ return hybrid("unconfigured");
393
403
  return substringFallback();
394
404
  }
395
405
  try {
@@ -422,9 +432,23 @@ async function semanticSearch(scope, query, topK, minScore, config = getMemoryCo
422
432
  }
423
433
  }
424
434
  const [queryVector] = await embedTexts(embeddingConfig, [query]);
435
+ if (options.strategy === "hybrid") {
436
+ const validVector = (vector) => {
437
+ if (!vector?.length || vector.length > 65536 || vector.some(value => !Number.isFinite(value)))
438
+ return false;
439
+ const squaredNorm = vector.reduce((total, value) => total + value * value, 0);
440
+ return squaredNorm > 0 && Number.isFinite(squaredNorm);
441
+ };
442
+ if (!validVector(queryVector) || allEntries.some(entry => !validVector(vectors.get(entry)) || vectors.get(entry).length !== queryVector.length)) {
443
+ throw new Error("Embedding vectors are incomplete or invalid.");
444
+ }
445
+ }
425
446
  for (const cache of caches) {
426
447
  cache.save();
427
448
  }
449
+ if (options.strategy === "hybrid") {
450
+ return hybrid("available", new Map(allEntries.map(entry => [entry, Math.max(-1, Math.min(1, cosineSimilarity(queryVector, vectors.get(entry))))])));
451
+ }
428
452
  const exact = options.typed
429
453
  ? allEntries.filter((entry) => isExactTextHit(entry, query)).sort(compareMemoryEntries).map((entry) => ({ ...entry, score: 1 }))
430
454
  : [];
@@ -444,6 +468,8 @@ async function semanticSearch(scope, query, topK, minScore, config = getMemoryCo
444
468
  }
445
469
  catch {
446
470
  // Embedding endpoint failure — degrade gracefully to substring matching.
471
+ if (options.strategy === "hybrid")
472
+ return hybrid("failed");
447
473
  return substringFallback();
448
474
  }
449
475
  }
@@ -1112,6 +1138,9 @@ function buildMemoryServer(context, capabilitySnapshot, withCapability, toolProf
1112
1138
  query: z.string().min(1),
1113
1139
  top_k: z.number().int().min(1).max(50).optional(),
1114
1140
  min_score: z.number().min(0).max(1).optional(),
1141
+ strategy: z.enum(["legacy", "hybrid"]).optional().describe("Default legacy preserves current retrieval. Hybrid admits lexical/identifier matches and semantic-only matches above an explicit positive min_score."),
1142
+ query_aliases: z.array(z.string().trim().min(1).max(256)).max(8).optional().describe("Explicit alternative query phrases for hybrid lexical retrieval; never widens scopes."),
1143
+ explain: z.boolean().optional().describe("Include hybrid match reasons, provenance locators and possible conflicting live values."),
1115
1144
  ...(typedNodesEnabled ? {
1116
1145
  address_space: NODE_ADDRESS_SPACE_SCHEMA.optional(),
1117
1146
  node_types: z.array(nodeTypeSchema).min(1).optional(),
@@ -1124,8 +1153,17 @@ function buildMemoryServer(context, capabilitySnapshot, withCapability, toolProf
1124
1153
  },
1125
1154
  }, async (input) => {
1126
1155
  const { scope, query, top_k, min_score } = input;
1156
+ if (input.strategy !== "hybrid" && (input.query_aliases !== undefined || input.explain !== undefined)) {
1157
+ throw new Error("query_aliases and explain require strategy='hybrid'.");
1158
+ }
1159
+ if (input.strategy === "hybrid") {
1160
+ // Validate bounds before opening scopes or contacting an embedding service.
1161
+ rankHybridMemory([], query, { topK: top_k ?? 8, minScore: min_score ?? 0, aliases: input.query_aliases });
1162
+ }
1127
1163
  const addressSpace = "address_space" in input && input.address_space !== undefined ? NODE_ADDRESS_SPACE_SCHEMA.parse(input.address_space) : undefined;
1128
1164
  if (addressSpace && addressSpace !== "memory") {
1165
+ if (input.strategy === "hybrid")
1166
+ throw new Error("Hybrid retrieval currently supports the memory address space only.");
1129
1167
  const noteStore = await import("./note-store.js");
1130
1168
  const entries = noteStore.searchAddressedNotes({ ...noteTargetOptions(scope, addressSpace), query, ...memoryFiltersFromInput(input) }).slice(0, top_k ?? 8);
1131
1169
  const results = entries.map((entry) => ({ ...entry, score: 1 }));
@@ -1135,11 +1173,14 @@ function buildMemoryServer(context, capabilitySnapshot, withCapability, toolProf
1135
1173
  structuredContent: payload,
1136
1174
  };
1137
1175
  }
1138
- const { results, mode, model } = await semanticSearch(scope, query, top_k ?? 8, min_score ?? 0, memoryConfig(), context.projectId, {
1176
+ const { results, mode, model, retrieval } = await semanticSearch(scope, query, top_k ?? 8, min_score ?? 0, memoryConfig(), context.projectId, {
1139
1177
  typed: typedNodesEnabled,
1140
1178
  filters: typedNodesEnabled ? memoryFiltersFromInput(input) : {},
1179
+ strategy: input.strategy,
1180
+ aliases: input.query_aliases,
1181
+ explain: input.explain,
1141
1182
  });
1142
- const payload = { mode, model, count: results.length, results };
1183
+ const payload = { mode, model, count: results.length, results, ...(retrieval ? { retrieval } : {}) };
1143
1184
  return {
1144
1185
  content: [{ type: "text", text: asToolText(payload) }],
1145
1186
  structuredContent: payload,
@@ -0,0 +1,128 @@
1
+ const STOP = new Set('a an and are as at be by can do does for from how i in is it me my of on or our please that the their this to use we what when where which with would you your un una uno il lo la gli le di da del della e che con come per posso mio mia sul nel dei'.split(' '));
2
+ const normalize = (value) => value.normalize('NFKC').toLowerCase();
3
+ const compare = (left, right) => left.scope.localeCompare(right.scope, 'en') || left.key.localeCompare(right.key, 'en');
4
+ const rounded = (value) => Math.round(value * 1e6) / 1e6;
5
+ export function retrievalTerms(text) {
6
+ const result = [];
7
+ for (const match of normalize(text).matchAll(/[\p{L}\p{N}]+/gu)) {
8
+ if (match[0].length > 1 && !STOP.has(match[0]))
9
+ result.push(match[0]);
10
+ if (result.length === 4096)
11
+ break;
12
+ }
13
+ return result;
14
+ }
15
+ function identifiers(text) {
16
+ // Match ordinary words too, then select identifiers. Requiring a separator
17
+ // in an unanchored greedy expression repeatedly rescans long plain words.
18
+ const result = new Set();
19
+ for (const match of text.matchAll(/[\p{L}\p{N}]+(?:[._:/-][\p{L}\p{N}]+)*/gu)) {
20
+ if (/[._:/\d-]/u.test(match[0]))
21
+ result.add(match[0]);
22
+ }
23
+ return result;
24
+ }
25
+ function variant(text) {
26
+ const normalized = normalize(text).trim();
27
+ return { normalized, terms: [...new Set(retrievalTerms(text))].slice(0, 64), identifiers: identifiers(normalized) };
28
+ }
29
+ /** BM25 is also used by the transparent Markdown-folder baseline. */
30
+ export function lexicalMemoryScores(entries, query) {
31
+ const queryTerms = [...new Set(retrievalTerms(query))].slice(0, 64);
32
+ const docs = entries.map(entry => [
33
+ ...retrievalTerms(entry.key), ...retrievalTerms(entry.key),
34
+ ...retrievalTerms(entry.value), ...retrievalTerms([entry.node_type, ...(entry.tags ?? [])].filter(Boolean).join(' ')),
35
+ ]);
36
+ const frequencies = docs.map(terms => {
37
+ const counts = new Map();
38
+ for (const term of terms)
39
+ counts.set(term, (counts.get(term) ?? 0) + 1);
40
+ return counts;
41
+ });
42
+ const averageLength = docs.reduce((total, terms) => total + terms.length, 0) / Math.max(1, docs.length) || 1;
43
+ const documentFrequencies = new Map(queryTerms.map(term => [term, frequencies.filter(counts => counts.has(term)).length]));
44
+ return docs.map((terms, index) => queryTerms.reduce((score, term) => {
45
+ const frequency = frequencies[index].get(term) ?? 0;
46
+ const df = documentFrequencies.get(term) ?? 0;
47
+ const idf = Math.log(1 + (entries.length - df + 0.5) / (df + 0.5));
48
+ return score + idf * frequency * 2.2 / (frequency + 1.2 * (0.25 + 0.75 * terms.length / averageLength));
49
+ }, 0));
50
+ }
51
+ export function rankHybridMemory(entries, query, options) {
52
+ const aliases = options.aliases ?? [];
53
+ if (Buffer.byteLength(query, 'utf8') > 4096 || !query.trim() || aliases.length > 8
54
+ || aliases.some(alias => !alias.trim() || Buffer.byteLength(alias, 'utf8') > 256)
55
+ || !Number.isInteger(options.topK) || options.topK < 1 || options.topK > 50
56
+ || !Number.isFinite(options.minScore) || options.minScore < 0 || options.minScore > 1) {
57
+ throw new Error('Hybrid memory search requires bounded query, aliases, result count and similarity threshold.');
58
+ }
59
+ const variants = [...new Set([query, ...aliases].map(text => normalize(text).trim()))].map(variant);
60
+ const lexicalScores = variants.map(item => lexicalMemoryScores(entries, item.normalized));
61
+ const candidates = entries.map((entry, index) => {
62
+ const text = normalize([entry.key, entry.value, entry.node_type, ...(entry.tags ?? [])].filter(Boolean).join('\n'));
63
+ const words = new Set(retrievalTerms(text));
64
+ const entryIdentifiers = identifiers(text);
65
+ const matches = variants.map((item, v) => {
66
+ const matched = item.terms.filter(term => words.has(term));
67
+ const coverage = item.terms.length ? matched.length / item.terms.length : 0;
68
+ const identifierMatch = [...item.identifiers].every(id => entryIdentifiers.has(id));
69
+ const exactKey = normalize(entry.key) === item.normalized;
70
+ return {
71
+ matched, coverage, identifierMatch, exactKey,
72
+ eligible: identifierMatch && (exactKey || (coverage >= 0.6 && matched.length >= Math.min(2, item.terms.length) && matched.length > 0)),
73
+ lexical: lexicalScores[v][index],
74
+ };
75
+ });
76
+ const best = matches.filter(match => match.eligible).sort((a, b) => Number(b.exactKey) - Number(a.exactKey) || b.lexical - a.lexical)[0];
77
+ const rawSemantic = options.semanticScores?.get(entry);
78
+ const semantic = typeof rawSemantic === 'number' && Number.isFinite(rawSemantic) && rawSemantic >= -1 && rawSemantic <= 1 ? rawSemantic : null;
79
+ const semanticAdmission = options.minScore > 0 && semantic !== null && semantic >= options.minScore && matches.some(match => match.identifierMatch);
80
+ const lexical = best?.lexical ?? 0;
81
+ const score = 0.8 * lexical / (1 + lexical) + 0.2 * Math.max(0, semantic ?? 0);
82
+ return { entry, best, semantic, semanticAdmission, score };
83
+ }).filter(item => item.best || item.semanticAdmission)
84
+ .sort((a, b) => Number(Boolean(b.best?.exactKey)) - Number(Boolean(a.best?.exactKey)) || b.score - a.score || compare(a.entry, b.entry));
85
+ const grouped = new Map();
86
+ for (const candidate of candidates) {
87
+ const key = JSON.stringify([candidate.entry.key, candidate.entry.value]);
88
+ const group = grouped.get(key) ?? [];
89
+ group.push(candidate);
90
+ grouped.set(key, group);
91
+ }
92
+ const results = [...grouped.values()].slice(0, options.topK).map(group => {
93
+ const item = group[0];
94
+ const reasons = [
95
+ ...(item.best?.exactKey ? ['exact-key'] : []),
96
+ ...(item.best ? ['lexical-coverage'] : []),
97
+ ...(item.semanticAdmission ? ['semantic-threshold'] : []),
98
+ ...(group.length > 1 ? ['identical-records-collapsed'] : []),
99
+ ];
100
+ return {
101
+ ...item.entry, score: rounded(item.score),
102
+ ...(options.explain ? { match: {
103
+ reasons, matched_terms: item.best?.matched ?? [], lexical_coverage: rounded(item.best?.coverage ?? 0),
104
+ semantic_score: item.semantic === null ? null : rounded(item.semantic),
105
+ sources: group.map(({ entry }) => ({ scope: entry.scope, key: entry.key })).sort(compare),
106
+ } } : {}),
107
+ };
108
+ });
109
+ const conflicts = [...new Set(results.map(item => item.key))].flatMap(key => {
110
+ const relevant = candidates.filter(item => item.entry.key === key);
111
+ return new Set(relevant.map(item => item.entry.value)).size > 1
112
+ ? [{ key, scopes: [...new Set(relevant.map(item => item.entry.scope))].sort() }] : [];
113
+ });
114
+ return {
115
+ results,
116
+ retrieval: {
117
+ strategy: 'hybrid',
118
+ score_kind: 'ranking-not-confidence',
119
+ embedding_status: options.embeddingStatus ?? (options.semanticScores ? 'available' : 'unconfigured'),
120
+ semantic_min_score: options.minScore > 0 ? options.minScore : null,
121
+ alias_count: aliases.length,
122
+ candidates: entries.length,
123
+ admitted: grouped.size,
124
+ ...(results.length === 0 ? { empty_reason: entries.length ? 'no_relevant_memory' : 'empty_scope' } : {}),
125
+ ...(options.explain ? { possible_conflicts: conflicts } : {}),
126
+ },
127
+ };
128
+ }
@@ -15,6 +15,8 @@
15
15
  "check:embeddings": "node scripts/smoke-embeddings.mjs",
16
16
  "check:extract": "node scripts/smoke-extract-cli.mjs",
17
17
  "check:search": "node scripts/smoke-search-e2e.mjs",
18
+ "check:memory-retrieval": "node scripts/smoke-memory-retrieval.mjs && node scripts/smoke-memory-retrieval-mcp.mjs && node scripts/run-memory-retrieval-study.mjs --check --summary",
19
+ "study:memory-retrieval": "npm run build && node scripts/run-memory-retrieval-study.mjs --summary",
18
20
  "check:scope-guard": "node scripts/smoke-scope-guard.mjs",
19
21
  "check:http-guard": "node scripts/smoke-http-guard.mjs",
20
22
  "check:remote-context": "node scripts/smoke-remote-context.mjs",
@@ -25,7 +25,14 @@ Read order (canonical sources still win over derived layers):
25
25
 
26
26
  <process>
27
27
  ## 1. AgentFS
28
- `cairn-memory_memory_search` scope `project` with $ARGUMENTS. If nothing relevant, retry scope `all`.
28
+ `cairn-memory_memory_search` scope `project` with $ARGUMENTS. When the discovered
29
+ schema supports it, use `strategy: "hybrid"` for identifier-sensitive or
30
+ mixed-wording queries and `explain: true` to inspect matches. Supply only aliases
31
+ already known from the task. Semantic-only hybrid matches require an explicitly
32
+ chosen positive `min_score`; retain the existing semantic strategy for paraphrases
33
+ when no model-specific threshold is available. Empty results are valid.
34
+ Use broader configured scopes only when the question genuinely spans them;
35
+ do not automatically widen a search merely to force a hit.
29
36
 
30
37
  ## 2. Wiki
31
38
  Read `.planning/wiki/index.md` if present and follow links relevant to $ARGUMENTS.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@cairnkeep/cli",
3
- "version": "2.22.5",
3
+ "version": "2.23.0",
4
4
  "description": "A durable, harness-agnostic memory + context layer for coding agents and MCP clients.",
5
5
  "license": "Apache-2.0",
6
6
  "type": "module",
@@ -0,0 +1,19 @@
1
+ #!/usr/bin/env bash
2
+ set -euo pipefail
3
+ ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
4
+ cd "$ROOT"
5
+ for path in docs/memory-retrieval.md docs/benchmarks/memory-retrieval-v1.json docs/learning/video-scripts/V04-memory-retrieval-supplement.md; do
6
+ [[ -f "$path" ]]
7
+ done
8
+ for path in docs/README.md docs/operating.md docs/storage.md docs/agents.md docs/learning/lessons/L04-memory-fundamentals.md; do
9
+ grep -qF 'memory-retrieval.md' "$path"
10
+ done
11
+ for path in claude/commands/recall.md opencode/command/recall.md; do
12
+ grep -qF 'do not automatically widen' "$path"
13
+ grep -qF 'strategy: "hybrid"' "$path"
14
+ done
15
+ grep -qF 'same admission' docs/memory-retrieval.md
16
+ grep -qF 'HTTP 502' docs/memory-retrieval.md
17
+ grep -qF 'ranking-not-confidence' docs/memory-retrieval.md
18
+ grep -qF 'Status: available since Cairnkeep 2.23.0' docs/memory-retrieval.md
19
+ echo 'PASS: hybrid retrieval, Markdown parity, provider limitations and learning/recording references'
@@ -14,6 +14,8 @@ fail() {
14
14
 
15
15
  node - "$ROOT/package.json" "$ROOT/package-lock.json" "$CANDIDATE_WORKFLOW" "$PUBLISH_WORKFLOW" <<'NODE'
16
16
  const fs = require("node:fs");
17
+ const assert = require("node:assert/strict");
18
+ const { parse } = require("yaml");
17
19
 
18
20
  const manifest = JSON.parse(fs.readFileSync(process.argv[2], "utf8"));
19
21
  const lock = JSON.parse(fs.readFileSync(process.argv[3], "utf8"));
@@ -36,6 +38,36 @@ if (!candidateWorkflow.includes("release-candidate:")
36
38
  || !candidateWorkflow.includes("name: release-candidate-${{ steps.candidate.outputs.tree }}")) {
37
39
  throw new Error("CI must build and upload a tree-addressed release candidate after its required jobs");
38
40
  }
41
+ function validateCandidateRetry(workflow) {
42
+ const job = workflow.jobs["release-candidate"];
43
+ assert.deepEqual(new Set(job.needs), new Set([
44
+ "memory-server", "native-windows", "repository", "shell-portability",
45
+ "server-boot", "container", "fresh-install-macos",
46
+ ]), "candidate retry must retain every platform/runtime gate");
47
+ assert.equal(job.if, undefined, "candidate retry must not bypass failed or pending gates");
48
+ const uploads = job.steps.filter(step => step.uses?.startsWith("actions/upload-artifact@"));
49
+ assert.equal(uploads.length, 1, "candidate retry must have one artifact publication");
50
+ assert.equal(uploads[0].with.name, "release-candidate-${{ steps.candidate.outputs.tree }}");
51
+ assert.equal(uploads[0].with.overwrite, true, "candidate upload must support same-run retries");
52
+ assert.equal(uploads[0].with["if-no-files-found"], "error");
53
+ }
54
+ const candidate = parse(candidateWorkflow);
55
+ const positive = structuredClone(candidate);
56
+ positive.jobs["release-candidate"].steps.find(step => step.uses?.startsWith("actions/upload-artifact@")).with.overwrite = true;
57
+ validateCandidateRetry(positive);
58
+ for (const overwrite of [undefined, false]) {
59
+ const invalid = structuredClone(positive);
60
+ invalid.jobs["release-candidate"].steps.find(step => step.uses?.startsWith("actions/upload-artifact@")).with.overwrite = overwrite;
61
+ assert.throws(() => validateCandidateRetry(invalid), /same-run retries/);
62
+ }
63
+ const missingGate = structuredClone(positive);
64
+ missingGate.jobs["release-candidate"].needs = missingGate.jobs["release-candidate"].needs.filter(name => name !== "native-windows");
65
+ assert.throws(() => validateCandidateRetry(missingGate), /every platform\/runtime gate/);
66
+ const bypass = structuredClone(positive);
67
+ bypass.jobs["release-candidate"].if = "always()";
68
+ assert.throws(() => validateCandidateRetry(bypass), /must not bypass/);
69
+ validateCandidateRetry(candidate);
70
+ console.log("PASS: retry-safe candidate upload and four gate/replacement mutation controls");
39
71
  if (!publishWorkflow.includes("- name: Download the CI-verified release candidate")
40
72
  || !publishWorkflow.includes("jq -r '.conclusion'")
41
73
  || !publishWorkflow.includes("== success")
@@ -10,6 +10,11 @@ inventory, or search commands such as `pwd`, `ls`, `rg --files`, `find`,
10
10
  - If `memory_search` is available, derive one short query from the task and
11
11
  search `scope: project` once. Derive it from the user's task; do not inspect
12
12
  the repository first merely to formulate the query.
13
+ - When the discovered schema supports hybrid search, use it for explicit
14
+ identifiers or mixed wording; request explanations when needed. Supply only
15
+ aliases already known from the task. Hybrid semantic-only matches require an
16
+ explicit model-specific positive similarity threshold; otherwise keep the
17
+ existing semantic strategy for paraphrases. Empty results are acceptable.
13
18
  - Treat returned memory as a locator, not authority. Read and verify the
14
19
  maintained repository sources it references before implementing a result.
15
20
  - If the tool is unavailable or no relevant result exists, continue with