@cairnkeep/cli 2.22.4 → 2.23.0

package/docs/memory-retrieval.md

@@ -0,0 +1,174 @@
+# Memory retrieval and the Markdown baseline
+
+Status: available since Cairnkeep 2.23.0 as an opt-in strategy. Existing search
+defaults are unchanged; the developer study commands require a source checkout.
+
+Cairnkeep keeps authoritative memory text readable. Ordinary memory lives in
+AgentFS-backed SQLite; derived note address spaces and wiki pages use files.
+Optional embeddings are a rebuildable search aid, not fine-tuning, encrypted
+storage, or a replacement for the original text. Semantic search computes
+missing vectors lazily and caches them outside the authoritative records.
+Configured providers receive memory text and queries; an endpoint can run
+locally or remotely. See [storage](storage.md).
+
+## Opt-in hybrid search
+
+Existing `memory_search` calls retain legacy behavior and response shapes.
+Clients can explicitly select the new strategy through MCP:
+
+```json
+{
+  "scope": "project",
+  "query": "office tunnel access",
+  "strategy": "hybrid",
+  "top_k": 5,
+  "explain": true
+}
+```
+
+The result has `mode: "hybrid"`, ordinary original-text `results`, and a
+`retrieval` diagnostic. Empty results are valid: `empty_reason` distinguishes
+an empty scope from `no_relevant_memory`. `embedding_status` distinguishes
+unconfigured, available, failed and not-used providers. Ranking scores are
+labelled `ranking-not-confidence`; neither a cosine nor a lexical score is a
+probability that a statement is true.
+
+The new strategy currently supports the `memory` address space. File-backed
+note searches retain their existing exact behavior; requesting hybrid there
+fails explicitly. Existing `memory_read` key/query semantics are unchanged.
+
+### Selection and ranking
+
+1. Resolve explicit scopes and apply enabled typed-node hard filters before
+   ranking or embedding. Neither aliases nor semantic similarity widens scope.
+2. Tokenize Unicode words, ignore a small source-defined set of common
+   English/Italian stop words, and rank with BM25 (key terms appear twice).
+   Query terms are capped at 64; tokenization is capped at 4,096 terms per field.
+3. Admit lexical matches covering at least 60% of significant query terms,
+   with at least two matched terms for multi-term queries or one for a single
+   term. An exact key is admitted directly. Compound identifiers and numeric
+   identifiers must match exactly within at least one explicit query variant.
+   Match complete identifiers, not prefixes: `ember-42` does not match
+   `ember-42-backup`, and `4.2.1` does not match `4.2.1-rc1`. Sentence-ending
+   punctuation is not part of an identifier.
+4. With embeddings available, semantic scores contribute to ranking. A match
+   based only on semantics additionally requires an **explicit positive**
+   `min_score`. Omitted or zero `min_score` never admits semantic-only matches.
+   Choose that threshold using examples from the configured model and corpus;
+   no universal cosine cutoff is supplied or claimed as calibrated confidence.
+5. Put exact keys first, then sort by 80% normalized BM25 contribution plus
+   20% nonnegative cosine contribution. Break ties by scope/key deterministically.
+   Collapse identical key/value pairs; keep different values separate.
+
+`top_k` is a maximum, not a quota to fill. Provider failure, incomplete vectors,
+zero vectors or invalid dimensions fall back to lexical admission. Failure
+diagnostics include no provider response bodies. This does not change the
+legacy strategy's existing fallback semantics.
+
+With `explain: true`, each result includes `match.reasons`, matched terms,
+lexical coverage, the optional semantic score, and source scope/key locators.
+Identical records retain all source locators. `possible_conflicts` identifies
+admitted records sharing a key but having different live values across scopes.
+That is a review aid, not a finding that one value is false or an instruction
+to merge them. Explanations exist only in the response; no new recall log is kept.
+
+### Explicit aliases
+
+Aliases can be curated into ordinary memory text, or supplied as bounded query
+alternatives:
+
+```json
+{
+  "scope": "project",
+  "query": "home connection",
+  "strategy": "hybrid",
+  "query_aliases": ["office tunnel access", "relaybox"],
+  "explain": true
+}
+```
+
+At most eight aliases of 256 UTF-8 bytes each are accepted. The original hybrid
+query is bounded to 4 KiB. Invalid bounds fail before opening scopes or making
+provider requests. Aliases affect lexical selection; the original query is
+embedded once. They are caller-supplied hints and may explicitly substitute an
+identifier, not automatically inferred identities or additional authority.
+`query_aliases` and `explain` require `strategy: "hybrid"`.
+
+## Current values, history and authority
+
+Search reads live records and excludes hidden supersession history. Updating a
+fact through the existing supersession workflow makes the new value current
+and preserves the displaced value for explicit history inspection. A new MCP
+process can retrieve the live value after the previous local process exits.
+Simultaneous clients should use one shared server: independent direct local
+AgentFS processes can contend for native database locks.
+
+Current means current in the store, not independently verified or still true
+in the real world. No automatic expiry, contradiction resolution, history
+deletion, knowledge promotion or durable memory mutation is introduced. Verify
+maintained sources and supersede stale facts only through an authorized workflow.
+Personal scopes in a trusted deployment are not a multi-user ACL mechanism.
+The separate team preview is unchanged.
+
+## A fair Markdown comparison
+
+A curated Markdown folder provides readable durable knowledge, human editing,
+navigation and Git history. Adding the same retrieval policy can reproduce the
+same retrieval results. Cairnkeep provides a maintained shared MCP interface,
+scope routing, lifecycle operations and harness integrations; SQLite itself is
+not evidence of better recall.
+
+The reproducible study builds real temporary Markdown files and equivalent
+memory candidates with identical keys, values, scopes and explicit aliases.
+It compares a searchable-folder BM25 baseline, the legacy substring policy,
+the new admission policy and the same new policy applied to Markdown. Both
+folder variants receive the same tokenizer/BM25 implementation; the baseline
+is deliberately strong, and the Markdown parity control distinguishes search
+policy gains from storage format. The substring comparator unions explicit
+query variants, equivalent to separate legacy requests.
+
+Thirteen declared scenarios include a changed tunnel configuration, aliases,
+Italian queries, continuation conventions, release policy, unrelated queries,
+nearby/obsolete identifiers and wrong-project content, with 0/100/1,000
+distractors. The [frozen report](benchmarks/memory-retrieval-v1.json) shows:
+
+| Adapter, 1,000 distractors | Correct first result, 9 positive cases | Negative queries returning results, 4 cases | Irrelevant value bytes |
+|---|---:|---:|---:|
+| Legacy substring | 3/9 | 0/4 | 0 |
+| Markdown + BM25 | 8/9 | 3/4 | 3,359 |
+| Hybrid lexical admission | 8/9 | 0/4 | 0 |
+| Markdown + same admission | 8/9 | 0/4 | 0 |
+
+These are synthetic regression cases, not general retrieval accuracy, an
+independent holdout, autonomous task quality or proof of human consent. The
+semantic-only paraphrase is deliberately missed by lexical modes; the report
+does not claim perfect recall. A real MCP control separately verifies actual
+supersession, new-process handoff, scope filtering before provider requests and
+provider-failure behavior. No live user memory is read by these checks.
+
+From a source checkout:
+
+```sh
+npm --prefix mcp-memory-server run build
+npm --prefix mcp-memory-server run check:memory-retrieval
+npm --prefix mcp-memory-server run study:memory-retrieval
+# Explicit provider evaluation, using synthetic fixtures only:
+node mcp-memory-server/scripts/run-memory-retrieval-study.mjs --live --summary
+```
+
+The live study reports a model-ID digest, not credentials or endpoint addresses.
+Its sample threshold of 0.75 is an evaluation arm, not a recommended default.
+An attempted live run during development accepted short probes but failed with
+HTTP 502 on some fixture inputs. No successful live-model result is claimed.
+The offline suite never requires a model, API key, paid service or network beyond
+its own loopback fixture. An optional learned reranker remains deferred until
+provider-specific evidence demonstrates a benefit beyond this simpler policy.
+
+## Research connection
+
+[Alex Wang's associative-memory article](https://alexwang.ai/posts/associative-memory/)
+provides a useful perspective on recall interference. Its neural-layer capacity
+experiments do not establish limits for a text database. This implementation
+applies selective retrieval at the application layer without changing model
+weights. Merely applying softmax to an existing score list preserves ordering
+and would not improve fixed-top-k selection.

package/docs/operating.md

@@ -32,6 +32,11 @@ sync command but never runs it automatically. This guide covers all three.
 - Optional: an OpenAI-compatible LLM endpoint for memory extraction and
   embedding-ranked search. Without it, memory search degrades to substring
   matching — everything else still works.
+  Explicit `memory_search` calls with `strategy: "hybrid"` additionally support
+  lexical/identifier admission without a provider, bounded `query_aliases`,
+  optional explanations and empty results. Semantic-only admission requires an
+  explicit positive `min_score`; see [memory retrieval](memory-retrieval.md)
+  for model-specific threshold limits and the Markdown comparison.
 - Optional: rootless Podman for the containerized memory server and isolated
   workspace base described in [Containers](containers.md).
 

package/docs/releasing.md

@@ -66,8 +66,73 @@ or moved. If the tree-addressed candidate has expired, rerun CI for the exact
 release commit before retrying publication; the workflow never substitutes a
 different version or tree.
 
+CI job retries can also regenerate the same tree-addressed candidate. Its
+upload explicitly replaces an existing artifact of that name within the same
+workflow run, only after all platform/runtime dependencies succeed. This avoids
+an artifact-name conflict after a completed job is retried; it does not move
+versioned image tags, republish npm versions or replace GitHub Release assets.
+Candidate checksums and tree/version binding are verified as usual.
+
 ## Repository configuration
 
+### Repeatable post-publication verification
+
+From a source checkout, use the maintained artifact verifier after the normal
+publication workflow succeeds:
+
+```sh
+node scripts/verify-published-release.mjs \
+  --version X.Y.Z --commit RELEASE_COMMIT --tree RELEASE_TREE \
+  --ci-run SUCCESSFUL_CI_RUN --security-run SUCCESSFUL_SECURITY_RUN \
+  --publish-run SUCCESSFUL_PUBLISH_RUN \
+  --out /private/evidence/release-verification.json
+```
+
+Provide the exact 40-character commit and tree and numeric run IDs. The output
+parent must already exist. The tool refuses to replace an existing report and
+writes one only after all checks succeed, with mode `0600` on POSIX. It supports
+stable releases whose npm version and both `latest` image tags are the expected
+release; it does not reinterpret an old release as the current channel.
+
+It requires a GitHub CLI supporting `attestation verify` and its exact-source,
+workflow, certificate-identity and hosted-runner policy flags, plus a current
+npm supporting registry-signature and provenance verification. Use `--gh PATH`
+or `--npm PATH` for explicit tool selection; stale trust data or unsupported
+flags fail closed rather than skipping a check. No runtime dependency is added.
+
+The tool checks successful maintained CI, security and publication workflows, immutable
+tag/commit/tree binding, retained CI candidate checksums and exact release-asset
+bytes, CycloneDX 1.6, registry bytes/integrity, npm signatures/attestations, both
+anonymous OCI architectures, per-architecture bound SPDX documents and signed
+exact-source publication provenance. It uses bounded network responses and
+subprocess deadlines. Registry blob redirects are limited to the designated
+GitHub container storage host, without forwarding registry authorization.
+
+Verification downloads assets and installs the exact registry package into
+private disposable staging with lifecycle scripts disabled; staging is removed
+on success or failure. It does not publish, move tags, change configuration,
+restart a server, enable capture or upgrade a machine. The report explicitly
+states `deployment_verified: false`: separately run the maintained backup,
+upgrade/rollback, routed/legacy-session and project-doctor procedures. Artifact
+verification is not deployment evidence or a security certification.
+Anonymous OCI checks read manifests, configuration and bound SBOM blobs, not
+every image layer. The report records `full_image_layer_pull_verified: false`;
+the separate anonymous full-pull release gate above still applies.
+
+Run the offline positive and mutation controls without network access:
+
+```sh
+node scripts/test-release-verification.mjs
+```
+
+These controls inject failed workflows, different trees/tags, expired
+candidates, tampered assets, unsafe checksum paths, registry mismatches,
+signature/provenance failures, missing architectures and unbound SBOMs. They
+test policy decisions using synthetic inputs, not real cryptographic signatures;
+the live verifier delegates signature validation to npm and the GitHub CLI.
+
+### Publication permissions
+
 The repository must provide an `NPM_TOKEN` Actions secret authorized to publish
 `@cairnkeep/cli`. Keep that credential out of local files and rotate it according
 to the npm account's security policy. Workflow defaults grant only

package/docs/security-assurance.md

@@ -164,6 +164,8 @@ node mcp-memory-server/scripts/smoke-outbound-security.mjs
 node scripts/test-runtime-file-security.mjs
 node scripts/test-omp-staged-security.mjs
 node scripts/test-container-health.mjs
+node mcp-memory-server/scripts/smoke-note-mcp.mjs --fixture-security-only
+node scripts/verify-phase19-runtime-evidence.mjs --self-test
 ```
 
 Build the server first when testing a source checkout. These regressions do not
@@ -171,6 +173,19 @@ assert that every historical CodeQL/Scorecard alert is fixed. Triage findings
 individually and use a fresh default-branch analysis to confirm alert status;
 never dismiss an uninvestigated finding merely to improve the score.
 
+The developer-evidence follow-up also hardens the shared runtime note pre-image
+hasher: it reads from an inspected descriptor under a 64 MiB cap and rejects
+replacement or growth before accepting a hash. Missing targets remain valid
+pre-images; linked or unsafe targets do not. Fixture creation uses exclusive
+creation, retaining a file created by a competing writer. Regression controls
+exercise both the actual transaction entry point and fixture creation.
+Developer snapshots retain exact hashes and modes from the same bounded read;
+the runtime-evidence verifier retains its 16 MiB cap and rejects both growth
+and replacement. Corrupt SQLite fixtures must still remain failed and unchanged
+after doctor. Remote-context test labels never interpolate provider values.
+These are read-integrity controls, not atomic multi-file note publication or a
+same-account sandbox. Re-run the full native-platform gates before shipping.
+
 The bounded-input control exercises real evaluation, OKF import/export, graph
 status, receipt and progressive-cache readers. Valid-file substitution and
 same-inode growth are injected at actual inspection/read boundaries; the test

package/docs/storage.md

@@ -48,6 +48,11 @@ services used to process/search memory. They do not change where the SQLite
 databases are stored. Git-provider and routing configuration do not change
 storage either.
 
+`memory_search` can optionally select [hybrid retrieval](memory-retrieval.md).
+This changes read-time ranking only: aliases are request hints, explanations
+are returned to the caller without a new log, and original memory text remains
+authoritative. No new store, migration or automatic memory capture is introduced.
+
 Trajectory storage is deliberately separate from MCP memory. The harness hook
 writes `.agentfs/trajectory.db` on the client machine where Claude Code,
 OpenCode, or Pi is running; it is never redirected to the remote HTTP memory server.

package/examples/eval/bundled-fake.json

@@ -1 +1 @@
-{"identifier":"cairn-offline-fake-v1","package_version":"2.22.4","schema_version":1,"task_set_digest":"13e95379f35839593047ec34521a65a8fa3ab92bd3993bca9dc8bf5c4a9ff36f"}
+{"identifier":"cairn-offline-fake-v1","package_version":"2.23.0","schema_version":1,"task_set_digest":"13e95379f35839593047ec34521a65a8fa3ab92bd3993bca9dc8bf5c4a9ff36f"}

package/mcp-memory-server/dist/index.js

@@ -25,6 +25,7 @@ import { profileAllowsTool, resolveMcpToolProfile } from "./mcp-tool-profile.js"
 import { isWorkEvidenceEnabled } from "./work-evidence-schema.js";
 import { extractMemoryCandidates as extractMemoryCandidatesFromProvider } from "./memory-extraction.js";
 import { applyReviewedMemory, invalidateReviewedMemory } from "./reviewed-memory-store.js";
+import { rankHybridMemory } from "./memory-retrieval.js";
 const MINIMUM_NODE_MAJOR = 22;
 const nodeMajor = Number.parseInt(process.versions.node, 10);
 if (!Number.isInteger(nodeMajor) || nodeMajor < MINIMUM_NODE_MAJOR) {
@@ -377,7 +378,14 @@ async function semanticSearch(scope, query, topK, minScore, config = getMemoryCo
         entries: filterEntries(await listEntries(candidate, "", { projectId, typed: options.typed }), options.filters ?? {}),
     })));
     const allEntries = perScopeEntries.flatMap((group) => group.entries);
+    const hybrid = (embeddingStatus, semanticScores) => ({
+        ...rankHybridMemory(allEntries, query, { topK, minScore, aliases: options.aliases, explain: options.explain, embeddingStatus, semanticScores }),
+        mode: "hybrid",
+        ...(embeddingConfig ? { model: embeddingConfig.model } : {}),
+    });
     if (allEntries.length === 0) {
+        if (options.strategy === "hybrid")
+            return hybrid(embeddingConfig ? "not_used" : "unconfigured");
         return { results: [], mode: embeddingConfig ? "semantic" : "substring" };
     }
     const substringFallback = () => {
@@ -390,6 +398,8 @@ async function semanticSearch(scope, query, topK, minScore, config = getMemoryCo
         };
     };
     if (!embeddingConfig) {
+        if (options.strategy === "hybrid")
+            return hybrid("unconfigured");
         return substringFallback();
     }
     try {
@@ -422,9 +432,23 @@ async function semanticSearch(scope, query, topK, minScore, config = getMemoryCo
             }
         }
         const [queryVector] = await embedTexts(embeddingConfig, [query]);
+        if (options.strategy === "hybrid") {
+            const validVector = (vector) => {
+                if (!vector?.length || vector.length > 65536 || vector.some(value => !Number.isFinite(value)))
+                    return false;
+                const squaredNorm = vector.reduce((total, value) => total + value * value, 0);
+                return squaredNorm > 0 && Number.isFinite(squaredNorm);
+            };
+            if (!validVector(queryVector) || allEntries.some(entry => !validVector(vectors.get(entry)) || vectors.get(entry).length !== queryVector.length)) {
+                throw new Error("Embedding vectors are incomplete or invalid.");
+            }
+        }
         for (const cache of caches) {
             cache.save();
         }
+        if (options.strategy === "hybrid") {
+            return hybrid("available", new Map(allEntries.map(entry => [entry, Math.max(-1, Math.min(1, cosineSimilarity(queryVector, vectors.get(entry))))])));
+        }
         const exact = options.typed
             ? allEntries.filter((entry) => isExactTextHit(entry, query)).sort(compareMemoryEntries).map((entry) => ({ ...entry, score: 1 }))
             : [];
@@ -444,6 +468,8 @@ async function semanticSearch(scope, query, topK, minScore, config = getMemoryCo
     }
     catch {
         // Embedding endpoint failure — degrade gracefully to substring matching.
+        if (options.strategy === "hybrid")
+            return hybrid("failed");
         return substringFallback();
     }
 }
@@ -1112,6 +1138,9 @@ function buildMemoryServer(context, capabilitySnapshot, withCapability, toolProf
             query: z.string().min(1),
             top_k: z.number().int().min(1).max(50).optional(),
             min_score: z.number().min(0).max(1).optional(),
+            strategy: z.enum(["legacy", "hybrid"]).optional().describe("Default legacy preserves current retrieval. Hybrid admits lexical/identifier matches and semantic-only matches above an explicit positive min_score."),
+            query_aliases: z.array(z.string().trim().min(1).max(256)).max(8).optional().describe("Explicit alternative query phrases for hybrid lexical retrieval; never widens scopes."),
+            explain: z.boolean().optional().describe("Include hybrid match reasons, provenance locators and possible conflicting live values."),
             ...(typedNodesEnabled ? {
                 address_space: NODE_ADDRESS_SPACE_SCHEMA.optional(),
                 node_types: z.array(nodeTypeSchema).min(1).optional(),
@@ -1124,8 +1153,17 @@ function buildMemoryServer(context, capabilitySnapshot, withCapability, toolProf
         },
     }, async (input) => {
         const { scope, query, top_k, min_score } = input;
+        if (input.strategy !== "hybrid" && (input.query_aliases !== undefined || input.explain !== undefined)) {
+            throw new Error("query_aliases and explain require strategy='hybrid'.");
+        }
+        if (input.strategy === "hybrid") {
+            // Validate bounds before opening scopes or contacting an embedding service.
+            rankHybridMemory([], query, { topK: top_k ?? 8, minScore: min_score ?? 0, aliases: input.query_aliases });
+        }
         const addressSpace = "address_space" in input && input.address_space !== undefined ? NODE_ADDRESS_SPACE_SCHEMA.parse(input.address_space) : undefined;
         if (addressSpace && addressSpace !== "memory") {
+            if (input.strategy === "hybrid")
+                throw new Error("Hybrid retrieval currently supports the memory address space only.");
             const noteStore = await import("./note-store.js");
             const entries = noteStore.searchAddressedNotes({ ...noteTargetOptions(scope, addressSpace), query, ...memoryFiltersFromInput(input) }).slice(0, top_k ?? 8);
             const results = entries.map((entry) => ({ ...entry, score: 1 }));
@@ -1135,11 +1173,14 @@ function buildMemoryServer(context, capabilitySnapshot, withCapability, toolProf
                 structuredContent: payload,
             };
         }
-        const { results, mode, model } = await semanticSearch(scope, query, top_k ?? 8, min_score ?? 0, memoryConfig(), context.projectId, {
+        const { results, mode, model, retrieval } = await semanticSearch(scope, query, top_k ?? 8, min_score ?? 0, memoryConfig(), context.projectId, {
             typed: typedNodesEnabled,
             filters: typedNodesEnabled ? memoryFiltersFromInput(input) : {},
+            strategy: input.strategy,
+            aliases: input.query_aliases,
+            explain: input.explain,
         });
-        const payload = { mode, model, count: results.length, results };
+        const payload = { mode, model, count: results.length, results, ...(retrieval ? { retrieval } : {}) };
         return {
             content: [{ type: "text", text: asToolText(payload) }],
             structuredContent: payload,

package/mcp-memory-server/dist/memory-retrieval.js

@@ -0,0 +1,128 @@
+const STOP = new Set('a an and are as at be by can do does for from how i in is it me my of on or our please that the their this to use we what when where which with would you your un una uno il lo la gli le di da del della e che con come per posso mio mia sul nel dei'.split(' '));
+const normalize = (value) => value.normalize('NFKC').toLowerCase();
+const compare = (left, right) => left.scope.localeCompare(right.scope, 'en') || left.key.localeCompare(right.key, 'en');
+const rounded = (value) => Math.round(value * 1e6) / 1e6;
+export function retrievalTerms(text) {
+    const result = [];
+    for (const match of normalize(text).matchAll(/[\p{L}\p{N}]+/gu)) {
+        if (match[0].length > 1 && !STOP.has(match[0]))
+            result.push(match[0]);
+        if (result.length === 4096)
+            break;
+    }
+    return result;
+}
+function identifiers(text) {
+    // Match ordinary words too, then select identifiers. Requiring a separator
+    // in an unanchored greedy expression repeatedly rescans long plain words.
+    const result = new Set();
+    for (const match of text.matchAll(/[\p{L}\p{N}]+(?:[._:/-][\p{L}\p{N}]+)*/gu)) {
+        if (/[._:/\d-]/u.test(match[0]))
+            result.add(match[0]);
+    }
+    return result;
+}
+function variant(text) {
+    const normalized = normalize(text).trim();
+    return { normalized, terms: [...new Set(retrievalTerms(text))].slice(0, 64), identifiers: identifiers(normalized) };
+}
+/** BM25 is also used by the transparent Markdown-folder baseline. */
+export function lexicalMemoryScores(entries, query) {
+    const queryTerms = [...new Set(retrievalTerms(query))].slice(0, 64);
+    const docs = entries.map(entry => [
+        ...retrievalTerms(entry.key), ...retrievalTerms(entry.key),
+        ...retrievalTerms(entry.value), ...retrievalTerms([entry.node_type, ...(entry.tags ?? [])].filter(Boolean).join(' ')),
+    ]);
+    const frequencies = docs.map(terms => {
+        const counts = new Map();
+        for (const term of terms)
+            counts.set(term, (counts.get(term) ?? 0) + 1);
+        return counts;
+    });
+    const averageLength = docs.reduce((total, terms) => total + terms.length, 0) / Math.max(1, docs.length) || 1;
+    const documentFrequencies = new Map(queryTerms.map(term => [term, frequencies.filter(counts => counts.has(term)).length]));
+    return docs.map((terms, index) => queryTerms.reduce((score, term) => {
+        const frequency = frequencies[index].get(term) ?? 0;
+        const df = documentFrequencies.get(term) ?? 0;
+        const idf = Math.log(1 + (entries.length - df + 0.5) / (df + 0.5));
+        return score + idf * frequency * 2.2 / (frequency + 1.2 * (0.25 + 0.75 * terms.length / averageLength));
+    }, 0));
+}
+export function rankHybridMemory(entries, query, options) {
+    const aliases = options.aliases ?? [];
+    if (Buffer.byteLength(query, 'utf8') > 4096 || !query.trim() || aliases.length > 8
+        || aliases.some(alias => !alias.trim() || Buffer.byteLength(alias, 'utf8') > 256)
+        || !Number.isInteger(options.topK) || options.topK < 1 || options.topK > 50
+        || !Number.isFinite(options.minScore) || options.minScore < 0 || options.minScore > 1) {
+        throw new Error('Hybrid memory search requires bounded query, aliases, result count and similarity threshold.');
+    }
+    const variants = [...new Set([query, ...aliases].map(text => normalize(text).trim()))].map(variant);
+    const lexicalScores = variants.map(item => lexicalMemoryScores(entries, item.normalized));
+    const candidates = entries.map((entry, index) => {
+        const text = normalize([entry.key, entry.value, entry.node_type, ...(entry.tags ?? [])].filter(Boolean).join('\n'));
+        const words = new Set(retrievalTerms(text));
+        const entryIdentifiers = identifiers(text);
+        const matches = variants.map((item, v) => {
+            const matched = item.terms.filter(term => words.has(term));
+            const coverage = item.terms.length ? matched.length / item.terms.length : 0;
+            const identifierMatch = [...item.identifiers].every(id => entryIdentifiers.has(id));
+            const exactKey = normalize(entry.key) === item.normalized;
+            return {
+                matched, coverage, identifierMatch, exactKey,
+                eligible: identifierMatch && (exactKey || (coverage >= 0.6 && matched.length >= Math.min(2, item.terms.length) && matched.length > 0)),
+                lexical: lexicalScores[v][index],
+            };
+        });
+        const best = matches.filter(match => match.eligible).sort((a, b) => Number(b.exactKey) - Number(a.exactKey) || b.lexical - a.lexical)[0];
+        const rawSemantic = options.semanticScores?.get(entry);
+        const semantic = typeof rawSemantic === 'number' && Number.isFinite(rawSemantic) && rawSemantic >= -1 && rawSemantic <= 1 ? rawSemantic : null;
+        const semanticAdmission = options.minScore > 0 && semantic !== null && semantic >= options.minScore && matches.some(match => match.identifierMatch);
+        const lexical = best?.lexical ?? 0;
+        const score = 0.8 * lexical / (1 + lexical) + 0.2 * Math.max(0, semantic ?? 0);
+        return { entry, best, semantic, semanticAdmission, score };
+    }).filter(item => item.best || item.semanticAdmission)
+        .sort((a, b) => Number(Boolean(b.best?.exactKey)) - Number(Boolean(a.best?.exactKey)) || b.score - a.score || compare(a.entry, b.entry));
+    const grouped = new Map();
+    for (const candidate of candidates) {
+        const key = JSON.stringify([candidate.entry.key, candidate.entry.value]);
+        const group = grouped.get(key) ?? [];
+        group.push(candidate);
+        grouped.set(key, group);
+    }
+    const results = [...grouped.values()].slice(0, options.topK).map(group => {
+        const item = group[0];
+        const reasons = [
+            ...(item.best?.exactKey ? ['exact-key'] : []),
+            ...(item.best ? ['lexical-coverage'] : []),
+            ...(item.semanticAdmission ? ['semantic-threshold'] : []),
+            ...(group.length > 1 ? ['identical-records-collapsed'] : []),
+        ];
+        return {
+            ...item.entry, score: rounded(item.score),
+            ...(options.explain ? { match: {
+                    reasons, matched_terms: item.best?.matched ?? [], lexical_coverage: rounded(item.best?.coverage ?? 0),
+                    semantic_score: item.semantic === null ? null : rounded(item.semantic),
+                    sources: group.map(({ entry }) => ({ scope: entry.scope, key: entry.key })).sort(compare),
+                } } : {}),
+        };
+    });
+    const conflicts = [...new Set(results.map(item => item.key))].flatMap(key => {
+        const relevant = candidates.filter(item => item.entry.key === key);
+        return new Set(relevant.map(item => item.entry.value)).size > 1
+            ? [{ key, scopes: [...new Set(relevant.map(item => item.entry.scope))].sort() }] : [];
+    });
+    return {
+        results,
+        retrieval: {
+            strategy: 'hybrid',
+            score_kind: 'ranking-not-confidence',
+            embedding_status: options.embeddingStatus ?? (options.semanticScores ? 'available' : 'unconfigured'),
+            semantic_min_score: options.minScore > 0 ? options.minScore : null,
+            alias_count: aliases.length,
+            candidates: entries.length,
+            admitted: grouped.size,
+            ...(results.length === 0 ? { empty_reason: entries.length ? 'no_relevant_memory' : 'empty_scope' } : {}),
+            ...(options.explain ? { possible_conflicts: conflicts } : {}),
+        },
+    };
+}

package/mcp-memory-server/dist/note-store.js

@@ -6,6 +6,7 @@ import { buildFailureSignature } from "./failure-signature.js";
 import { NOTE_SCHEMA_VERSION, noteNodeSchema, } from "./note-schema.js";
 import { memoryImportEnvelopeSchema, nodePathSchema, } from "./node-schema.js";
 import { atomicReplace } from "./platform-security.js";
+import { readStableFile } from "./stable-file.js";
 const MANAGED_START = "<!-- cairnkeep:managed:v1:start -->";
 const MANAGED_END = "<!-- cairnkeep:managed:v1:end -->";
 const MAX_PROCESSED_SESSIONS_PER_PROJECT = 4096;
@@ -569,7 +570,22 @@ export function doctorNoteStore(repair = false, options = {}) {
     }
 }
 function fileHash(path) {
-    return existsSync(path) ? createHash("sha256").update(readFileSync(path)).digest("hex") : null;
+    try {
+        const { bytes } = readStableFile(path, { label: "Note transaction pre-image", maxBytes: 64 * 1024 * 1024 });
+        return createHash("sha256").update(bytes).digest("hex");
+    }
+    catch (error) {
+        // Missing targets are valid pre-images. Diagnose absence only after
+        // the descriptor-first read failed; never authorize a read by a check.
+        try {
+            lstatSync(path);
+        }
+        catch (missing) {
+            if (missing.code === "ENOENT")
+                return null;
+        }
+        throw error;
+    }
 }
 function bytesHash(bytes) {
     return bytes === null ? null : createHash("sha256").update(bytes).digest("hex");
@@ -1024,9 +1040,13 @@ export async function commitNoteImport(plan) {
 export function createNoteMutationFixture(options) {
     const notesRoot = join(options.storeRoot, ".cairnkeep-note-fixture");
     const target = join(notesRoot, "live.txt");
-    if (!existsSync(target)) {
-        mkdirSync(dirname(target), { recursive: true });
-        writeFileSync(target, "before\n");
+    mkdirSync(dirname(target), { recursive: true });
+    try {
+        writeFileSync(target, "before\n", { flag: "wx", mode: 0o600 });
+    }
+    catch (error) {
+        if (error.code !== "EEXIST")
+            throw error;
     }
     const bytes = options.operation === "delete" ? null : `${options.operation}-after\n`;
     return { schema_version: 1, operation: options.operation, notes_root: notesRoot, changes: [{ path: target, bytes, before_hash: fileHash(target), final_hash: bytesHash(bytes) }], result: { ok: true, fixture: true } };

package/mcp-memory-server/package.json

@@ -15,6 +15,8 @@
         "check:embeddings": "node scripts/smoke-embeddings.mjs",
         "check:extract": "node scripts/smoke-extract-cli.mjs",
         "check:search": "node scripts/smoke-search-e2e.mjs",
+        "check:memory-retrieval": "node scripts/smoke-memory-retrieval.mjs && node scripts/smoke-memory-retrieval-mcp.mjs && node scripts/run-memory-retrieval-study.mjs --check --summary",
+        "study:memory-retrieval": "npm run build && node scripts/run-memory-retrieval-study.mjs --summary",
         "check:scope-guard": "node scripts/smoke-scope-guard.mjs",
         "check:http-guard": "node scripts/smoke-http-guard.mjs",
         "check:remote-context": "node scripts/smoke-remote-context.mjs",

package/opencode/command/recall.md

@@ -25,7 +25,14 @@ Read order (canonical sources still win over derived layers):
 
 <process>
 ## 1. AgentFS
-`cairn-memory_memory_search` scope `project` with $ARGUMENTS. If nothing relevant, retry scope `all`.
+`cairn-memory_memory_search` scope `project` with $ARGUMENTS. When the discovered
+schema supports it, use `strategy: "hybrid"` for identifier-sensitive or
+mixed-wording queries and `explain: true` to inspect matches. Supply only aliases
+already known from the task. Semantic-only hybrid matches require an explicitly
+chosen positive `min_score`; retain the existing semantic strategy for paraphrases
+when no model-specific threshold is available. Empty results are valid.
+Use broader configured scopes only when the question genuinely spans them;
+do not automatically widen a search merely to force a hit.
 
 ## 2. Wiki
 Read `.planning/wiki/index.md` if present and follow links relevant to $ARGUMENTS.

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "@cairnkeep/cli",
-    "version": "2.22.4",
+    "version": "2.23.0",
     "description": "A durable, harness-agnostic memory + context layer for coding agents and MCP clients.",
     "license": "Apache-2.0",
     "type": "module",