@cairnkeep/cli 2.16.0 → 2.17.0

package/docs/learning/lessons/L26-context-intelligence.md

@@ -0,0 +1,180 @@
+# L26 - Context intelligence with explicit authority
+
+## Outcome
+
+You will measure retrieval before changing it, retrieve an enabled context pack
+progressively, link a privacy-preserving usage receipt to work evidence, and
+review a trajectory-derived memory proposal before any durable write. You will
+also understand when the optional OpenViking adapter is useful and what it does
+not replace.
+
+## Prerequisites
+
+- Complete [L21 - Immutable context packs](L21-context-packs.md) and
+  [L25 - Bounded workflow playbooks](L25-playbooks.md).
+- Work in a project already bootstrapped with Cairnkeep.
+- Enable at least one document-bearing context pack. Skill entries stay hidden
+  unless separately approved for this project.
+- For the proposal exercise, capture a trajectory and configure the existing
+  Cairnkeep extraction endpoint. The retrieval and benchmark exercises are
+  local and offline.
+
+## 1. Freeze the retrieval baseline
+
+From a Cairnkeep source checkout, run the checked-in synthetic suite:
+
+```sh
+npm --prefix mcp-memory-server run build
+npm --prefix mcp-memory-server run check:retrieval-benchmark
+```
+
+The suite contains hierarchical documents, multiple versions, and a hidden
+skill. It checks frozen substring and deterministic mock-embedding results, then
+reports Hit@1, Recall@5, MRR, relevant and irrelevant bytes, estimated tokens,
+latency, mutation counts, and forbidden leakage. A quality improvement is not
+accepted if it leaks the skill, mutates storage, or quietly uses the network.
+
+For a non-golden current report:
+
+```sh
+npm --prefix mcp-memory-server run benchmark:retrieval
+```
+
+Record the mode with the score. Comparing one mode's score with another mode's
+label omitted is not meaningful evidence.
+
+## 2. Retrieve progressively
+
+Start the memory server with `CAIRN_CONTEXT_PACKS=1`. In an MCP-capable harness,
+inspect the visible hierarchy first:
+
+```json
+context_pack_tree { "detail": "abstract" }
+```
+
+Now ask a broad question without loading every file:
+
+```json
+context_pack_search {
+  "query": "release validation",
+  "strategy": "hierarchical",
+  "detail": "overview",
+  "explain": true,
+  "include_refs": true
+}
+```
+
+Check four things in the result:
+
+1. The selected pack, directory, and file fit the question.
+2. The bounded trace explains selection without exposing hidden content.
+3. Each result includes a `chunk_digest` and the response includes a
+   `result_digest`.
+4. No unapproved skill appears in the tree, summaries, results, or trace.
+
+Ask for `detail: "content"` only after summaries narrow the candidate set. For a
+known path, use `context_pack_read`; exact path/title searches can bypass the
+hierarchy. Omitting all new fields exercises the compatible flat search.
+
+## 3. Record use without recording content
+
+Create or activate a work-evidence record for the task, then enable the local
+receipt mutation with both `CAIRN_WORK_EVIDENCE=1` and
+`CAIRN_CONTEXT_USAGE=1`. After deciding whether you used a
+search result, call:
+
+```json
+context_usage_record {
+  "task_digest": "<64-lowercase-hex-task-digest>",
+  "result_digest": "<result_digest-from-search>",
+  "outcome": "used"
+}
+```
+
+Use `unused` or `unknown` when that is the honest outcome. Supply `evidence_id`
+only when linking a non-active record deliberately. Inspect the evidence and
+verify that the receipt contains identifiers and outcome—not query text,
+retrieved content, prompts, or responses.
+
+This is a mutation. A read-only MCP profile correctly hides it even though the
+pack retrieval tools remain visible.
+
+## 4. Promote a trajectory through review
+
+List or capture a trajectory you want to distill, then create a proposal:
+
+```sh
+cairn proposals create --session SESSION_ID --scope project --json
+cairn proposals list --json
+cairn proposals show PROPOSAL_DIGEST --json
+```
+
+Review every candidate for accuracy, scope, sensitivity, and durability. The
+proposal is not reviewed memory. If it is acceptable, apply the exact digest:
+
+```sh
+cairn proposals apply PROPOSAL_DIGEST --json
+cairn proposals doctor --json
+```
+
+Creation redacts the stored trajectory again before configured model extraction.
+Application rechecks the source digest and base reviewed-memory hashes. If it
+reports stale state, create and review a new proposal; do not edit the digest-
+addressed file or bypass the check.
+
+## 5. Decide whether an external provider helps
+
+AnythingLLM remains Cairnkeep's default domain provider. Use OpenViking only when
+you deliberately operate an OpenViking index and want its search results through
+the existing `domain_knowledge_query` surface:
+
+```sh
+CAIRN_DOMAIN_RETRIEVAL_PROVIDER=openviking
+CAIRN_OPENVIKING=1
+CAIRN_OPENVIKING_BASE_URL=http://127.0.0.1:1933
+```
+
+The adapter is read-only and calls search only. It does not synchronize data,
+manage OpenViking, or replace Cairnkeep's reviewed memory, context-pack pins,
+approvals, evidence, or tool profiles. Leave the variables unset when you do not
+need the external index.
+
+## Common failures
+
+- **A hidden skill appears:** revoke exposure, run `cairn doctor`, and treat it as
+  a visibility regression. Approval is bound to pack/path/file digests.
+- **Hierarchical search looks worse:** compare it with the frozen benchmark and
+  inspect a bounded trace. Do not change default flat behavior to hide a miss.
+- **A receipt conflicts:** the same task/result identity was recorded with a
+  different outcome. Resolve the evidence rather than overwriting history.
+- **Proposal apply is stale:** source or reviewed memory changed after creation.
+  Generate a new proposal and review it.
+- **OpenViking is unreachable:** verify the explicit gates, HTTPS requirement for
+  non-loopback hosts, endpoint, timeout, and service separately. Cairnkeep's
+  default operation does not depend on it.
+
+## Privacy and trust boundary
+
+Context-pack and proposal digests prove content integrity and bind decisions to
+specific state; they do not authenticate a publisher or prove that extracted
+claims are true. Derived summaries can contain visible pack content, so protect
+the cache like the pack store. Proposal extraction can cross a model-provider
+boundary only through the operator's existing extraction configuration and only
+after redaction. Context-usage receipts deliberately avoid raw content.
+
+## Recovery and acceptance
+
+You are done when:
+
+- the frozen benchmark passes without network or filesystem mutation;
+- tree and hierarchical search expose enabled documents and only approved skills;
+- flat search still works when optional fields are omitted;
+- a receipt links to work evidence without copying content;
+- a proposal is inspected before exact-digest apply, or intentionally rejected;
+- you can explain why OpenViking is optional retrieval rather than a replacement
+  for Cairnkeep.
+
+Continue with the operational references in
+[Context intelligence](../../context-intelligence.md),
+[Immutable context packs](../../context-packs.md), and
+[Git-linked work evidence](../../work-evidence.md).

package/docs/learning/tracks/evidence-and-evaluation.md

@@ -11,6 +11,7 @@
 4. [L16 - Evaluation and ablation](../lessons/L16-evaluation.md)
 5. [L17 - The agent boundary](../lessons/L17-agent-boundary.md)
 6. [L19 - Validated skill improvement](../lessons/L19-validated-skill-improvement.md)
+7. [L26 - Context intelligence with explicit authority](../lessons/L26-context-intelligence.md)
 
 The track is complete when the learner can explain each consent boundary,
 inspect and manage every local evidence store, including Git-linked work

package/docs/learning/tracks/operator.md

@@ -16,6 +16,7 @@
 10. [L23 - Guided setup and Pi memory](../lessons/L23-guided-setup.md)
 11. [L24 - Reviewed knowledge exchange](../lessons/L24-okf-exchange.md)
 12. [L25 - Bounded workflow playbooks](../lessons/L25-playbooks.md)
+13. [L26 - Context intelligence with explicit authority](../lessons/L26-context-intelligence.md)
 
 All modules are optional. The track is complete when the operator can add and
 remove each integration without breaking the standalone local workflow and can

package/docs/learning/tracks/practitioner.md

@@ -12,6 +12,7 @@
 6. [L08 - Multiple machines](../lessons/L08-multiple-machines.md)
 7. [L13 - Session evidence and hindsight](../lessons/L13-session-evidence.md)
 8. [L25 - Bounded workflow playbooks](../lessons/L25-playbooks.md)
+9. [L26 - Context intelligence with explicit authority](../lessons/L26-context-intelligence.md)
 
 The track is complete when the learner can decide what should become memory,
 review derived knowledge, run governed repository checks, locate the store, and

package/docs/learning/video-scripts/V26-context-intelligence.md

@@ -0,0 +1,144 @@
+# V26 - Context intelligence with explicit authority
+
+Target length: 12–15 minutes. Audience: experienced developers who understand
+MCP and have completed the context-pack lesson.
+
+## Recording outline
+
+### 0:00–0:45 — The contract
+
+Show a simple four-step card: **measure → retrieve → attest → review**.
+
+Narration:
+
+> Cairnkeep now makes context retrieval more selective without making memory
+> promotion automatic. We measure retrieval first, inspect context progressively,
+> record only a digest-level usage receipt, and review every durable-memory
+> proposal before applying it.
+
+Pause. State that OpenViking is optional external retrieval and does not replace
+Cairnkeep.
+
+### 0:45–2:15 — Run the frozen benchmark
+
+Use a real terminal at readable zoom and human typing speed:
+
+```sh
+npm --prefix mcp-memory-server run build
+npm --prefix mcp-memory-server run check:retrieval-benchmark
+```
+
+Highlight Hit@1, Recall@5, MRR, byte/token estimates, latency, mutations, and
+forbidden leakage. Explain that the corpus includes versions, hierarchy, and a
+hidden skill. Do not claim that synthetic scores predict every project.
+
+### 2:15–5:15 — Tree, overview, then content
+
+Start from an already installed and enabled demo pack. Show:
+
+```json
+context_pack_tree { "detail": "abstract" }
+```
+
+Expand one directory visually. Then run:
+
+```json
+context_pack_search {
+  "query": "release validation",
+  "strategy": "hierarchical",
+  "detail": "overview",
+  "explain": true,
+  "include_refs": true
+}
+```
+
+Zoom briefly into the bounded trace, `chunk_digest`, and `result_digest`. Follow
+with one content-level read of the selected file. Cut between overview and
+content so the terminal remains dynamic without rushing.
+
+Call out that omitting the new options keeps the existing flat content response,
+and that an unapproved skill is absent from every visible surface.
+
+### 5:15–7:00 — Privacy-preserving receipt
+
+With work evidence active and `CAIRN_CONTEXT_USAGE=1`, record the result:
+
+```json
+context_usage_record {
+  "task_digest": "<demo-task-digest>",
+  "result_digest": "<search-result-digest>",
+  "outcome": "used"
+}
+```
+
+Show the stored receipt fields. Add an on-screen comparison:
+
+- stored: evidence ID, task digest, result digest, outcome;
+- not stored: query, prompt, content, response.
+
+Pause after the comparison. Mention that this is a mutation and is unavailable
+in the read-only MCP profile.
+
+### 7:00–10:15 — Review-gated proposal
+
+Use a real captured demo trajectory and type:
+
+```sh
+cairn proposals create --session SESSION_ID --scope project --json
+cairn proposals show PROPOSAL_DIGEST --json
+```
+
+Scroll through a small proposal. Inspect one candidate aloud for correctness,
+scope, sensitivity, and durability. Then apply the exact digest:
+
+```sh
+cairn proposals apply PROPOSAL_DIGEST --json
+cairn proposals doctor --json
+```
+
+Use a diagram or terminal split to show that creation redacts again, extraction
+creates immutable candidates, and apply rechecks source and base-memory state.
+Do not imply that the model writes reviewed memory directly.
+
+### 10:15–12:00 — Optional provider boundary
+
+Show the minimal settings, without exposing a real key:
+
+```sh
+CAIRN_DOMAIN_RETRIEVAL_PROVIDER=openviking
+CAIRN_OPENVIKING=1
+CAIRN_OPENVIKING_BASE_URL=http://127.0.0.1:1933
+```
+
+Run one `domain_knowledge_query`, then return to a boundary diagram:
+
+- OpenViking: optional read-only search index;
+- Cairnkeep: identity, reviewed memory, pack pins, approvals, evidence, profiles.
+
+Mention loopback-only HTTP, HTTPS elsewhere, no redirects, and the separate
+remote-MCP consent gate. Do not show installation as required.
+
+### 12:00–13:00 — Recap and acceptance
+
+Return to **measure → retrieve → attest → review**. Recap:
+
+1. Frozen evidence before retrieval changes.
+2. Summaries before content, with compatible flat fallback.
+3. Digest-level receipts instead of prompt logging.
+4. Explicit proposal review and exact-digest apply.
+5. Optional providers do not replace Cairnkeep's authority model.
+
+Point viewers to L26 and `docs/context-intelligence.md`.
+
+## Recording cautions
+
+- Use a synthetic pack, trajectory, digests, endpoint, and API key placeholder.
+- Ensure the skill shown in the manifest is unapproved and absent from tree,
+  search, trace, and summaries.
+- Keep command pauses long enough to read output; cut dead time rather than
+  accelerating narration.
+- Use terminal autocomplete and real command execution, but prewarm builds so the
+  lesson focuses on behavior rather than compilation.
+- Never label OpenViking as a dependency, synchronization target, memory store,
+  or Cairnkeep replacement.
+- Do not call a proposal “memory” until exact-digest application succeeds.

package/docs/operating.md

@@ -1409,3 +1409,47 @@ read-only related-document traversal. Privacy-reviewed export is an explicit
 `export-okf --check` followed by `--apply --confirm PREVIEW-DIGEST`; only named
 Markdown files and promoted shared notes are eligible. See
 [Open Knowledge Format exchange](open-knowledge-format.md).
+
+## Context-intelligence operations
+
+The v2.17 context-intelligence surfaces are opt-in and independently bounded:
+
+```sh
+# Existing context-pack gate; default search stays flat and compatible.
+CAIRN_CONTEXT_PACKS=1
+
+# Optional local usage-receipt mutation.
+CAIRN_CONTEXT_USAGE=1
+
+# Optional external retrieval provider (AnythingLLM remains the default).
+CAIRN_DOMAIN_RETRIEVAL_PROVIDER=openviking
+CAIRN_OPENVIKING=1
+CAIRN_OPENVIKING_BASE_URL=http://127.0.0.1:1933
+```
+
+For authenticated remote MCP, context packs still require
+`CAIRN_CONTEXT_PACK_HTTP=1`; OpenViking retrieval separately requires
+`CAIRN_OPENVIKING_MCP_HTTP=1`. Neither setting weakens authentication, Host, or
+CORS checks.
+
+Use these source-checkout checks during upgrades and incident diagnosis:
+
+```sh
+npm --prefix mcp-memory-server run check:context-pack-retrieval
+npm --prefix mcp-memory-server run check:context-usage
+npm --prefix mcp-memory-server run check:domain-retrieval
+npm --prefix mcp-memory-server run check:retrieval-benchmark
+cairn proposals doctor --project /path/to/project --json
+(cd /path/to/project && cairn doctor)
+```
+
+Proposal extraction is manual. Create, inspect, and apply the exact digest:
+
+```sh
+cairn proposals create --session SESSION_ID --scope project --project /path/to/project --json
+cairn proposals show PROPOSAL_DIGEST --project /path/to/project --json
+cairn proposals apply PROPOSAL_DIGEST --project /path/to/project --json
+```
+
+No background process synchronizes OpenViking, creates proposals, applies
+memory, enables packs, or approves skills.

package/docs/privacy-and-data-flow.md

@@ -630,3 +630,25 @@ All enabled documents carry pack and file provenance in MCP results. Skill files
 are excluded until the current project approves the exact pack digest, path, and
 file digest. Updating the pack invalidates that approval. Approval only makes a
 skill readable; Cairnkeep never copies it into a harness or executes it.
+
+## Context-intelligence privacy boundaries
+
+- Progressive pack summaries are deterministic local derivatives of currently
+  visible files. Unapproved skills are excluded from trees, summaries, search,
+  and explanation traces.
+- `include_refs: true` produces content-derived digests. A usage receipt stores
+  only evidence/task/result identifiers and `used`, `unused`, or `unknown`; it
+  does not store the query, prompt, retrieved content, or model response.
+- Proposal creation starts from a locally stored trajectory and runs redaction
+  again before configured model extraction. Extraction remains a proposal until
+  an operator or authorized agent reviews and applies its exact digest.
+- AnythingLLM is still the default external domain provider. Selecting
+  OpenViking requires `CAIRN_DOMAIN_RETRIEVAL_PROVIDER=openviking` and
+  `CAIRN_OPENVIKING=1`; remote MCP requires an additional
+  `CAIRN_OPENVIKING_MCP_HTTP=1` consent flag.
+- The OpenViking adapter is query-only. It sends the requested workspace target
+  and query to the configured endpoint, but performs no synchronization, watch,
+  commit, update, or deletion. Non-loopback endpoints must use HTTPS.
+
+Pack digests and proposal digests protect integrity and stale-state detection;
+they do not authenticate a publisher or make extracted statements true.

package/docs/storage.md

@@ -576,3 +576,24 @@ link index is derived data under `cache/graphs/<pack-digest>.json`; corruption
 or deletion does not affect identity and causes a rebuild from the object. OKF
 exports are not stored automatically: `export-okf --apply` creates a new
 operator-selected directory only after its preview digest is confirmed.
+
+## Context-intelligence derived and review state
+
+Progressive context-pack abstracts and overviews live below
+`${CAIRN_PACK_BASE_DIR:-~/.cairnkeep/packs}/cache/context/<pack-digest>/` as
+digest-named JSON. This cache is derived, permission-restricted, outside
+immutable pack objects, and safe to discard or deterministically rebuild.
+Validation rejects symlinks, invalid names, oversized records, digest mismatches,
+and temporary remnants. Repair removes invalid derived entries only; it never
+changes a project pointer or skill approval.
+
+Review-gated memory proposals live under
+`<project>/.agentfs/memory-proposals/<proposal-digest>.json`. The directory is mode
+`0700` and files are mode `0600` on POSIX. Proposals are immutable and include
+the source trajectory digest and base reviewed-memory hashes used to reject stale
+application. Applying a proposal writes through the existing reviewed-memory
+store; it does not turn the proposal directory into a second memory authority.
+
+Context-usage receipts are strict links in the existing work-evidence store.
+They inherit work-evidence inspection, retention, and doctor behavior; there is
+no separate prompt or retrieved-content log.

package/docs/work-evidence.md

@@ -52,6 +52,20 @@ bodies are not included. The patch is redacted, bounded by the lower of the
 work-evidence and artifact limits, and stored in the existing artifact store.
 Cairnkeep provides no patch apply, restore or replay command.
 
+## Context-usage receipts
+
+With context packs enabled, `context_pack_search` can return stable references
+when called with `include_refs: true`. With `CAIRN_WORK_EVIDENCE=1` active, set
+`CAIRN_CONTEXT_USAGE=1` as the second gate to expose the
+local-only `context_usage_record` mutation. It links `task_digest`,
+`result_digest`, and an outcome (`used`, `unused`, or `unknown`) to the active
+work-evidence record, or to an explicit `evidence_id`.
+
+Receipts intentionally exclude the query, prompt, retrieved text, and model
+response. Their ID is deterministic, repeat recording is idempotent, conflicting
+outcomes fail, and the normal work-evidence doctor and retention rules apply.
+Because recording changes evidence, the read-only MCP profile excludes the tool.
+
 ## Inspect and maintain
 
 ```sh

package/examples/eval/bundled-fake.json

@@ -1 +1 @@
-{"identifier":"cairn-offline-fake-v1","package_version":"2.16.0","schema_version":1,"task_set_digest":"13e95379f35839593047ec34521a65a8fa3ab92bd3993bca9dc8bf5c4a9ff36f"}
+{"identifier":"cairn-offline-fake-v1","package_version":"2.17.0","schema_version":1,"task_set_digest":"13e95379f35839593047ec34521a65a8fa3ab92bd3993bca9dc8bf5c4a9ff36f"}

package/mcp-memory-server/dist/context-pack-cli.js

@@ -21,6 +21,7 @@ function usage() {
   cairn pack skills [--project PATH | --project-id ID] [--json]
   cairn pack approve-skill SELECTOR PATH --confirm FILE_DIGEST [--project PATH | --project-id ID]
   cairn pack revoke-skill SELECTOR PATH [--project PATH | --project-id ID]
+  cairn pack doctor [--repair] [--json]
 `);
     process.exit(2);
 }
@@ -205,9 +206,10 @@ async function main() {
         return;
     }
     if (command === "doctor") {
+        const repair = flag(args, "--repair");
         if (args.length)
             usage();
-        const result = await doctorContextPacks();
+        const result = await doctorContextPacks({ repair });
         output(result, json, result.ok ? `Context packs healthy (${result.objects} objects, ${result.projects} projects).` : `Context pack problems: ${[...result.issues, ...result.temporary_remnants].join("; ")}`);
         if (!result.ok)
             process.exitCode = 1;