@cairnkeep/cli 2.16.0 → 2.17.0

package/CHANGELOG.md

@@ -5,6 +5,30 @@ All notable user-facing changes are documented here. This project follows
 
 ## [Unreleased]
 
+## [2.17.0] - 2026-08-29
+
+### Added
+
+- Frozen, offline retrieval benchmark with substring and deterministic mock-
+  embedding goldens, quality/cost/latency metrics, mutation accounting, and
+  forbidden-result leakage checks.
+- Progressive context-pack hierarchy and derived summary cache, the
+  `context_pack_tree` MCP tool, and opt-in hierarchical search detail and
+  sanitized explanation traces. Compatible flat content search remains the
+  default.
+- Opt-in, privacy-preserving context-usage receipts linked to work evidence.
+- Review-gated `cairn proposals create|list|show|apply|doctor` workflow for
+  promoting redacted trajectory candidates into reviewed durable memory.
+- Provider-neutral domain retrieval with an explicitly gated, read-only
+  OpenViking adapter; AnythingLLM remains the default provider.
+
+### Changed
+
+- Context-pack search can return stable result and chunk digests when requested,
+  without changing the default response shape.
+- `cairn doctor` validates progressive context-cache records while preserving
+  immutable pack objects, project pins, and skill-approval authority.
+
 ## [2.16.0] - 2026-08-27
 
 ### Added

package/README.md

@@ -99,6 +99,7 @@ guides. In particular:
 
 - [Quickstart](docs/quickstart.md)
 - [For coding agents](docs/agents.md)
+- [Context intelligence](docs/context-intelligence.md)
 - [Operating guide](docs/operating.md)
 - [Harness compatibility](docs/harness-compatibility.md)
 - [Privacy and data flow](docs/privacy-and-data-flow.md)

package/docs/README.md

@@ -27,6 +27,14 @@ linked guides rather than the project landing page.
 - [Validated skill improvement](skill-improvement.md) describes the reviewed,
   measured, reversible path for changing a live skill.
 
+## Context intelligence
+
+- [Context intelligence](context-intelligence.md) connects progressive context-pack
+  retrieval, the frozen benchmark, usage receipts, review-gated proposals, and
+  the optional read-only OpenViking provider.
+- [Immutable context packs](context-packs.md), [domain knowledge](domain-knowledge.md),
+  and [work evidence](work-evidence.md) document each boundary in depth.
+
 ## Operations, storage, and privacy
 
 - [Operating Cairnkeep](operating.md) is the full CLI and configuration

package/docs/agents.md

@@ -139,6 +139,26 @@ and file digest. Even then, they are discoverable text through MCP—not copied
 into a harness, automatically activated, or executed. A pack update invalidates
 the old approval. See [Immutable context packs](context-packs.md).
 
+## Progressive retrieval and reviewed-memory proposals
+
+When `context_pack_tree` is available, inspect abstracts before requesting full
+content. Use hierarchical search for broad questions, exact path/title retrieval
+for known leaves, and `explain: true` only when the bounded selection trace helps
+debug retrieval. Default flat search remains valid and compatible.
+
+Stable result references are opt-in (`include_refs: true`). If the operator also
+enables `context_usage_record`, record only whether the cited result was used;
+never put prompts or content into digest fields. Treat recording as a mutation.
+
+`cairn proposals create` may suggest reviewed-memory candidates from a stored,
+redacted trajectory, but an agent must inspect the proposal and use
+`cairn proposals apply DIGEST` explicitly. Never treat extraction output as
+reviewed truth, auto-apply it, or work around a stale-source/base-memory failure.
+
+The optional OpenViking provider changes only how `domain_knowledge_query`
+retrieves external context. It does not replace Cairnkeep's memory, evidence,
+pack, or authority model.
+
 ## Graceful fallback
 
 Cairnkeep should never block ordinary engineering merely because optional

package/docs/context-intelligence.md

@@ -0,0 +1,143 @@
+# Context intelligence
+
+Cairnkeep can retrieve immutable project context progressively, measure retrieval
+quality, record privacy-preserving usage evidence, and propose reviewed-memory
+changes. These features keep the existing authority model: retrieval is an
+observation; durable-memory changes require an explicit review and apply step.
+
+## Progressive context-pack retrieval
+
+Enable context packs for the process with `CAIRN_CONTEXT_PACKS=1`. The existing
+`context_pack_list`, `context_pack_search`, and `context_pack_read` tools remain
+available, and the default search remains the compatible flat, content-level
+search. Two optional surfaces add progressive disclosure:
+
+- `context_pack_tree` returns only enabled documents and approved skills, using
+  `abstract` or `overview` detail.
+- `context_pack_search` accepts `strategy: "hierarchical"`,
+  `detail: "abstract" | "overview" | "content"`, and `explain: true`.
+
+For example:
+
+```json
+{
+  "query": "release validation",
+  "strategy": "hierarchical",
+  "detail": "overview",
+  "explain": true,
+  "include_refs": true
+}
+```
+
+Hierarchical search ranks pack, directory, and file summaries before reading
+content. An exact path or title can bypass the hierarchy and select the matching
+leaf directly. `explain` adds a bounded, sanitized decision trace. It never adds
+hidden file content to the trace.
+
+Abstracts and overviews are deterministic derived data. They are cached below
+the pack base directory in `cache/context/`, outside immutable pack objects, and
+are bound to the pack digest and exact visible-file set. `cairn doctor` validates
+this cache and can remove invalid derived entries; it does not enable a pack or
+approve a skill.
+
+Visibility is unchanged: a document must belong to an enabled pack, while a
+skill must also have a current project approval matching the pack digest, path,
+and file digest. Updating a pack therefore invalidates its old skill approvals.
+Pack SHA-256 digests establish integrity, not publisher authenticity.
+
+## Retrieval benchmark
+
+The repository includes a frozen synthetic benchmark with hierarchical files,
+multiple versions, and a hidden skill. Positive retrieval cases report Hit@1,
+Recall@5, and MRR; negative-only isolation cases report their count, pass rate,
+and leaks separately. The report also covers relevant/irrelevant bytes, an
+estimated token count, latency, network and filesystem mutations, and leakage
+from disabled packs, unapproved skills, case-specific exclusions, or undeclared
+result identities. Frozen substring and deterministic mock-embedding goldens
+make regression checks offline and reproducible.
+
+From a source checkout:
+
+```sh
+npm --prefix mcp-memory-server run build
+npm --prefix mcp-memory-server run check:retrieval-benchmark
+npm --prefix mcp-memory-server run benchmark:retrieval
+```
+
+The first command compiles the TypeScript sources. The second runs the contract
+smoke test and checks the frozen golden. The third rebuilds and emits a current
+report.
+The benchmark is an engineering baseline, not a claim about every real project.
+
+## Context-usage receipts
+
+Search results can include stable references by setting `include_refs: true`.
+Each result then has a `chunk_digest`, and the response has a `result_digest`.
+With work evidence enabled, set `CAIRN_CONTEXT_USAGE=1` as the second consent
+gate to expose the local-only `context_usage_record` MCP mutation, then record
+whether a referenced result was used. The tool is not registered unless both
+`CAIRN_WORK_EVIDENCE=1` and `CAIRN_CONTEXT_USAGE=1` are active:
+
+```json
+{
+  "task_digest": "<64-lowercase-hex-digest>",
+  "result_digest": "<64-lowercase-hex-digest>",
+  "outcome": "used"
+}
+```
+
+`outcome` is `used`, `unused`, or `unknown`. The receipt links to the active work
+evidence unless `evidence_id` is supplied explicitly. It stores digests and the
+outcome, not the query, prompt, retrieved content, or model response. The tool is
+a mutation, so the read-only MCP profile does not expose it. Receipts are strict,
+idempotent links in the existing work-evidence store and follow its retention.
+
+## Review-gated memory proposals
+
+`cairn proposals` turns a stored, redacted trajectory into candidates for
+reviewed durable memory. It never applies candidates during extraction:
+
+```sh
+cairn proposals create --session SESSION_ID --scope project --json
+cairn proposals list --json
+cairn proposals show PROPOSAL_DIGEST --json
+cairn proposals apply PROPOSAL_DIGEST --json
+cairn proposals doctor --json
+```
+
+`create` loads the named local trajectory, redacts it again before any configured
+model call, and writes an immutable, digest-addressed proposal under
+`.agentfs/memory-proposals/`. The digest binds the source, extraction, base memory,
+and candidates. `apply` accepts that exact digest, rechecks source and base-memory
+freshness, and then performs the reviewed-memory write atomically. A stale or
+tampered proposal fails instead of being rebased or silently accepted.
+
+Model extraction uses the existing `CAIRN_LLM_API_URL`, `CAIRN_LLM_API_KEY`, and
+`CAIRN_LLM_EXTRACTION_MODEL` settings. No background hook creates or applies
+proposals.
+
+## Optional OpenViking retrieval provider
+
+AnythingLLM remains the default domain-retrieval provider. OpenViking is an
+optional, read-only adapter behind two explicit settings:
+
+```sh
+CAIRN_DOMAIN_RETRIEVAL_PROVIDER=openviking
+CAIRN_OPENVIKING=1
+CAIRN_OPENVIKING_BASE_URL=http://127.0.0.1:1933
+# Optional:
+CAIRN_OPENVIKING_API_KEY=...
+CAIRN_OPENVIKING_TIMEOUT_MS=5000
+```
+
+The existing `domain_knowledge_query` tool keeps its schema. The adapter calls
+only OpenViking's search endpoint; Cairnkeep does not import, sync, watch, delete,
+or commit OpenViking data. Plain HTTP is accepted only for loopback hosts; other
+hosts require HTTPS. Embedded URL credentials, redirects, oversized responses,
+and invalid response shapes are rejected. Authenticated remote MCP deployment
+also requires the separate `CAIRN_OPENVIKING_MCP_HTTP=1` consent flag.
+
+OpenViking does not replace Cairnkeep. It can supply an external retrieval index;
+Cairnkeep still owns project identity, reviewed memory, immutable context-pack
+pins, tool authority, evidence, and approval boundaries. No OpenViking package or
+service is required for the default Cairnkeep installation.

package/docs/context-packs.md

@@ -95,6 +95,39 @@ Authenticated HTTP deployments need the separate
 project header. Pack digests prove integrity, not publisher authenticity.
 Publisher signatures remain future work.
 
+## Progressive hierarchy and stable result references
+
+The default `context_pack_search` request remains flat content search, preserving
+the established schema and response shape. Agents can opt into progressive
+disclosure without changing pack contents:
+
+```json
+{
+  "query": "deployment rollback",
+  "strategy": "hierarchical",
+  "detail": "overview",
+  "explain": true,
+  "include_refs": true
+}
+```
+
+`strategy` is `flat` or `hierarchical`; `detail` is `abstract`, `overview`, or
+`content`. The optional trace is bounded and sanitized. `context_pack_tree`
+returns the visible hierarchy at `abstract` or `overview` detail, optionally
+limited by pack ID, `id@version`, or digest.
+
+Summaries are deterministic derived cache entries below
+`${CAIRN_PACK_BASE_DIR:-~/.cairnkeep/packs}/cache/context/`. They are bound to the
+pack digest and visible-file-set digest, not stored in immutable objects, and can
+be rebuilt without choosing an enabled version or granting approval. Search with
+`include_refs: true` adds per-result `chunk_digest` values and a response-level
+`result_digest` for optional usage receipts.
+
+Only enabled documents and explicitly approved skills contribute to the tree,
+summary cache, search results, or explanation trace. A skill approval remains
+bound to project identity, pack digest, path, and file digest; a changed pack
+digest cannot inherit it.
+
 ## Open Knowledge Format exchange
 
 Use `cairn pack validate-okf` and `cairn pack import-okf` to consume OKF 0.1 or

package/docs/domain-knowledge.md

@@ -96,6 +96,35 @@ copy of the example JSON at the default user config path, or set
 and state beside the script retain that legacy behavior. To customize the
 implementation itself, set `CAIRN_ANYTHINGLLM_SYNC_SCRIPT` to the adapted copy.
 
+## Retrieval providers
+
+AnythingLLM remains the default. Selecting it explicitly is optional:
+
+```sh
+CAIRN_DOMAIN_RETRIEVAL_PROVIDER=anythingllm
+```
+
+OpenViking can instead back the existing read-only `domain_knowledge_query`
+tool. It is deliberately double-gated and has no sync or write path:
+
+```sh
+CAIRN_DOMAIN_RETRIEVAL_PROVIDER=openviking
+CAIRN_OPENVIKING=1
+CAIRN_OPENVIKING_BASE_URL=http://127.0.0.1:1933
+# CAIRN_OPENVIKING_API_KEY=...
+# CAIRN_OPENVIKING_TIMEOUT_MS=5000
+```
+
+Plain HTTP is limited to loopback. Use HTTPS for any other host. Credentials in
+the URL, redirects, invalid response shapes, and oversized bodies are rejected.
+Authenticated remote MCP use requires the additional explicit consent setting
+`CAIRN_OPENVIKING_MCP_HTTP=1`.
+
+This adapter queries `/api/v1/search/find` only. Cairnkeep does not synchronize
+documents into OpenViking, start its service, or adopt its session and memory
+model. OpenViking is optional external retrieval; Cairnkeep remains the authority
+for reviewed memory, pack pins, approvals, evidence, and MCP profiles.
+
 ## Not using RAG?
 
 Do nothing. The tools stay inert, cost nothing, and the rest of cairnkeep is

package/docs/learning/CURRICULUM-MAP.md

@@ -1,6 +1,6 @@
 # Curriculum Coverage Map
 
-**Baseline:** Cairnkeep 2.16.0
+**Baseline:** Cairnkeep 2.17.0
 **Last reviewed:** 2026-08-19
 
 This map prevents the course from drifting behind the product. It assigns each
@@ -41,6 +41,7 @@ been rehearsed.
 | Pi local stdio memory bridge, trusted annotation details, and explicit lifecycle | L23 | L23 | Ready; Pi 0.84.1 validated minimum |
 | OKF 0.1/0.2 validation/import, provenance, links, regressions, and reviewed 0.2 export | L24 | L24 | Ready; local/offline by default |
 | Bounded project playbooks, generic memory retrieval guidance, enforcement, and private receipts | L25 | L25 | Ready; balanced/local by default; invocation requires runtime evidence; actor identity unverified |
+| Progressive pack retrieval, frozen benchmark, context-usage receipts, review-gated memory proposals, optional provider adapter | L26 | V26 | Ready; default flat search and AnythingLLM remain compatible; OpenViking and receipt mutation are explicitly gated |
 
 ## Release Review
 
@@ -85,6 +86,7 @@ against `cairn help` so a new command cannot bypass a curriculum decision.
 |---|---|
 | `cairn bootstrap` | L03 compatibility challenge |
 | `cairn setup` | L03 for Claude/Codex; advanced Pi lifecycle in L23 |
+| `cairn proposals` | L26 |
 | `cairn memory-server` | L02 |
 | `cairn sync` | L02 |
 | `cairn sync-pi` | L13 and L23 |

package/docs/learning/FEATURE-GUIDE.md

@@ -1,6 +1,6 @@
 # Cairnkeep feature guide
 
-**Baseline:** Cairnkeep 2.16.0
+**Baseline:** Cairnkeep 2.17.0
 
 This is the step-back map for choosing and operating Cairnkeep features. The
 [course labs](https://github.com/cairnkeep/cairnkeep-course-labs) provide one
@@ -117,5 +117,24 @@ prerequisites for ordinary memory.
 - Use `cairn uninstall --dry-run` before uninstall; durable memory is retained
   unless purge is explicitly requested.
 - Revisit the [storage guide](../storage.md) and
-  [privacy/data-flow guide](../privacy-and-data-flow.md) before enabling a new
-  network or retention path.
+[privacy/data-flow guide](../privacy-and-data-flow.md) before enabling a new
+network or retention path.
+
+## Context intelligence
+
+L26 joins five related surfaces without collapsing their authority boundaries:
+
+- the frozen retrieval benchmark measures relevance, context cost, latency,
+  mutations, and forbidden leakage offline;
+- context-pack trees and hierarchical summaries support progressive disclosure,
+  while omitted options retain compatible flat content search;
+- opt-in result digests can produce content-free usage receipts linked to work
+  evidence;
+- trajectory extraction creates an immutable proposal that must be reviewed and
+  applied by exact digest before it becomes durable memory;
+- the OpenViking adapter is optional read-only domain retrieval, with
+  AnythingLLM still the default and Cairnkeep still owning memory and evidence.
+
+See [L26](lessons/L26-context-intelligence.md), its
+[video script](video-scripts/V26-context-intelligence.md), and the
+[context-intelligence reference](../context-intelligence.md).

package/docs/learning/README.md

@@ -60,6 +60,7 @@ expanded only after their labs are executable.
 | L23 | [Guided setup and Pi memory](lessons/L23-guided-setup.md) | Ready | 35 min |
 | L24 | [Reviewed knowledge exchange](lessons/L24-okf-exchange.md) | Ready | 35 min |
 | L25 | [Bounded workflow playbooks](lessons/L25-playbooks.md) | Ready | 35 min |
+| L26 | [Context intelligence with explicit authority](lessons/L26-context-intelligence.md) | Ready | 45 min |
 
 The [curriculum coverage map](CURRICULUM-MAP.md) records where every public
 feature is introduced, practised, and operated. It also marks design-only work

package/docs/learning/lessons/L00-why-cairnkeep.md

@@ -3,7 +3,7 @@
 **Status:** Ready
 **Track:** Quickstart
 **Time:** 10 minutes
-**Tested with:** Cairnkeep 2.16.0
+**Tested with:** Cairnkeep 2.17.0
 
 ## Outcome
 

package/docs/learning/lessons/L01-safe-trial.md

@@ -3,7 +3,7 @@
 **Status:** Ready
 **Track:** Quickstart
 **Time:** 15 minutes
-**Tested with:** Cairnkeep 2.16.0
+**Tested with:** Cairnkeep 2.17.0
 
 ## Outcome
 
@@ -26,7 +26,7 @@ ephemeral; the named course volume is persistent.
 1. Pull the public image:
 
    ```bash
-   podman pull ghcr.io/cairnkeep/cairnkeep:2.16.0
+   podman pull ghcr.io/cairnkeep/cairnkeep:2.17.0
    ```
 
 2. Start the stdio server with a read-only root filesystem and no Linux
@@ -38,7 +38,7 @@ ephemeral; the named course volume is persistent.
      --read-only --cap-drop=all --security-opt=no-new-privileges \
      --tmpfs=/tmp:rw,noexec,nosuid,size=64m,mode=1777 \
      --volume cairnkeep-course-data:/data:Z,U \
-     ghcr.io/cairnkeep/cairnkeep:2.16.0 stdio
+     ghcr.io/cairnkeep/cairnkeep:2.17.0 stdio
    ```
 
 3. The process waits quietly for MCP input. Press `Ctrl-C` after confirming it

package/docs/learning/lessons/L02-installation.md

@@ -3,7 +3,7 @@
 **Status:** Ready
 **Track:** Quickstart
 **Time:** 25 minutes
-**Tested with:** Cairnkeep 2.16.0 and Node.js 22 or newer
+**Tested with:** Cairnkeep 2.17.0 and Node.js 22 or newer
 
 ## Outcome
 
@@ -47,7 +47,7 @@ sync. Codex project trust remains an explicit operator decision.
 2. Install Cairnkeep:
 
    ```bash
-   npm install --global @cairnkeep/cli@2.16.0
+   npm install --global @cairnkeep/cli@2.17.0
    cairn --version
    ```
 

package/docs/learning/lessons/L03-first-project.md

@@ -3,7 +3,7 @@
 **Status:** Ready
 **Track:** Quickstart
 **Time:** 25 minutes
-**Tested with:** Cairnkeep 2.16.0
+**Tested with:** Cairnkeep 2.17.0
 
 ## Outcome
 

package/docs/learning/lessons/L19-validated-skill-improvement.md

@@ -3,7 +3,7 @@
 **Status:** Ready
 **Track:** Evidence and Evaluation
 **Time:** 50 minutes
-**Tested with:** Cairnkeep 2.16.0
+**Tested with:** Cairnkeep 2.17.0
 
 ## Outcome
 

package/docs/learning/lessons/L20-mcp-tool-profiles.md

@@ -1,7 +1,7 @@
 # L20 - Least-authority MCP tool profiles
 
 **Status:** Ready
-**Tested with:** Cairnkeep 2.16.0 and Node.js 22 or newer
+**Tested with:** Cairnkeep 2.17.0 and Node.js 22 or newer
 **Time:** 25 minutes
 
 ## Outcome

package/docs/learning/lessons/L21-context-packs.md

@@ -1,7 +1,7 @@
 # L21 - Immutable context packs and skill approval
 
 **Status:** Ready
-**Tested with:** Cairnkeep 2.16.0 and Node.js 22 or newer
+**Tested with:** Cairnkeep 2.17.0 and Node.js 22 or newer
 **Time:** 40 minutes
 
 ## Outcome

package/docs/learning/lessons/L22-native-windows.md

@@ -3,7 +3,7 @@
 **Status:** Ready
 **Track:** Operator
 **Time:** 30 minutes
-**Tested with:** Cairnkeep 2.16.0, Windows x64, and Node.js 22 or newer
+**Tested with:** Cairnkeep 2.17.0, Windows x64, and Node.js 22 or newer
 
 ## Outcome
 
@@ -16,7 +16,7 @@ perform a reversible uninstall without relying on WSL or Git Bash.
 From PowerShell:
 
 ```powershell
-npm install --global @cairnkeep/cli@2.16.0
+npm install --global @cairnkeep/cli@2.17.0
 cairn version
 
 $Lab = Join-Path $env:TEMP 'Cairnkeep Windows Lab'

package/docs/learning/lessons/L23-guided-setup.md

@@ -3,7 +3,7 @@
 **Status:** Ready
 **Track:** Quickstart and Operator
 **Time:** 35 minutes
-**Tested with:** Cairnkeep 2.16.0; Pi 0.84.1; Node.js 22, 24, and 26
+**Tested with:** Cairnkeep 2.17.0; Pi 0.84.1; Node.js 22, 24, and 26
 
 ## Outcome
 

package/docs/learning/lessons/L24-okf-exchange.md

@@ -1,7 +1,7 @@
 # L24 - Reviewed knowledge exchange
 
 **Status:** Ready
-**Tested with:** Cairnkeep 2.16.0 and Node.js 22 or newer
+**Tested with:** Cairnkeep 2.17.0 and Node.js 22 or newer
 **Time:** 35 minutes
 
 ## Outcome

package/docs/learning/lessons/L25-playbooks.md

@@ -1,7 +1,7 @@
 # L25 - Bounded workflow playbooks
 
 **Status:** Ready
-**Tested with:** Cairnkeep 2.16.0 and Node.js 22 or newer
+**Tested with:** Cairnkeep 2.17.0 and Node.js 22 or newer
 **Time:** 35 minutes
 
 ## Outcome