@cairnkeep/cli 2.7.0 → 2.9.0

package/docs/privacy-and-data-flow.md

@@ -16,7 +16,10 @@ request unless the corresponding endpoint and credential are configured.
 | `domain_knowledge_sync` | Files selected by the sync configuration | `ANYTHINGLLM_BASE_URL` |
 | `route_check` | A health request, with no memory or prompt content | `CAIRN_ROUTE_ENDPOINT` |
 | `context_explore` | Repository path and query are passed to the configured local executable | `CAIRN_EXPLORE_BINARY`; any further data flow is controlled by that tool |
-| Optional Graphify workflow | The repository path, exact query/symbol names, and local published graph path are passed to the operator-installed `graphify` executable | Local Graphify subprocess with a minimal environment and no provider credentials; managed build uses code-only `update`, never semantic document extraction |
+| Local context-pack install/list/read/substring search | None | Verified immutable files and project pointers under `CAIRN_PACK_BASE_DIR` |
+| Embedding-ranked context-pack search | Enabled file chunks not already cached, plus the query | The explicitly configured embedding endpoint; failure falls back locally |
+| Authenticated HTTP context-pack retrieval | MCP requests and enabled document/approved-skill content | The explicit Cairnkeep HTTP server; requires separate pack HTTP consent |
+| Optional Graphify workflow | The repository path, exact query/symbol names, root `graphify-out/` work directory, and local `.planning/graphs/` publication are passed to or produced by the operator-installed `graphify` executable | Local Graphify subprocess with a minimal environment and no provider credentials; managed build uses code-only `update`, never semantic document extraction |
 | Remote HTTP memory | MCP requests and responses, including memory content | The explicitly registered Cairnkeep HTTP server |
 | Opt-in trajectory capture | None | Local `<project>/.agentfs/trajectory.db` only; no model or HTTP path exists |
 | Opt-in capability callback records | None | Payload-free final metadata in local `<project>/.agentfs/trajectory.db`; HTTP transport is always skipped |
@@ -175,6 +178,47 @@ the full committed population and note-eligible subset with paired counts and
 seeded uncertainty; they do not convert framework output into a causal,
 significance, quality, or efficiency claim.
 
+## Validated skill improvement flow
+
+Harvest and review are local-only operations over project hindsight notes.
+Before approval, no proposal subprocess is started and no evidence crosses a
+process boundary. After explicit approval, the configured proposal adapter
+receives exactly one bounded JSON request containing the approved candidate,
+candidate digest, selected target path, current target content and digest, and
+edit budget.
+
+The proposal adapter receives an isolated private HOME and TMPDIR, a minimal
+PATH and locale, and only environment variables named in its strict
+`environment_allowlist`. It does not inherit the rest of the parent
+environment. HOME, PATH, temporary and XDG roots, and Node runtime injection
+variables are isolation-controlled and cannot be allowlisted. Its stdout must
+be one strict bounded response containing status
+and bounded edits; arbitrary stderr is live operator diagnostics and is not
+stored. Cairnkeep records the adapter executable digest and rejects a change
+during the invocation. The adapter may perform network access according to the
+executable and host policy, but Cairnkeep provides no endpoint, credential, or
+network default. Operators must treat approval as consent to send those exact
+evidence and target fields to that adapter.
+
+The reduced environment is not an operating-system sandbox. The explicit
+adapter executable retains the invoking user's filesystem and network
+authority, so operators must trust it or wrap it in a separately enforced host
+sandbox.
+
+Evaluation sends neither candidate evidence nor proposal rationales to the
+harness adapter. It overlays baseline or candidate skill bytes into separate
+fresh worktrees and otherwise uses the existing evaluation request contract.
+Exploration and confirmation use disjoint task IDs and definitions. Confirmation
+is not opened unless exploration passes, and its results cannot be supplied
+back to proposal generation by this workflow. Both stages bind one evaluation
+adapter executable digest and reject a change during either stage.
+
+Apply and rollback invoke no model and make no network request. They operate on
+one contained regular target and private local artifacts, require exact
+digests, reject symlinks, and fail closed on concurrent target changes. See
+[storage](storage.md#validated-skill-storage) for retained bytes and
+[Validated skill improvement](skill-improvement.md) for the operator sequence.
+
 ## Compaction and artifact flows
 
 Both `CAIRN_COMPACTION_CAPTURE` and `CAIRN_ARTIFACT_STORE` are default-off and
@@ -471,3 +515,17 @@ provide tenant isolation or per-scope authorization.
 Before sharing diagnostics, remove credentials, private endpoints, database
 files, memory values, local paths, and project names. Report vulnerabilities
 through the private channel in [SECURITY.md](../SECURITY.md).
+
+## Context-pack trust boundary
+
+Pack validation proves that installed bytes match the manifest and immutable
+digest. It does not authenticate a publisher. Review the source, commit, license,
+and content before enabling a digest; signatures are not yet supported. Local
+directory installation, listing, reading, and substring search make no network
+request. Git installation/update contacts only the operator-supplied source and
+is never scheduled in the background.
+
+All enabled documents carry pack and file provenance in MCP results. Skill files
+are excluded until the current project approves the exact pack digest, path, and
+file digest. Updating the pack invalidates that approval. Approval only makes a
+skill readable; Cairnkeep never copies it into a harness or executes it.

package/docs/skill-improvement.md

@@ -0,0 +1,124 @@
+# Validated skill improvement
+
+`cairn skill` turns recurring project hindsight into a proposed change to one
+existing skill file, then requires independent evidence before that change can
+be applied. It is an operator-controlled lifecycle, not an autonomous rewrite
+loop.
+
+## Safety contract
+
+- Harvest reads only project hindsight notes and requires failures from at
+  least two distinct sessions by default.
+- Evidence remains local until `review --approve` records explicit approval.
+- Proposal generation invokes only the configured executable, sends one strict
+  bounded JSON request, exposes only allowlisted environment variables, and
+  accepts a bounded edit list rather than an arbitrary rewritten tree.
+- Proposal and evaluation artifacts bind the adapter executable digest;
+  evaluation rechecks it after exploration and confirmation.
+- Proposal and evaluation use candidate content without modifying the live
+  target.
+- Evaluation requires repository-root tasks in two different committed task
+  sets with at least two tasks each. Task IDs and exact task definitions cannot
+  overlap, and both sets must bind the same immutable source revision.
+- Confirmation runs only after exploration passes. Confirmation results never
+  return to proposal generation.
+- Apply requires an eligible exact evaluation, an unchanged target, and the
+  full proposal digest. It creates a private backup before atomic replacement.
+- Rollback refuses to overwrite a target changed after application.
+
+## Prepare a target and proposal adapter
+
+The target is one existing project-relative regular file named `SKILL.md`, for example
+`.claude/skills/generated-client-review/SKILL.md`. It cannot be a symlink and
+is limited to 256 KiB.
+
+The proposal adapter configuration is strict JSON:
+
+```json
+{
+  "schema_version": 1,
+  "id": "local-skill-proposer",
+  "command": {
+    "program": "/absolute/path/to/proposal-adapter",
+    "args": []
+  },
+  "environment_allowlist": ["PROPOSAL_PROVIDER_TOKEN"],
+  "limits": {
+    "elapsed_ms": 120000,
+    "stdout_bytes": 1048576
+  }
+}
+```
+
+The executable receives the approved candidate, its digest, current target
+content and digest, and edit budget on stdin. It returns `add`, `replace`, or
+`delete` edits with exact anchors and rationales. The executable path must be
+absolute, executable, and not a symlink. Cairnkeep does not provide a model,
+endpoint, credential, or network default. The machine-readable contracts are
+[`schemas/skill-adapter.schema.json`](../schemas/skill-adapter.schema.json) and
+[`schemas/skill-proposal-protocol.schema.json`](../schemas/skill-proposal-protocol.schema.json).
+Process-isolation variables such as `HOME`, `PATH`, temporary directories,
+XDG roots, and Node runtime injection settings cannot be allowlisted.
+This environment boundary is not an operating-system sandbox: the configured
+program runs with the invoking user's filesystem and network authority. Use
+only a trusted adapter or add a separate host sandbox.
+
+## Run the lifecycle
+
+```bash
+# 1. Find recurring hindsight evidence. This performs no model call.
+cairn skill harvest --project . --json
+cairn skill show --kind candidate --id CANDIDATE_ID
+
+# 2. Approve the exact evidence that may cross the adapter boundary.
+cairn skill review --candidate CANDIDATE_ID --approve --json
+
+# 3. Generate a bounded proposal. The live target remains unchanged.
+cairn skill propose --candidate CANDIDATE_ID \
+  --target .claude/skills/generated-client-review/SKILL.md \
+  --adapter ./private/proposal-adapter.json --edit-budget 4 --json
+
+# 4. Measure baseline versus candidate in fresh isolated worktrees.
+export CAIRN_EVAL=1
+cairn skill evaluate --proposal PROPOSAL_ID \
+  --exploration-task-set eval/skill-exploration.json \
+  --confirmation-task-set eval/skill-confirmation.json \
+  --adapter eval/harness-adapter.json \
+  --repetitions 2 --minimum-improvement 1 --yes --json
+
+# 5. Inspect artifacts and apply only the exact eligible proposal.
+cairn skill show --kind proposal --id PROPOSAL_ID
+cairn skill show --kind evaluation --id EVALUATION_ID
+cairn skill apply --proposal PROPOSAL_ID --evaluation EVALUATION_ID \
+  --confirm FULL_PROPOSAL_DIGEST --json
+
+# 6. Revert if later evidence warrants it.
+cairn skill rollback --application APPLICATION_ID --confirm --json
+```
+
+`propose --json` includes `proposal_digest`, which is the value required by
+`apply --confirm`. `list` shows every artifact class; use `--kind candidate`,
+`proposal`, `evaluation`, or `application` to narrow it.
+
+## Evaluation semantics
+
+Each task runs twice in each arm. The baseline arm overlays the exact unchanged
+target content; the treatment arm overlays the exact candidate content. Every
+observation must complete, return the expected capability digest, and receive a
+known result from the independent verifier. A task with any candidate-only pass
+counts once as improved; a task with any baseline-only pass counts once as
+regressed and rejects the candidate.
+
+Exploration must have no unknown pairs, no regression, and at least
+`--minimum-improvement` improvements. Otherwise evaluation stops without
+opening the confirmation set. Confirmation must independently satisfy the same
+gate. This is deliberately strict and suited to small, high-value task sets;
+it is not a statistical proof that a skill is universally better.
+
+## What this does not do
+
+It does not scrape native harness transcripts, schedule periodic self-edits,
+invent task sets, approve evidence, choose an inference provider, or publish a
+changed skill. These remain explicit operator responsibilities. Start with a
+small target and representative tasks; expand only when the stored evaluation
+artifacts make the decision auditable.

package/docs/storage.md

@@ -21,6 +21,9 @@ registration, memory remains on that computer.
 | Opt-in session trajectories and capability callback state | `<harness project root>/.agentfs/trajectory.db` |
 | Opt-in local artifacts and compaction revisions | `<project root>/.agentfs/artifacts.db` |
 | Opt-in evaluation reports and note snapshots | `<project root>/.agentfs/eval/experiments/<experiment-id>/` |
+| Immutable context-pack objects and source records | `${CAIRN_PACK_BASE_DIR:-~/.cairnkeep/packs}/objects/` and `sources/` |
+| Digest-pinned context-pack project pointers and approvals | `${CAIRN_PACK_BASE_DIR:-~/.cairnkeep/packs}/projects/` |
+| Optional context-pack embedding cache | `${CAIRN_PACK_BASE_DIR:-~/.cairnkeep/packs}/cache/` (outside immutable objects) |
 | Opt-in derived hindsight notes | `${CAIRN_AGENTFS_BASE_DIR:-~/.cairnkeep}/notes/` |
 | Any named/global scope, such as `identity` or `work` | `${CAIRN_AGENTFS_BASE_DIR:-~/.cairnkeep}/<scope>.db` |
 
@@ -213,6 +216,31 @@ the process stores databases below `/data`, normally backed by the
 removing the volume removes the databases. See [Containers](containers.md) for
 the exact paths and backup boundary.
 
+## Validated skill storage
+
+The skill-improvement lifecycle stores private JSON artifacts below
+`<project>/.agentfs/skills/`: `candidates/`, `proposals/`, `evaluations/`, and
+`applications/`. Pre-application target backups live in `backups/`; isolated
+proposal-adapter homes are created temporarily in `adapter-tmp/` and removed
+after every invocation. Directories are mode `0700`, JSON and backup files are
+mode `0600`, artifacts are bounded to 2 MiB, and candidate target content is
+bounded to 256 KiB.
+
+Candidate and proposal artifacts can contain hindsight excerpts and full skill
+content, so the entire directory is sensitive. Evaluation reports and
+experiment-owned snapshots remain in `.agentfs/eval/experiments/` under the
+existing evaluation retention rules. No skill artifact is written to global or
+remote memory, and remote MCP configuration does not redirect this project
+state.
+
+`apply` writes the original target to a unique private backup before atomically
+replacing the target. If replacement, digest verification, or ledger storage
+fails, Cairnkeep restores the original and removes the incomplete backup.
+`rollback` restores only when both the current target and backup match the
+application ledger, preventing an unrelated later edit from being overwritten.
+Default uninstall retains `.agentfs/`; explicit purge uses the existing
+backup-first project boundary.
+
 ## Remote HTTP mode
 
 Remote storage is explicit. An operator starts Cairnkeep in HTTP mode on a
@@ -468,3 +496,17 @@ bearer_token_env_var = "CAIRN_MEMORY_HTTP_TOKEN"
 "X-Cairn-Scopes" = "identity,personal,project"
 "X-Cairn-AnythingLLM-Workspaces" = "engineering-patterns"
 ```
+
+## Context-pack storage
+
+Context-pack objects are content-addressed, immutable directories. Installation
+copies verified content through a temporary directory and atomically publishes
+the digest; a failed or concurrent install never changes project enablement.
+Project pointers are separate atomic mode-`0600` files keyed by canonical local
+project identity or a validated remote project ID. Updates retain old objects
+and switch only the named project after exact-digest confirmation.
+
+`cairn doctor` verifies objects, pointers, and interrupted-install remnants.
+It never chooses a pack version or grants a skill approval. `cairn pack remove`
+refuses any referenced digest. Ordinary uninstall retains this directory even
+when memory is removed; `--purge-packs` is the separate backup-first consent.

package/examples/eval/bundled-fake.json

@@ -1 +1 @@
-{"identifier":"cairn-offline-fake-v1","package_version":"2.7.0","schema_version":1,"task_set_digest":"13e95379f35839593047ec34521a65a8fa3ab92bd3993bca9dc8bf5c4a9ff36f"}
+{"identifier":"cairn-offline-fake-v1","package_version":"2.9.0","schema_version":1,"task_set_digest":"13e95379f35839593047ec34521a65a8fa3ab92bd3993bca9dc8bf5c4a9ff36f"}

package/mcp-memory-server/dist/context-pack-cli.js

@@ -0,0 +1,185 @@
+import { existsSync } from "node:fs";
+import { rm } from "node:fs/promises";
+import { resolve } from "node:path";
+import { applyContextPackUpdate, approvePackSkill, contextPackBaseDirectory, disableContextPack, doctorContextPacks, enableContextPack, initializeContextPack, inspectContextPackUpdate, installContextPack, listInstalledContextPacks, listPackSkills, lockContextPack, readProjectPointer, removeContextPack, resolveInstalledPack, revokePackSkill, validateContextPack, } from "./context-pack.js";
+function usage() {
+    process.stderr.write(`Usage:
+  cairn pack init [DIRECTORY] [--id ID] [--version VERSION] [--title TITLE] [--description TEXT] [--license LICENSE]
+  cairn pack lock|validate [DIRECTORY]
+  cairn pack install SOURCE [--ref REF]
+  cairn pack list [--json]
+  cairn pack show SELECTOR [--json]
+  cairn pack remove SELECTOR
+  cairn pack enable|disable SELECTOR [--project PATH | --project-id ID]
+  cairn pack update SELECTOR --check [--project PATH | --project-id ID]
+  cairn pack update SELECTOR --apply --confirm DIGEST [--project PATH | --project-id ID]
+  cairn pack skills [--project PATH | --project-id ID] [--json]
+  cairn pack approve-skill SELECTOR PATH --confirm FILE_DIGEST [--project PATH | --project-id ID]
+  cairn pack revoke-skill SELECTOR PATH [--project PATH | --project-id ID]
+`);
+    process.exit(2);
+}
+function take(args, name) {
+    const index = args.indexOf(name);
+    if (index < 0)
+        return undefined;
+    const value = args[index + 1];
+    if (!value || value.startsWith("--"))
+        usage();
+    args.splice(index, 2);
+    return value;
+}
+function flag(args, name) {
+    const index = args.indexOf(name);
+    if (index < 0)
+        return false;
+    args.splice(index, 1);
+    return true;
+}
+function projectOptions(args) {
+    const projectRoot = take(args, "--project");
+    const projectId = take(args, "--project-id");
+    if (projectRoot && projectId)
+        usage();
+    return projectId ? { projectId } : { projectRoot: projectRoot ?? process.cwd() };
+}
+function output(value, json, human) {
+    process.stdout.write(`${json ? JSON.stringify(value) : human ?? JSON.stringify(value, null, 2)}\n`);
+}
+async function main() {
+    const args = process.argv.slice(2);
+    const command = args.shift();
+    const json = flag(args, "--json");
+    if (command === "init") {
+        const directory = args[0] && !args[0].startsWith("--") ? args.shift() : process.cwd();
+        const manifest = await initializeContextPack(directory, {
+            id: take(args, "--id"), version: take(args, "--version"), title: take(args, "--title"),
+            description: take(args, "--description"), license: take(args, "--license"),
+        });
+        if (args.length)
+            usage();
+        output(manifest, json, `Initialized ${resolve(directory)}/context-pack.json`);
+        return;
+    }
+    if (command === "lock" || command === "validate") {
+        const directory = args.shift() ?? process.cwd();
+        if (args.length)
+            usage();
+        const result = command === "lock" ? await lockContextPack(directory) : await validateContextPack(directory);
+        output(result, json, command === "lock" ? `Locked ${resolve(directory)}/context-pack.json` : `Valid context pack: ${result.digest}`);
+        return;
+    }
+    if (command === "install") {
+        const source = args.shift();
+        if (!source)
+            usage();
+        const ref = take(args, "--ref");
+        if (args.length)
+            usage();
+        const result = await installContextPack(source, { ref });
+        output({ id: result.pack.manifest.id, version: result.pack.manifest.version, digest: result.pack.digest, source: result.source, existing: result.existing }, json, `${result.existing ? "Already installed" : "Installed"} ${result.pack.manifest.id}@${result.pack.manifest.version} ${result.pack.digest}`);
+        return;
+    }
+    if (command === "list") {
+        if (args.length)
+            usage();
+        const packs = await listInstalledContextPacks();
+        output({ schema_version: 1, packs }, json, packs.length ? packs.map((pack) => `${pack.id}@${pack.version}\t${pack.digest}`).join("\n") : "No context packs installed.");
+        return;
+    }
+    if (command === "show") {
+        const selector = args.shift();
+        if (!selector || args.length)
+            usage();
+        const pack = await resolveInstalledPack(selector);
+        output({ ...pack.manifest, digest: pack.digest, total_bytes: pack.total_bytes }, json);
+        return;
+    }
+    if (command === "remove") {
+        const selector = args.shift();
+        if (!selector || args.length)
+            usage();
+        const digest = await removeContextPack(selector);
+        output({ removed: true, digest }, json, `Removed context pack ${digest}.`);
+        return;
+    }
+    if (command === "enable" || command === "disable") {
+        const selector = args.shift();
+        if (!selector)
+            usage();
+        const project = projectOptions(args);
+        if (args.length)
+            usage();
+        const pointer = command === "enable" ? await enableContextPack(selector, project) : await disableContextPack(selector, project);
+        output(pointer, json, `${command === "enable" ? "Enabled" : "Disabled"} context pack ${selector}.`);
+        return;
+    }
+    if (command === "update") {
+        const selector = args.shift();
+        if (!selector)
+            usage();
+        const check = flag(args, "--check");
+        const apply = flag(args, "--apply");
+        const confirm = take(args, "--confirm");
+        const project = projectOptions(args);
+        if (args.length || check === apply || (apply && !confirm) || (check && confirm))
+            usage();
+        const result = check ? await inspectContextPackUpdate(selector, project) : await applyContextPackUpdate(selector, confirm, project);
+        output(result, json, check ? `${result.changed ? "Update available" : "Up to date"}: ${result.candidate_digest}` : `Applied context pack update ${result.candidate_digest}.`);
+        return;
+    }
+    if (command === "skills") {
+        const project = projectOptions(args);
+        if (args.length)
+            usage();
+        const skills = await listPackSkills(project);
+        output({ schema_version: 1, skills }, json, skills.length ? skills.map((row) => `${row.approved ? "approved" : "unapproved"}\t${row.pack_id}\t${row.path}\t${row.file_digest}`).join("\n") : "No context pack skills.");
+        return;
+    }
+    if (command === "approve-skill" || command === "revoke-skill") {
+        const selector = args.shift();
+        const path = args.shift();
+        if (!selector || !path)
+            usage();
+        const confirm = take(args, "--confirm");
+        const project = projectOptions(args);
+        if (args.length || (command === "approve-skill" && !confirm) || (command === "revoke-skill" && confirm))
+            usage();
+        const pointer = command === "approve-skill"
+            ? await approvePackSkill(selector, path, confirm, project)
+            : await revokePackSkill(selector, path, project);
+        output(pointer, json, `${command === "approve-skill" ? "Approved" : "Revoked"} ${path}.`);
+        return;
+    }
+    if (command === "doctor") {
+        if (args.length)
+            usage();
+        const result = await doctorContextPacks();
+        output(result, json, result.ok ? `Context packs healthy (${result.objects} objects, ${result.projects} projects).` : `Context pack problems: ${[...result.issues, ...result.temporary_remnants].join("; ")}`);
+        if (!result.ok)
+            process.exitCode = 1;
+        return;
+    }
+    if (command === "purge") {
+        if (!flag(args, "--yes") || args.length)
+            usage();
+        const base = contextPackBaseDirectory();
+        if (existsSync(base))
+            await rm(base, { recursive: true, force: true });
+        output({ purged: true, path: base }, json, `Purged context pack data at ${base}.`);
+        return;
+    }
+    if (command === "status") {
+        const project = projectOptions(args);
+        if (args.length)
+            usage();
+        const pointer = readProjectPointer(project);
+        output(pointer, json);
+        return;
+    }
+    usage();
+}
+main().catch((error) => {
+    process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
+    process.exitCode = 1;
+});