@cairnkeep/cli 2.6.0 → 2.8.0

package/docs/learning/tracks/evidence-and-evaluation.md

@@ -2,7 +2,7 @@
 
 **Audience:** maintainers and advanced practitioners evaluating optional layers
 **Prerequisite:** Practitioner track
-**Total time:** about 3.25 hours
+**Total time:** about 4 hours
 **Infrastructure:** local fixtures first; a live harness is operator-provided
 
 1. [L13 - Session evidence and hindsight](../lessons/L13-session-evidence.md)
@@ -10,11 +10,13 @@
 3. [L15 - Capability governance](../lessons/L15-capability-governance.md)
 4. [L16 - Evaluation and ablation](../lessons/L16-evaluation.md)
 5. [L17 - The agent boundary](../lessons/L17-agent-boundary.md)
+6. [L19 - Validated skill improvement](../lessons/L19-validated-skill-improvement.md)
 
 The track is complete when the learner can explain each consent boundary,
 inspect and manage every local evidence store, compare baseline and ablation
-reports without making unsupported claims, and distinguish current runtime
-features from the design-only meta-agent contract.
+reports without making unsupported claims, apply or roll back an exact eligible
+skill proposal, and distinguish that narrow runtime from the design-only
+general meta-agent contract.
 
 Hands-on spine: `course-05-evidence` through `course-07-evaluation` in the
 public [course labs](https://github.com/cairnkeep/cairnkeep-course-labs).

package/docs/learning/tracks/practitioner.md

@@ -6,10 +6,11 @@
 
 1. [L04 - Memory fundamentals](../lessons/L04-memory-fundamentals.md)
 2. [L05 - The daily workflow](../lessons/L05-daily-workflow.md)
-3. [L06 - Repository review and security](../lessons/L06-repository-quality.md)
-4. [L07 - Privacy and storage](../lessons/L07-privacy-and-storage.md)
-5. [L08 - Multiple machines](../lessons/L08-multiple-machines.md)
-6. [L13 - Session evidence and hindsight](../lessons/L13-session-evidence.md)
+3. [L18 - Local code graph](../lessons/L18-local-code-graph.md)
+4. [L06 - Repository review and security](../lessons/L06-repository-quality.md)
+5. [L07 - Privacy and storage](../lessons/L07-privacy-and-storage.md)
+6. [L08 - Multiple machines](../lessons/L08-multiple-machines.md)
+7. [L13 - Session evidence and hindsight](../lessons/L13-session-evidence.md)
 
 The track is complete when the learner can decide what should become memory,
 review derived knowledge, run governed repository checks, locate the store, and

package/docs/operating.md

@@ -17,9 +17,10 @@ installs. This guide covers all three in order.
 ## Prerequisites
 
 - Node.js 22 or newer (for the memory server) and a supported harness. Claude
-  Code and OpenCode receive the full operating layer. Kimi Code and Qwen Code
-  receive the memory MCP and launcher. Pi receives its native trajectory
-  adapter.
+  Code and OpenCode receive the full operating layer. Kimi Code receives the
+  memory MCP, launcher, and opt-in graph Skill; Qwen Code receives the memory
+  MCP and launcher. Pi receives its native trajectory adapter, graph prompt,
+  and launcher.
 - Optional: the `sqlite3` CLI for `cairn memory export`. Runtime memory and
   `cairn memory import` do not require it.
 - Optional: an OpenAI-compatible LLM endpoint for memory extraction and
@@ -159,10 +160,85 @@ Model selection stays operator-env-driven through the existing
 variables (see "Configuration" below) — the harness commits no default model
 and carries no known-good allowlist.
 
+## Optional Graphify workflow
+
+Graphify is an optional local structural index. Install only its isolated CLI:
+
+```bash
+uv tool install graphifyy
+# or: pipx install graphifyy
+```
+
+Do not run `graphify install`: Cairnkeep owns the harness commands, policies,
+and hooks, so Graphify-owned assets would duplicate that operating layer.
+With the managed capability contract enabled, run
+`cairn capabilities enable graph`; otherwise enable the compatibility setting
+`graphify.enabled` in `.planning/config.json`. Then use the managed command
+surface. Claude Code and OpenCode receive the native wrapper:
+
+```text
+/graphify build
+/graphify status
+/graphify query putTrajectory
+/graphify explain putTrajectory
+/graphify path putTrajectory resolveCapabilityStatus
+/graphify diff
+```
+
+Install the equivalent thin adapters for Kimi Code and Pi:
+
+```bash
+cairn sync-kimi --apply
+cairn sync-pi --apply
+```
+
+Kimi then exposes `/graphify` and `/skill:graphify`; Pi exposes `/graphify`.
+Both adapters delegate exclusively to the same portable CLI, which also remains
+available directly from their project shell:
+
+```bash
+cairn graph build
+cairn graph status
+cairn graph query putTrajectory
+cairn graph explain putTrajectory
+cairn graph path putTrajectory resolveCapabilityStatus
+cairn graph diff
+```
+
+The same direct CLI works with Qwen Code, Codex, and other shell-capable
+clients. This does not add a Pi MCP bridge or imply that those harnesses receive
+the rest of Cairnkeep's operating layer.
+
+All six modes delegate to `cairn graph`. Managed builds run Graphify's local
+code-only `update` path with provider credentials removed from the subprocess
+environment, then atomically publish validated artifacts under
+`.planning/graphs/`. Previous published artifacts are isolated during the scan
+so Graphify cannot re-index its own generated HTML/report/JSON. Managed builds
+do not perform Graphify's optional semantic document extraction. Prefer exact
+function, class, or file names; broad natural-language graph queries are less
+precise. Cairnkeep does not install or update Graphify automatically.
+
+Graphify also maintains an incremental `graphify-out/` work directory at the
+repository root. It is separate from Cairnkeep's published
+`.planning/graphs/` view and can contain the same sensitive structural data.
+Keep both locations local. Before the first build in a Git repository, add
+`/graphify-out/` to the clone-local exclude file or the repository's ignore
+policy, and verify with `git status --short` after every build.
+
 ## Setup order (Kimi Code)
 
-Kimi Code can use the Cairnkeep memory MCP and the generated project launcher.
-It does not yet receive Cairnkeep-specific skills, commands, or hooks.
+Kimi Code can use the Cairnkeep memory MCP, generated project launcher, and one
+thin graph Skill. Install the Skill into `$KIMI_CODE_HOME` (default
+`~/.kimi-code`) with:
+
+```bash
+cairn sync-kimi --apply
+```
+
+Use `cairn sync-kimi --check` to detect drift or `--live-root DIR` for an
+isolated Kimi root. The command owns only `skills/graphify/SKILL.md`; `cairn
+uninstall --kimi-live-root DIR` removes it backup-first. It adds no hooks,
+agents, plugins, or new data flow.
 
 For local stdio memory, create `.kimi-code/mcp.json` in the project:
 
@@ -223,8 +299,8 @@ configuration, approval behavior, and overlay guidance.
 
 ## Setup order (Pi)
 
-Pi trajectory capture uses the same project scaffold and store, but its source
-is a native TypeScript extension installed into Pi's agent root:
+Pi trajectory capture uses the same project scaffold and store. Its sync also
+installs a thin `/graphify` prompt into Pi's agent root:
 
 ```bash
 npm install -g @cairnkeep/cli
@@ -240,8 +316,9 @@ a user-chosen Pi extension/bridge separately.
 
 Use `cairn sync-pi --check` to report drift without writing, or
 `--live-root DIR` to target an isolated Pi agent root. The command owns exactly
-`extensions/cairnkeep-trajectory.ts`; `cairn uninstall --pi-live-root DIR`
-removes that file backup-first and leaves every other Pi asset untouched.
+`extensions/cairnkeep-trajectory.ts` and `prompts/graphify.md`; `cairn
+uninstall --pi-live-root DIR` removes those files backup-first and leaves every
+other Pi asset untouched.
 
 The extension listens for Pi's native `session_shutdown` event and reads only
 the active branch from the read-only session manager. It returns before doing
@@ -827,6 +904,22 @@ See [storage](storage.md#evaluation-report-and-note-snapshot-storage) for local
 retention/removal and [privacy](privacy-and-data-flow.md#evaluation-adapter-and-report-flow)
 for the exact request, observation, environment, and persistence boundaries.
 
+### Validated skill improvement (opt-in)
+
+After hindsight-note distillation has recorded the same failure family in at
+least two sessions, `cairn skill harvest` creates a local candidate under
+`.agentfs/skills/`. Review its exact evidence with `show`, approve it explicitly,
+then use a private proposal adapter to generate a bounded edit list for one
+existing skill file. Proposal and evaluation do not modify the live target.
+
+Evaluation reuses the default-off evaluation coordinator and therefore requires
+`CAIRN_EVAL=1`, `--yes`, an explicit harness adapter, and two disjoint committed
+task sets. Only a complete, no-regression improvement in exploration opens the
+confirmation set. Only the same result on confirmation makes the proposal
+eligible for exact-digest application. See
+[Validated skill improvement](skill-improvement.md) for the command sequence,
+adapter contract, task constraints, and rollback procedure.
+
 ### Structured session trajectories (opt-in)
 
 Trajectory capture is disabled by default. To enable it for a launched Claude

package/docs/privacy-and-data-flow.md

@@ -16,6 +16,7 @@ request unless the corresponding endpoint and credential are configured.
 | `domain_knowledge_sync` | Files selected by the sync configuration | `ANYTHINGLLM_BASE_URL` |
 | `route_check` | A health request, with no memory or prompt content | `CAIRN_ROUTE_ENDPOINT` |
 | `context_explore` | Repository path and query are passed to the configured local executable | `CAIRN_EXPLORE_BINARY`; any further data flow is controlled by that tool |
+| Optional Graphify workflow | The repository path, exact query/symbol names, root `graphify-out/` work directory, and local `.planning/graphs/` publication are passed to or produced by the operator-installed `graphify` executable | Local Graphify subprocess with a minimal environment and no provider credentials; managed build uses code-only `update`, never semantic document extraction |
 | Remote HTTP memory | MCP requests and responses, including memory content | The explicitly registered Cairnkeep HTTP server |
 | Opt-in trajectory capture | None | Local `<project>/.agentfs/trajectory.db` only; no model or HTTP path exists |
 | Opt-in capability callback records | None | Payload-free final metadata in local `<project>/.agentfs/trajectory.db`; HTTP transport is always skipped |
@@ -174,6 +175,47 @@ the full committed population and note-eligible subset with paired counts and
 seeded uncertainty; they do not convert framework output into a causal,
 significance, quality, or efficiency claim.
 
+## Validated skill improvement flow
+
+Harvest and review are local-only operations over project hindsight notes.
+Before approval, no proposal subprocess is started and no evidence crosses a
+process boundary. After explicit approval, the configured proposal adapter
+receives exactly one bounded JSON request containing the approved candidate,
+candidate digest, selected target path, current target content and digest, and
+edit budget.
+
+The proposal adapter receives an isolated private HOME and TMPDIR, a minimal
+PATH and locale, and only environment variables named in its strict
+`environment_allowlist`. It does not inherit the rest of the parent
+environment. HOME, PATH, temporary and XDG roots, and Node runtime injection
+variables are isolation-controlled and cannot be allowlisted. Its stdout must
+be one strict bounded response containing status
+and bounded edits; arbitrary stderr is live operator diagnostics and is not
+stored. Cairnkeep records the adapter executable digest and rejects a change
+during the invocation. The adapter may perform network access according to the
+executable and host policy, but Cairnkeep provides no endpoint, credential, or
+network default. Operators must treat approval as consent to send those exact
+evidence and target fields to that adapter.
+
+The reduced environment is not an operating-system sandbox. The explicit
+adapter executable retains the invoking user's filesystem and network
+authority, so operators must trust it or wrap it in a separately enforced host
+sandbox.
+
+Evaluation sends neither candidate evidence nor proposal rationales to the
+harness adapter. It overlays baseline or candidate skill bytes into separate
+fresh worktrees and otherwise uses the existing evaluation request contract.
+Exploration and confirmation use disjoint task IDs and definitions. Confirmation
+is not opened unless exploration passes, and its results cannot be supplied
+back to proposal generation by this workflow. Both stages bind one evaluation
+adapter executable digest and reject a change during either stage.
+
+Apply and rollback invoke no model and make no network request. They operate on
+one contained regular target and private local artifacts, require exact
+digests, reject symlinks, and fail closed on concurrent target changes. See
+[storage](storage.md#validated-skill-storage) for retained bytes and
+[Validated skill improvement](skill-improvement.md) for the operator sequence.
+
 ## Compaction and artifact flows
 
 Both `CAIRN_COMPACTION_CAPTURE` and `CAIRN_ARTIFACT_STORE` are default-off and

package/docs/skill-improvement.md

@@ -0,0 +1,124 @@
+# Validated skill improvement
+
+`cairn skill` turns recurring project hindsight into a proposed change to one
+existing skill file, then requires independent evidence before that change can
+be applied. It is an operator-controlled lifecycle, not an autonomous rewrite
+loop.
+
+## Safety contract
+
+- Harvest reads only project hindsight notes and requires failures from at
+  least two distinct sessions by default.
+- Evidence remains local until `review --approve` records explicit approval.
+- Proposal generation invokes only the configured executable, sends one strict
+  bounded JSON request, exposes only allowlisted environment variables, and
+  accepts a bounded edit list rather than an arbitrary rewritten tree.
+- Proposal and evaluation artifacts bind the adapter executable digest;
+  evaluation rechecks it after exploration and confirmation.
+- Proposal and evaluation use candidate content without modifying the live
+  target.
+- Evaluation requires repository-root tasks in two different committed task
+  sets with at least two tasks each. Task IDs and exact task definitions cannot
+  overlap, and both sets must bind the same immutable source revision.
+- Confirmation runs only after exploration passes. Confirmation results never
+  return to proposal generation.
+- Apply requires an eligible exact evaluation, an unchanged target, and the
+  full proposal digest. It creates a private backup before atomic replacement.
+- Rollback refuses to overwrite a target changed after application.
+
+## Prepare a target and proposal adapter
+
+The target is one existing project-relative regular file named `SKILL.md`, for example
+`.claude/skills/generated-client-review/SKILL.md`. It cannot be a symlink and
+is limited to 256 KiB.
+
+The proposal adapter configuration is strict JSON:
+
+```json
+{
+  "schema_version": 1,
+  "id": "local-skill-proposer",
+  "command": {
+    "program": "/absolute/path/to/proposal-adapter",
+    "args": []
+  },
+  "environment_allowlist": ["PROPOSAL_PROVIDER_TOKEN"],
+  "limits": {
+    "elapsed_ms": 120000,
+    "stdout_bytes": 1048576
+  }
+}
+```
+
+The executable receives the approved candidate, its digest, current target
+content and digest, and edit budget on stdin. It returns `add`, `replace`, or
+`delete` edits with exact anchors and rationales. The executable path must be
+absolute, executable, and not a symlink. Cairnkeep does not provide a model,
+endpoint, credential, or network default. The machine-readable contracts are
+[`schemas/skill-adapter.schema.json`](../schemas/skill-adapter.schema.json) and
+[`schemas/skill-proposal-protocol.schema.json`](../schemas/skill-proposal-protocol.schema.json).
+Process-isolation variables such as `HOME`, `PATH`, temporary directories,
+XDG roots, and Node runtime injection settings cannot be allowlisted.
+This environment boundary is not an operating-system sandbox: the configured
+program runs with the invoking user's filesystem and network authority. Use
+only a trusted adapter or add a separate host sandbox.
+
+## Run the lifecycle
+
+```bash
+# 1. Find recurring hindsight evidence. This performs no model call.
+cairn skill harvest --project . --json
+cairn skill show --kind candidate --id CANDIDATE_ID
+
+# 2. Approve the exact evidence that may cross the adapter boundary.
+cairn skill review --candidate CANDIDATE_ID --approve --json
+
+# 3. Generate a bounded proposal. The live target remains unchanged.
+cairn skill propose --candidate CANDIDATE_ID \
+  --target .claude/skills/generated-client-review/SKILL.md \
+  --adapter ./private/proposal-adapter.json --edit-budget 4 --json
+
+# 4. Measure baseline versus candidate in fresh isolated worktrees.
+export CAIRN_EVAL=1
+cairn skill evaluate --proposal PROPOSAL_ID \
+  --exploration-task-set eval/skill-exploration.json \
+  --confirmation-task-set eval/skill-confirmation.json \
+  --adapter eval/harness-adapter.json \
+  --repetitions 2 --minimum-improvement 1 --yes --json
+
+# 5. Inspect artifacts and apply only the exact eligible proposal.
+cairn skill show --kind proposal --id PROPOSAL_ID
+cairn skill show --kind evaluation --id EVALUATION_ID
+cairn skill apply --proposal PROPOSAL_ID --evaluation EVALUATION_ID \
+  --confirm FULL_PROPOSAL_DIGEST --json
+
+# 6. Revert if later evidence warrants it.
+cairn skill rollback --application APPLICATION_ID --confirm --json
+```
+
+`propose --json` includes `proposal_digest`, which is the value required by
+`apply --confirm`. `list` shows every artifact class; use `--kind candidate`,
+`proposal`, `evaluation`, or `application` to narrow it.
+
+## Evaluation semantics
+
+Each task runs twice in each arm. The baseline arm overlays the exact unchanged
+target content; the treatment arm overlays the exact candidate content. Every
+observation must complete, return the expected capability digest, and receive a
+known result from the independent verifier. A task with any candidate-only pass
+counts once as improved; a task with any baseline-only pass counts once as
+regressed and rejects the candidate.
+
+Exploration must have no unknown pairs, no regression, and at least
+`--minimum-improvement` improvements. Otherwise evaluation stops without
+opening the confirmation set. Confirmation must independently satisfy the same
+gate. This is deliberately strict and suited to small, high-value task sets;
+it is not a statistical proof that a skill is universally better.
+
+## What this does not do
+
+It does not scrape native harness transcripts, schedule periodic self-edits,
+invent task sets, approve evidence, choose an inference provider, or publish a
+changed skill. These remain explicit operator responsibilities. Start with a
+small target and representative tasks; expand only when the stored evaluation
+artifacts make the decision auditable.

package/docs/storage.md

@@ -213,6 +213,31 @@ the process stores databases below `/data`, normally backed by the
 removing the volume removes the databases. See [Containers](containers.md) for
 the exact paths and backup boundary.
 
+## Validated skill storage
+
+The skill-improvement lifecycle stores private JSON artifacts below
+`<project>/.agentfs/skills/`: `candidates/`, `proposals/`, `evaluations/`, and
+`applications/`. Pre-application target backups live in `backups/`; isolated
+proposal-adapter homes are created temporarily in `adapter-tmp/` and removed
+after every invocation. Directories are mode `0700`, JSON and backup files are
+mode `0600`, artifacts are bounded to 2 MiB, and candidate target content is
+bounded to 256 KiB.
+
+Candidate and proposal artifacts can contain hindsight excerpts and full skill
+content, so the entire directory is sensitive. Evaluation reports and
+experiment-owned snapshots remain in `.agentfs/eval/experiments/` under the
+existing evaluation retention rules. No skill artifact is written to global or
+remote memory, and remote MCP configuration does not redirect this project
+state.
+
+`apply` writes the original target to a unique private backup before atomically
+replacing the target. If replacement, digest verification, or ledger storage
+fails, Cairnkeep restores the original and removes the incomplete backup.
+`rollback` restores only when both the current target and backup match the
+application ledger, preventing an unrelated later edit from being overwritten.
+Default uninstall retains `.agentfs/`; explicit purge uses the existing
+backup-first project boundary.
+
 ## Remote HTTP mode
 
 Remote storage is explicit. An operator starts Cairnkeep in HTTP mode on a

package/examples/eval/bundled-fake.json

@@ -1 +1 @@
-{"identifier":"cairn-offline-fake-v1","package_version":"2.6.0","schema_version":1,"task_set_digest":"13e95379f35839593047ec34521a65a8fa3ab92bd3993bca9dc8bf5c4a9ff36f"}
+{"identifier":"cairn-offline-fake-v1","package_version":"2.8.0","schema_version":1,"task_set_digest":"13e95379f35839593047ec34521a65a8fa3ab92bd3993bca9dc8bf5c4a9ff36f"}

package/kimi/skills/graphify/SKILL.md

@@ -0,0 +1,33 @@
+---
+name: graphify
+description: Build, query, and inspect the local project graph exclusively through Cairnkeep
+type: flow
+---
+
+<!-- managed-by:cairnkeep -->
+# Cairnkeep graph
+
+Handle this request through Cairnkeep's public graph owner. The invocation
+arguments are:
+
+```text
+$ARGUMENTS
+```
+
+Accept exactly one of these forms:
+
+- `build`
+- `build --force`
+- `query <term>`
+- `status`
+- `diff`
+- `explain <symbol>`
+- `path <from> <to>`
+
+Invoke the Bash tool exactly once with the corresponding `cairn graph ...`
+command. Pass each term or symbol as one safely quoted argument. Return the
+command output verbatim and stop.
+
+If the arguments do not match a supported form, show the supported forms and
+stop without invoking a tool. Do not call the Graphify executable, edit graph
+artifacts, spawn another agent, or implement graph logic in this Skill.

package/mcp-memory-server/dist/capability-registry.js

@@ -6,7 +6,7 @@ const MAX_COMPATIBILITY_CONFIG_BYTES = 64 * 1024;
 function environmentName(id) {
     return `CAIRN_CAPABILITY_${id.toUpperCase().replaceAll(".", "_")}`;
 }
-async function readGraphCompatibility(projectRoot) {
+export async function isGraphCapabilityCompatible(projectRoot) {
     const planningDirectory = join(projectRoot, ".planning");
     const configPath = join(planningDirectory, "config.json");
     try {
@@ -84,7 +84,7 @@ export const CAPABILITY_REGISTRY = [
         compatibility_default: false,
         restart_required: false,
         logging_policy: "final-callback",
-        resolveCompatibility: ({ projectRoot, graphifyEnabled }) => graphifyEnabled ?? readGraphCompatibility(projectRoot),
+        resolveCompatibility: ({ projectRoot, graphifyEnabled }) => graphifyEnabled ?? isGraphCapabilityCompatible(projectRoot),
     },
     {
         id: "security.audit",

package/mcp-memory-server/dist/eval-plan.js

@@ -186,7 +186,7 @@ function validateBundledBinding(taskSetPath, taskSet, taskSetBytes) {
         binding_path: realpathSync(bindingPath),
     };
 }
-function validateArms(arms) {
+function validateArms(arms, experimentKind) {
     if (arms.length < 1 || arms.length > 2)
         throw new Error("Evaluation requires one or two arms.");
     const seen = new Set();
@@ -201,8 +201,12 @@ function validateArms(arms) {
         }
         if (arm.id === "baseline" && arm.disabled_capability !== null)
             throw new Error("Baseline cannot disable a capability.");
-        if (arm.id === "treatment" && arm.disabled_capability === null)
-            throw new Error("Treatment must disable exactly one capability.");
+        if (arm.id === "treatment" && experimentKind === "ablation" && arm.disabled_capability === null) {
+            throw new Error("Ablation treatment must disable exactly one capability.");
+        }
+        if (experimentKind === "skill_candidate" && arm.disabled_capability !== null) {
+            throw new Error("Skill-candidate arms cannot disable capabilities.");
+        }
         return { id: arm.id, disabled_capability: arm.disabled_capability };
     });
 }
@@ -214,7 +218,10 @@ export function isEvalEnabled(value = process.env.CAIRN_EVAL) {
 }
 export function buildEvalSchedule(options) {
     const taskSet = evalTaskSetSchema.parse(options.taskSet);
-    const arms = validateArms(options.arms);
+    const inferredKind = options.arms.length === 1 ? "two_pass" : options.arms.some(({ disabled_capability }) => disabled_capability !== null)
+        ? "ablation"
+        : "skill_candidate";
+    const arms = validateArms(options.arms, inferredKind);
     if (!Number.isSafeInteger(options.repetitions) || options.repetitions < 1 || options.repetitions > MAX_REPETITIONS) {
         throw new Error(`Evaluation repetitions must be an integer from 1 to ${MAX_REPETITIONS}.`);
     }
@@ -282,7 +289,15 @@ export function validateEvalInputs(options) {
     }
     const repetitions = options.repetitions ?? 1;
     const seed = options.seed ?? "cairn-eval-v1";
-    const arms = validateArms(options.arms ?? [{ id: "baseline", disabled_capability: null }]);
+    const suppliedArms = options.arms ?? [{ id: "baseline", disabled_capability: null }];
+    const experimentKind = options.experimentKind
+        ?? (suppliedArms.length === 1 ? "two_pass" : "ablation");
+    if (experimentKind === "two_pass" && suppliedArms.length !== 1)
+        throw new Error("Two-pass evaluation requires one arm.");
+    if ((experimentKind === "ablation" || experimentKind === "skill_candidate") && suppliedArms.length !== 2) {
+        throw new Error(`${experimentKind} evaluation requires two arms.`);
+    }
+    const arms = validateArms(suppliedArms, experimentKind);
     const passes = ["run1", "run2"];
     const schedule = buildEvalSchedule({ taskSet, arms, repetitions, passes, seed });
     const adapterBase = dirname(adapterInput.path);
@@ -304,6 +319,7 @@ export function validateEvalInputs(options) {
         repetitions,
         seed,
         arms,
+        experiment_kind: experimentKind,
         passes,
         concurrency: 1,
         task_set_commit: taskSetCommit,
@@ -320,6 +336,7 @@ export function validateEvalInputs(options) {
         repetitions,
         seed,
         arms,
+        experiment_kind: experimentKind,
         passes: ["run1", "run2"],
         concurrency: 1,
         invocation_count: schedule.invocation_count,

package/mcp-memory-server/dist/eval-process.js

@@ -246,6 +246,37 @@ function hasTrailingJsonDocument(value) {
     }
     return false;
 }
+export async function runBoundedJsonProcess(options) {
+    const request = options.request_schema.safeParse(options.request);
+    if (!request.success)
+        throw new EvalProcessError("invalid_request");
+    const commandResult = await runCommandInternal({
+        command: options.command,
+        cwd: options.cwd,
+        env: options.env,
+        timeout_ms: options.timeout_ms,
+        max_stdout_bytes: options.max_stdout_bytes,
+        kill_grace_ms: options.kill_grace_ms,
+        signal: options.signal,
+        stdout_mode: "raw",
+        stdin_document: Buffer.from(JSON.stringify(request.data), "utf8"),
+    });
+    if (commandResult.exit_code !== 0 || commandResult.signal !== null || commandResult.cleanup !== "closed") {
+        throw new EvalProcessError("adapter_error", commandResult);
+    }
+    let parsed;
+    try {
+        parsed = JSON.parse(commandResult.stdout ?? "");
+    }
+    catch {
+        throw new EvalProcessError(hasTrailingJsonDocument(commandResult.stdout ?? "") ? "multiple_json" : "invalid_json", commandResult);
+    }
+    const result = options.result_schema.safeParse(parsed);
+    if (!result.success)
+        throw new EvalProcessError("invalid_result", commandResult);
+    const { stdout: _stdout, ...processResult } = commandResult;
+    return { ...processResult, result: result.data };
+}
 export async function runBoundedJsonAdapter(options) {
     const request = evalAdapterRequestSchema.safeParse(options.request);
     if (!request.success)

package/mcp-memory-server/dist/eval-runner.js

@@ -9,7 +9,7 @@ import { CAPABILITY_IDS, capabilityStatusSchema, } from "./capability-schema.js"
 import { canonicalDigest, evalReportSchema, evalReportRuntimeContractDigest, } from "./eval-schema.js";
 import { EvalProcessError, runBoundedCommand, runBoundedJsonAdapter, } from "./eval-process.js";
 import { checkpointEvalReport, createEvalReportStore, } from "./eval-report.js";
-import { cleanupEvalWorkspace, createEvalWorkspace, runTaskPreparation, runTaskVerifier, } from "./eval-workspace.js";
+import { applyEvalWorkspaceOverlay, cleanupEvalWorkspace, createEvalWorkspace, runTaskPreparation, runTaskVerifier, } from "./eval-workspace.js";
 const moduleDirectory = dirname(fileURLToPath(import.meta.url));
 const publishedReportSchemaPath = resolve(moduleDirectory, "..", "..", "schemas", "eval-report.schema.json");
 let reportSchemaDigestsCache;
@@ -557,6 +557,8 @@ export async function runEvalObservation(options) {
             observation.missing_reasons.push("preparation_failed");
         }
         else {
+            if (options.workspace_overlay)
+                await applyEvalWorkspaceOverlay(workspace, options.workspace_overlay);
             const request = {
                 schema_version: 1,
                 experiment_id: options.report.experiment_id,
@@ -698,7 +700,7 @@ export async function runEvalObservation(options) {
     await checkpoint(options.report_store, options.report);
     return { observation, snapshot: distilledSnapshot, cancelled: observation.terminal_state === "cancelled" };
 }
-function initialReport(plan, experimentId) {
+function initialReport(plan, experimentId, fallbackExperimentKind) {
     const now = new Date().toISOString();
     const sourceCommit = plan.task_set_commit ?? (plan.source.kind === "git" ? plan.source.revision : plan.task_set_digest);
     const missingness = { count: 0, reasons: [] };
@@ -706,7 +708,7 @@ function initialReport(plan, experimentId) {
         schema_version: 1,
         experiment_id: experimentId,
         status: "partial",
-        experiment_kind: plan.arms.length === 1 ? "two_pass" : "ablation",
+        experiment_kind: plan.experiment_kind ?? fallbackExperimentKind,
         task_set_digest: plan.task_set_digest,
         adapter_config_digest: plan.adapter_config_digest,
         source_revision: plan.source.kind === "git" ? plan.source.revision : plan.task_set_digest,
@@ -750,7 +752,7 @@ async function runExperiment(options, capabilityArms) {
     });
     if (reportStore.experiment_id !== experimentId)
         throw new Error("report_experiment_mismatch");
-    const report = initialReport(options.plan, experimentId);
+    const report = initialReport(options.plan, experimentId, capabilityArms ? "ablation" : "two_pass");
     const snapshotsByTask = new Map();
     const notesByTask = new Map();
     const snapshots = [];
@@ -782,6 +784,7 @@ async function runExperiment(options, capabilityArms) {
                 distill_command: options.distill_command,
                 distill_timeout_ms: options.distill_timeout_ms,
                 capability_runtime: capabilityRuntime,
+                workspace_overlay: options.workspace_overlays?.[row.arm],
             });
             executed.push(row.observation_id);
             if (result.snapshot) {
@@ -813,6 +816,9 @@ export async function runTwoPassExperiment(options) {
     return runExperiment(options);
 }
 export async function runCapabilityAblation(options) {
+    if (options.plan.experiment_kind !== undefined && options.plan.experiment_kind !== "ablation") {
+        throw new Error("ablation_experiment_kind_required");
+    }
     const arms = buildAblationArms(options.disabled_capability);
     const intended = arms.map(({ id, disabled_capability }) => ({ id, disabled_capability }));
     if (canonicalDigest(options.plan.arms) !== canonicalDigest(intended))

package/mcp-memory-server/dist/eval-schema.js

@@ -13,7 +13,7 @@ export const EVAL_OBSERVATION_TERMINAL_STATES = [
     "invalid_result",
 ];
 export const EVAL_PASS_STATES = ["passed", "failed", "unknown"];
-export const EVAL_EXPERIMENT_KINDS = ["two_pass", "ablation"];
+export const EVAL_EXPERIMENT_KINDS = ["two_pass", "ablation", "skill_candidate"];
 const MAX_TEXT_BYTES = 1024 * 1024;
 const MAX_COMMAND_ARGS = 256;
 const MAX_TASKS = 10_000;