@cairnkeep/cli 2.7.0 → 2.8.0

package/docs/learning/lessons/L19-validated-skill-improvement.md

@@ -0,0 +1,87 @@
+# L19 - Validated skill improvement
+
+**Status:** Brief
+**Track:** Evidence and Evaluation
+**Planned time:** 50 minutes
+**Introduced in:** Cairnkeep 2.8.0
+
+## Outcome
+
+Turn repeated failure and demonstrated resolution evidence into one reviewed
+skill proposal, measure it on separate exploration and confirmation tasks,
+apply only the exact eligible digest, and prove rollback without risking a real
+project.
+
+## Planned lesson
+
+- Why harvest requires failures from at least two sessions and at least one
+  recorded resolution.
+- How `harvest`, `show`, and `review --approve` keep evidence local until the
+  learner consents to proposal-adapter egress.
+- How the strict proposal adapter receives a bounded request, a private process
+  environment, and an edit budget for one existing `SKILL.md`.
+- Why proposal generation and evaluation never modify the live target.
+- How separate committed sets with at least two disjoint task IDs and
+  definitions each, one immutable source revision, fresh Git worktrees, exact
+  adapter digests, and an independent verifier protect the gate.
+- Why improvements count distinct tasks, any regressed task rejects the
+  candidate, and small passing fixtures are not universal quality proof.
+- How full proposal-digest confirmation, backup-first atomic replacement, and
+  concurrent-edit-safe rollback limit application authority.
+
+## Hands-on lab
+
+Use a future disposable `course-09-skill` checkpoint containing synthetic
+hindsight evidence, a small `SKILL.md`, deterministic proposal and evaluation
+adapters, and separate committed task sets. The lab will run:
+
+```bash
+cairn skill harvest --project . --json
+cairn skill show --kind candidate --id CANDIDATE_ID
+cairn skill review --candidate CANDIDATE_ID --approve --json
+cairn skill propose --candidate CANDIDATE_ID --target skills/demo/SKILL.md \
+  --adapter fixtures/proposal-adapter.json --json
+
+export CAIRN_EVAL=1
+cairn skill evaluate --proposal PROPOSAL_ID \
+  --exploration-task-set eval/exploration.json \
+  --confirmation-task-set eval/confirmation.json \
+  --adapter eval/adapter.json --repetitions 2 --yes --json
+
+cairn skill apply --proposal PROPOSAL_ID --evaluation EVALUATION_ID \
+  --confirm FULL_PROPOSAL_DIGEST --json
+cairn skill rollback --application APPLICATION_ID --confirm --json
+```
+
+The lesson remains a Brief until that checkpoint is public and every command is
+rehearsed against the release package.
+
+## Acceptance criteria
+
+- No candidate exists for repeated failure without resolution evidence.
+- Proposal execution cannot inherit unapproved credentials or isolation-owned
+  environment variables.
+- The live `SKILL.md` is byte-identical before and after propose/evaluate.
+- Overlapping task IDs, changed adapters, unknown verifier results, and any
+  regression fail closed.
+- Apply rejects a wrong digest or changed target.
+- Rollback restores the original bytes and refuses to overwrite a later edit.
+- The learner can explain why an eligible small task set supports one local
+  decision but not a universal performance claim.
+
+## Privacy and trust boundary
+
+Hindsight excerpts and skill content remain local until candidate approval.
+Approval permits those exact fields to enter the explicitly configured proposal
+adapter. Evaluation adapters receive isolated worktrees with baseline or
+candidate skill bytes, not the hindsight evidence. Candidate, proposal,
+evaluation, application, backup, and report artifacts remain sensitive local
+project state under `.agentfs/`.
+
+## Source material
+
+- [Validated skill improvement](../../skill-improvement.md)
+- [Evaluation and ablation](L16-evaluation.md)
+- [The agent boundary](L17-agent-boundary.md)
+- [Storage](../../storage.md#validated-skill-storage)
+- [Privacy and data flow](../../privacy-and-data-flow.md#validated-skill-improvement-flow)

package/docs/learning/tracks/evidence-and-evaluation.md

@@ -2,7 +2,7 @@
 
 **Audience:** maintainers and advanced practitioners evaluating optional layers
 **Prerequisite:** Practitioner track
-**Total time:** about 3.25 hours
+**Total time:** about 4 hours
 **Infrastructure:** local fixtures first; a live harness is operator-provided
 
 1. [L13 - Session evidence and hindsight](../lessons/L13-session-evidence.md)
@@ -10,11 +10,13 @@
 3. [L15 - Capability governance](../lessons/L15-capability-governance.md)
 4. [L16 - Evaluation and ablation](../lessons/L16-evaluation.md)
 5. [L17 - The agent boundary](../lessons/L17-agent-boundary.md)
+6. [L19 - Validated skill improvement](../lessons/L19-validated-skill-improvement.md)
 
 The track is complete when the learner can explain each consent boundary,
 inspect and manage every local evidence store, compare baseline and ablation
-reports without making unsupported claims, and distinguish current runtime
-features from the design-only meta-agent contract.
+reports without making unsupported claims, apply or roll back an exact eligible
+skill proposal, and distinguish that narrow runtime from the design-only
+general meta-agent contract.
 
 Hands-on spine: `course-05-evidence` through `course-07-evaluation` in the
 public [course labs](https://github.com/cairnkeep/cairnkeep-course-labs).

package/docs/learning/tracks/practitioner.md

@@ -6,10 +6,11 @@
 
 1. [L04 - Memory fundamentals](../lessons/L04-memory-fundamentals.md)
 2. [L05 - The daily workflow](../lessons/L05-daily-workflow.md)
-3. [L06 - Repository review and security](../lessons/L06-repository-quality.md)
-4. [L07 - Privacy and storage](../lessons/L07-privacy-and-storage.md)
-5. [L08 - Multiple machines](../lessons/L08-multiple-machines.md)
-6. [L13 - Session evidence and hindsight](../lessons/L13-session-evidence.md)
+3. [L18 - Local code graph](../lessons/L18-local-code-graph.md)
+4. [L06 - Repository review and security](../lessons/L06-repository-quality.md)
+5. [L07 - Privacy and storage](../lessons/L07-privacy-and-storage.md)
+6. [L08 - Multiple machines](../lessons/L08-multiple-machines.md)
+7. [L13 - Session evidence and hindsight](../lessons/L13-session-evidence.md)
 
 The track is complete when the learner can decide what should become memory,
 review derived knowledge, run governed repository checks, locate the store, and

package/docs/operating.md

@@ -218,6 +218,13 @@ do not perform Graphify's optional semantic document extraction. Prefer exact
 function, class, or file names; broad natural-language graph queries are less
 precise. Cairnkeep does not install or update Graphify automatically.
 
+Graphify also maintains an incremental `graphify-out/` work directory at the
+repository root. It is separate from Cairnkeep's published
+`.planning/graphs/` view and can contain the same sensitive structural data.
+Keep both locations local. Before the first build in a Git repository, add
+`/graphify-out/` to the clone-local exclude file or the repository's ignore
+policy, and verify with `git status --short` after every build.
+
 ## Setup order (Kimi Code)
 
 Kimi Code can use the Cairnkeep memory MCP, generated project launcher, and one
@@ -897,6 +904,22 @@ See [storage](storage.md#evaluation-report-and-note-snapshot-storage) for local
 retention/removal and [privacy](privacy-and-data-flow.md#evaluation-adapter-and-report-flow)
 for the exact request, observation, environment, and persistence boundaries.
 
+### Validated skill improvement (opt-in)
+
+After hindsight-note distillation has recorded the same failure family in at
+least two sessions, `cairn skill harvest` creates a local candidate under
+`.agentfs/skills/`. Review its exact evidence with `show`, approve it explicitly,
+then use a private proposal adapter to generate a bounded edit list for one
+existing skill file. Proposal and evaluation do not modify the live target.
+
+Evaluation reuses the default-off evaluation coordinator and therefore requires
+`CAIRN_EVAL=1`, `--yes`, an explicit harness adapter, and two disjoint committed
+task sets. Only a complete, no-regression improvement in exploration opens the
+confirmation set. Only the same result on confirmation makes the proposal
+eligible for exact-digest application. See
+[Validated skill improvement](skill-improvement.md) for the command sequence,
+adapter contract, task constraints, and rollback procedure.
+
 ### Structured session trajectories (opt-in)
 
 Trajectory capture is disabled by default. To enable it for a launched Claude

package/docs/privacy-and-data-flow.md

@@ -16,7 +16,7 @@ request unless the corresponding endpoint and credential are configured.
 | `domain_knowledge_sync` | Files selected by the sync configuration | `ANYTHINGLLM_BASE_URL` |
 | `route_check` | A health request, with no memory or prompt content | `CAIRN_ROUTE_ENDPOINT` |
 | `context_explore` | Repository path and query are passed to the configured local executable | `CAIRN_EXPLORE_BINARY`; any further data flow is controlled by that tool |
-| Optional Graphify workflow | The repository path, exact query/symbol names, and local published graph path are passed to the operator-installed `graphify` executable | Local Graphify subprocess with a minimal environment and no provider credentials; managed build uses code-only `update`, never semantic document extraction |
+| Optional Graphify workflow | The repository path, exact query/symbol names, root `graphify-out/` work directory, and local `.planning/graphs/` publication are passed to or produced by the operator-installed `graphify` executable | Local Graphify subprocess with a minimal environment and no provider credentials; managed build uses code-only `update`, never semantic document extraction |
 | Remote HTTP memory | MCP requests and responses, including memory content | The explicitly registered Cairnkeep HTTP server |
 | Opt-in trajectory capture | None | Local `<project>/.agentfs/trajectory.db` only; no model or HTTP path exists |
 | Opt-in capability callback records | None | Payload-free final metadata in local `<project>/.agentfs/trajectory.db`; HTTP transport is always skipped |
@@ -175,6 +175,47 @@ the full committed population and note-eligible subset with paired counts and
 seeded uncertainty; they do not convert framework output into a causal,
 significance, quality, or efficiency claim.
 
+## Validated skill improvement flow
+
+Harvest and review are local-only operations over project hindsight notes.
+Before approval, no proposal subprocess is started and no evidence crosses a
+process boundary. After explicit approval, the configured proposal adapter
+receives exactly one bounded JSON request containing the approved candidate,
+candidate digest, selected target path, current target content and digest, and
+edit budget.
+
+The proposal adapter receives an isolated private HOME and TMPDIR, a minimal
+PATH and locale, and only environment variables named in its strict
+`environment_allowlist`. It does not inherit the rest of the parent
+environment. HOME, PATH, temporary and XDG roots, and Node runtime injection
+variables are isolation-controlled and cannot be allowlisted. Its stdout must
+be one strict bounded response containing status
+and bounded edits; arbitrary stderr is live operator diagnostics and is not
+stored. Cairnkeep records the adapter executable digest and rejects a change
+during the invocation. The adapter may perform network access according to the
+executable and host policy, but Cairnkeep provides no endpoint, credential, or
+network default. Operators must treat approval as consent to send those exact
+evidence and target fields to that adapter.
+
+The reduced environment is not an operating-system sandbox. The explicit
+adapter executable retains the invoking user's filesystem and network
+authority, so operators must trust it or wrap it in a separately enforced host
+sandbox.
+
+Evaluation sends neither candidate evidence nor proposal rationales to the
+harness adapter. It overlays baseline or candidate skill bytes into separate
+fresh worktrees and otherwise uses the existing evaluation request contract.
+Exploration and confirmation use disjoint task IDs and definitions. Confirmation
+is not opened unless exploration passes, and its results cannot be supplied
+back to proposal generation by this workflow. Both stages bind one evaluation
+adapter executable digest and reject a change during either stage.
+
+Apply and rollback invoke no model and make no network request. They operate on
+one contained regular target and private local artifacts, require exact
+digests, reject symlinks, and fail closed on concurrent target changes. See
+[storage](storage.md#validated-skill-storage) for retained bytes and
+[Validated skill improvement](skill-improvement.md) for the operator sequence.
+
 ## Compaction and artifact flows
 
 Both `CAIRN_COMPACTION_CAPTURE` and `CAIRN_ARTIFACT_STORE` are default-off and

package/docs/skill-improvement.md

@@ -0,0 +1,124 @@
+# Validated skill improvement
+
+`cairn skill` turns recurring project hindsight into a proposed change to one
+existing skill file, then requires independent evidence before that change can
+be applied. It is an operator-controlled lifecycle, not an autonomous rewrite
+loop.
+
+## Safety contract
+
+- Harvest reads only project hindsight notes and requires failures from at
+  least two distinct sessions by default.
+- Evidence remains local until `review --approve` records explicit approval.
+- Proposal generation invokes only the configured executable, sends one strict
+  bounded JSON request, exposes only allowlisted environment variables, and
+  accepts a bounded edit list rather than an arbitrary rewritten tree.
+- Proposal and evaluation artifacts bind the adapter executable digest;
+  evaluation rechecks it after exploration and confirmation.
+- Proposal and evaluation use candidate content without modifying the live
+  target.
+- Evaluation requires repository-root tasks in two different committed task
+  sets with at least two tasks each. Task IDs and exact task definitions cannot
+  overlap, and both sets must bind the same immutable source revision.
+- Confirmation runs only after exploration passes. Confirmation results never
+  return to proposal generation.
+- Apply requires an eligible exact evaluation, an unchanged target, and the
+  full proposal digest. It creates a private backup before atomic replacement.
+- Rollback refuses to overwrite a target changed after application.
+
+## Prepare a target and proposal adapter
+
+The target is one existing project-relative regular file named `SKILL.md`, for example
+`.claude/skills/generated-client-review/SKILL.md`. It cannot be a symlink and
+is limited to 256 KiB.
+
+The proposal adapter configuration is strict JSON:
+
+```json
+{
+  "schema_version": 1,
+  "id": "local-skill-proposer",
+  "command": {
+    "program": "/absolute/path/to/proposal-adapter",
+    "args": []
+  },
+  "environment_allowlist": ["PROPOSAL_PROVIDER_TOKEN"],
+  "limits": {
+    "elapsed_ms": 120000,
+    "stdout_bytes": 1048576
+  }
+}
+```
+
+The executable receives the approved candidate, its digest, current target
+content and digest, and edit budget on stdin. It returns `add`, `replace`, or
+`delete` edits with exact anchors and rationales. The executable path must be
+absolute, executable, and not a symlink. Cairnkeep does not provide a model,
+endpoint, credential, or network default. The machine-readable contracts are
+[`schemas/skill-adapter.schema.json`](../schemas/skill-adapter.schema.json) and
+[`schemas/skill-proposal-protocol.schema.json`](../schemas/skill-proposal-protocol.schema.json).
+Process-isolation variables such as `HOME`, `PATH`, temporary directories,
+XDG roots, and Node runtime injection settings cannot be allowlisted.
+This environment boundary is not an operating-system sandbox: the configured
+program runs with the invoking user's filesystem and network authority. Use
+only a trusted adapter or add a separate host sandbox.
+
+## Run the lifecycle
+
+```bash
+# 1. Find recurring hindsight evidence. This performs no model call.
+cairn skill harvest --project . --json
+cairn skill show --kind candidate --id CANDIDATE_ID
+
+# 2. Approve the exact evidence that may cross the adapter boundary.
+cairn skill review --candidate CANDIDATE_ID --approve --json
+
+# 3. Generate a bounded proposal. The live target remains unchanged.
+cairn skill propose --candidate CANDIDATE_ID \
+  --target .claude/skills/generated-client-review/SKILL.md \
+  --adapter ./private/proposal-adapter.json --edit-budget 4 --json
+
+# 4. Measure baseline versus candidate in fresh isolated worktrees.
+export CAIRN_EVAL=1
+cairn skill evaluate --proposal PROPOSAL_ID \
+  --exploration-task-set eval/skill-exploration.json \
+  --confirmation-task-set eval/skill-confirmation.json \
+  --adapter eval/harness-adapter.json \
+  --repetitions 2 --minimum-improvement 1 --yes --json
+
+# 5. Inspect artifacts and apply only the exact eligible proposal.
+cairn skill show --kind proposal --id PROPOSAL_ID
+cairn skill show --kind evaluation --id EVALUATION_ID
+cairn skill apply --proposal PROPOSAL_ID --evaluation EVALUATION_ID \
+  --confirm FULL_PROPOSAL_DIGEST --json
+
+# 6. Revert if later evidence warrants it.
+cairn skill rollback --application APPLICATION_ID --confirm --json
+```
+
+`propose --json` includes `proposal_digest`, which is the value required by
+`apply --confirm`. `list` shows every artifact class; use `--kind candidate`,
+`proposal`, `evaluation`, or `application` to narrow it.
+
+## Evaluation semantics
+
+Each task runs twice in each arm. The baseline arm overlays the exact unchanged
+target content; the treatment arm overlays the exact candidate content. Every
+observation must complete, return the expected capability digest, and receive a
+known result from the independent verifier. A task with any candidate-only pass
+counts once as improved; a task with any baseline-only pass counts once as
+regressed and rejects the candidate.
+
+Exploration must have no unknown pairs, no regression, and at least
+`--minimum-improvement` improvements. Otherwise evaluation stops without
+opening the confirmation set. Confirmation must independently satisfy the same
+gate. This is deliberately strict and suited to small, high-value task sets;
+it is not a statistical proof that a skill is universally better.
+
+## What this does not do
+
+It does not scrape native harness transcripts, schedule periodic self-edits,
+invent task sets, approve evidence, choose an inference provider, or publish a
+changed skill. These remain explicit operator responsibilities. Start with a
+small target and representative tasks; expand only when the stored evaluation
+artifacts make the decision auditable.

package/docs/storage.md

@@ -213,6 +213,31 @@ the process stores databases below `/data`, normally backed by the
 removing the volume removes the databases. See [Containers](containers.md) for
 the exact paths and backup boundary.
 
+## Validated skill storage
+
+The skill-improvement lifecycle stores private JSON artifacts below
+`<project>/.agentfs/skills/`: `candidates/`, `proposals/`, `evaluations/`, and
+`applications/`. Pre-application target backups live in `backups/`; isolated
+proposal-adapter homes are created temporarily in `adapter-tmp/` and removed
+after every invocation. Directories are mode `0700`, JSON and backup files are
+mode `0600`, artifacts are bounded to 2 MiB, and candidate target content is
+bounded to 256 KiB.
+
+Candidate and proposal artifacts can contain hindsight excerpts and full skill
+content, so the entire directory is sensitive. Evaluation reports and
+experiment-owned snapshots remain in `.agentfs/eval/experiments/` under the
+existing evaluation retention rules. No skill artifact is written to global or
+remote memory, and remote MCP configuration does not redirect this project
+state.
+
+`apply` writes the original target to a unique private backup before atomically
+replacing the target. If replacement, digest verification, or ledger storage
+fails, Cairnkeep restores the original and removes the incomplete backup.
+`rollback` restores only when both the current target and backup match the
+application ledger, preventing an unrelated later edit from being overwritten.
+Default uninstall retains `.agentfs/`; explicit purge uses the existing
+backup-first project boundary.
+
 ## Remote HTTP mode
 
 Remote storage is explicit. An operator starts Cairnkeep in HTTP mode on a

package/examples/eval/bundled-fake.json

@@ -1 +1 @@
-{"identifier":"cairn-offline-fake-v1","package_version":"2.7.0","schema_version":1,"task_set_digest":"13e95379f35839593047ec34521a65a8fa3ab92bd3993bca9dc8bf5c4a9ff36f"}
+{"identifier":"cairn-offline-fake-v1","package_version":"2.8.0","schema_version":1,"task_set_digest":"13e95379f35839593047ec34521a65a8fa3ab92bd3993bca9dc8bf5c4a9ff36f"}

package/mcp-memory-server/dist/eval-plan.js

@@ -186,7 +186,7 @@ function validateBundledBinding(taskSetPath, taskSet, taskSetBytes) {
         binding_path: realpathSync(bindingPath),
     };
 }
-function validateArms(arms) {
+function validateArms(arms, experimentKind) {
     if (arms.length < 1 || arms.length > 2)
         throw new Error("Evaluation requires one or two arms.");
     const seen = new Set();
@@ -201,8 +201,12 @@ function validateArms(arms) {
         }
         if (arm.id === "baseline" && arm.disabled_capability !== null)
             throw new Error("Baseline cannot disable a capability.");
-        if (arm.id === "treatment" && arm.disabled_capability === null)
-            throw new Error("Treatment must disable exactly one capability.");
+        if (arm.id === "treatment" && experimentKind === "ablation" && arm.disabled_capability === null) {
+            throw new Error("Ablation treatment must disable exactly one capability.");
+        }
+        if (experimentKind === "skill_candidate" && arm.disabled_capability !== null) {
+            throw new Error("Skill-candidate arms cannot disable capabilities.");
+        }
         return { id: arm.id, disabled_capability: arm.disabled_capability };
     });
 }
@@ -214,7 +218,10 @@ export function isEvalEnabled(value = process.env.CAIRN_EVAL) {
 }
 export function buildEvalSchedule(options) {
     const taskSet = evalTaskSetSchema.parse(options.taskSet);
-    const arms = validateArms(options.arms);
+    const inferredKind = options.arms.length === 1 ? "two_pass" : options.arms.some(({ disabled_capability }) => disabled_capability !== null)
+        ? "ablation"
+        : "skill_candidate";
+    const arms = validateArms(options.arms, inferredKind);
     if (!Number.isSafeInteger(options.repetitions) || options.repetitions < 1 || options.repetitions > MAX_REPETITIONS) {
         throw new Error(`Evaluation repetitions must be an integer from 1 to ${MAX_REPETITIONS}.`);
     }
@@ -282,7 +289,15 @@ export function validateEvalInputs(options) {
     }
     const repetitions = options.repetitions ?? 1;
     const seed = options.seed ?? "cairn-eval-v1";
-    const arms = validateArms(options.arms ?? [{ id: "baseline", disabled_capability: null }]);
+    const suppliedArms = options.arms ?? [{ id: "baseline", disabled_capability: null }];
+    const experimentKind = options.experimentKind
+        ?? (suppliedArms.length === 1 ? "two_pass" : "ablation");
+    if (experimentKind === "two_pass" && suppliedArms.length !== 1)
+        throw new Error("Two-pass evaluation requires one arm.");
+    if ((experimentKind === "ablation" || experimentKind === "skill_candidate") && suppliedArms.length !== 2) {
+        throw new Error(`${experimentKind} evaluation requires two arms.`);
+    }
+    const arms = validateArms(suppliedArms, experimentKind);
     const passes = ["run1", "run2"];
     const schedule = buildEvalSchedule({ taskSet, arms, repetitions, passes, seed });
     const adapterBase = dirname(adapterInput.path);
@@ -304,6 +319,7 @@ export function validateEvalInputs(options) {
         repetitions,
         seed,
         arms,
+        experiment_kind: experimentKind,
         passes,
         concurrency: 1,
         task_set_commit: taskSetCommit,
@@ -320,6 +336,7 @@ export function validateEvalInputs(options) {
         repetitions,
         seed,
         arms,
+        experiment_kind: experimentKind,
         passes: ["run1", "run2"],
         concurrency: 1,
         invocation_count: schedule.invocation_count,

package/mcp-memory-server/dist/eval-process.js

@@ -246,6 +246,37 @@ function hasTrailingJsonDocument(value) {
     }
     return false;
 }
+export async function runBoundedJsonProcess(options) {
+    const request = options.request_schema.safeParse(options.request);
+    if (!request.success)
+        throw new EvalProcessError("invalid_request");
+    const commandResult = await runCommandInternal({
+        command: options.command,
+        cwd: options.cwd,
+        env: options.env,
+        timeout_ms: options.timeout_ms,
+        max_stdout_bytes: options.max_stdout_bytes,
+        kill_grace_ms: options.kill_grace_ms,
+        signal: options.signal,
+        stdout_mode: "raw",
+        stdin_document: Buffer.from(JSON.stringify(request.data), "utf8"),
+    });
+    if (commandResult.exit_code !== 0 || commandResult.signal !== null || commandResult.cleanup !== "closed") {
+        throw new EvalProcessError("adapter_error", commandResult);
+    }
+    let parsed;
+    try {
+        parsed = JSON.parse(commandResult.stdout ?? "");
+    }
+    catch {
+        throw new EvalProcessError(hasTrailingJsonDocument(commandResult.stdout ?? "") ? "multiple_json" : "invalid_json", commandResult);
+    }
+    const result = options.result_schema.safeParse(parsed);
+    if (!result.success)
+        throw new EvalProcessError("invalid_result", commandResult);
+    const { stdout: _stdout, ...processResult } = commandResult;
+    return { ...processResult, result: result.data };
+}
 export async function runBoundedJsonAdapter(options) {
     const request = evalAdapterRequestSchema.safeParse(options.request);
     if (!request.success)

package/mcp-memory-server/dist/eval-runner.js

@@ -9,7 +9,7 @@ import { CAPABILITY_IDS, capabilityStatusSchema, } from "./capability-schema.js"
 import { canonicalDigest, evalReportSchema, evalReportRuntimeContractDigest, } from "./eval-schema.js";
 import { EvalProcessError, runBoundedCommand, runBoundedJsonAdapter, } from "./eval-process.js";
 import { checkpointEvalReport, createEvalReportStore, } from "./eval-report.js";
-import { cleanupEvalWorkspace, createEvalWorkspace, runTaskPreparation, runTaskVerifier, } from "./eval-workspace.js";
+import { applyEvalWorkspaceOverlay, cleanupEvalWorkspace, createEvalWorkspace, runTaskPreparation, runTaskVerifier, } from "./eval-workspace.js";
 const moduleDirectory = dirname(fileURLToPath(import.meta.url));
 const publishedReportSchemaPath = resolve(moduleDirectory, "..", "..", "schemas", "eval-report.schema.json");
 let reportSchemaDigestsCache;
@@ -557,6 +557,8 @@ export async function runEvalObservation(options) {
             observation.missing_reasons.push("preparation_failed");
         }
         else {
+            if (options.workspace_overlay)
+                await applyEvalWorkspaceOverlay(workspace, options.workspace_overlay);
             const request = {
                 schema_version: 1,
                 experiment_id: options.report.experiment_id,
@@ -698,7 +700,7 @@ export async function runEvalObservation(options) {
     await checkpoint(options.report_store, options.report);
     return { observation, snapshot: distilledSnapshot, cancelled: observation.terminal_state === "cancelled" };
 }
-function initialReport(plan, experimentId) {
+function initialReport(plan, experimentId, fallbackExperimentKind) {
     const now = new Date().toISOString();
     const sourceCommit = plan.task_set_commit ?? (plan.source.kind === "git" ? plan.source.revision : plan.task_set_digest);
     const missingness = { count: 0, reasons: [] };
@@ -706,7 +708,7 @@ function initialReport(plan, experimentId) {
         schema_version: 1,
         experiment_id: experimentId,
         status: "partial",
-        experiment_kind: plan.arms.length === 1 ? "two_pass" : "ablation",
+        experiment_kind: plan.experiment_kind ?? fallbackExperimentKind,
         task_set_digest: plan.task_set_digest,
         adapter_config_digest: plan.adapter_config_digest,
         source_revision: plan.source.kind === "git" ? plan.source.revision : plan.task_set_digest,
@@ -750,7 +752,7 @@ async function runExperiment(options, capabilityArms) {
     });
     if (reportStore.experiment_id !== experimentId)
         throw new Error("report_experiment_mismatch");
-    const report = initialReport(options.plan, experimentId);
+    const report = initialReport(options.plan, experimentId, capabilityArms ? "ablation" : "two_pass");
     const snapshotsByTask = new Map();
     const notesByTask = new Map();
     const snapshots = [];
@@ -782,6 +784,7 @@ async function runExperiment(options, capabilityArms) {
                 distill_command: options.distill_command,
                 distill_timeout_ms: options.distill_timeout_ms,
                 capability_runtime: capabilityRuntime,
+                workspace_overlay: options.workspace_overlays?.[row.arm],
             });
             executed.push(row.observation_id);
             if (result.snapshot) {
@@ -813,6 +816,9 @@ export async function runTwoPassExperiment(options) {
     return runExperiment(options);
 }
 export async function runCapabilityAblation(options) {
+    if (options.plan.experiment_kind !== undefined && options.plan.experiment_kind !== "ablation") {
+        throw new Error("ablation_experiment_kind_required");
+    }
     const arms = buildAblationArms(options.disabled_capability);
     const intended = arms.map(({ id, disabled_capability }) => ({ id, disabled_capability }));
     if (canonicalDigest(options.plan.arms) !== canonicalDigest(intended))

package/mcp-memory-server/dist/eval-schema.js

@@ -13,7 +13,7 @@ export const EVAL_OBSERVATION_TERMINAL_STATES = [
     "invalid_result",
 ];
 export const EVAL_PASS_STATES = ["passed", "failed", "unknown"];
-export const EVAL_EXPERIMENT_KINDS = ["two_pass", "ablation"];
+export const EVAL_EXPERIMENT_KINDS = ["two_pass", "ablation", "skill_candidate"];
 const MAX_TEXT_BYTES = 1024 * 1024;
 const MAX_COMMAND_ARGS = 256;
 const MAX_TASKS = 10_000;

package/mcp-memory-server/dist/eval-workspace.js

@@ -1,3 +1,4 @@
+import { createHash } from "node:crypto";
 import { constants, existsSync } from "node:fs";
 import { chmod, lstat, mkdir, mkdtemp, open, readFile, realpath, writeFile, } from "node:fs/promises";
 import { tmpdir } from "node:os";
@@ -155,7 +156,7 @@ export async function createEvalWorkspace(options) {
     };
     try {
         if (options.plan.source.kind === "git") {
-            const added = await gitCommand(options.plan.source.repository_root, ["worktree", "add", "--detach", workspace.source_path, options.plan.source.revision], "exit-only");
+            const added = await gitCommand(options.plan.source.repository_root, ["worktree", "add", "--quiet", "--detach", workspace.source_path, options.plan.source.revision], "exit-only");
             requireSuccessfulCommand(added, "workspace_add_failed");
             workspace.worktree_registered = true;
             await verifyGitWorkspace(workspace);
@@ -192,6 +193,54 @@ export async function runTaskPreparation(workspace, options = {}) {
         signal: options.signal,
     });
 }
+export async function applyEvalWorkspaceOverlay(workspace, overlay) {
+    const segments = overlay.relative_path.split("/");
+    if (overlay.relative_path.startsWith("/") || overlay.relative_path.endsWith("/")
+        || overlay.relative_path.includes("\\") || segments.some((segment) => segment === "" || segment === "." || segment === "..")
+        || /[\u0000-\u001f\u007f]/.test(overlay.relative_path)) {
+        throw new Error("workspace_overlay_path_invalid");
+    }
+    const target = resolve(workspace.source_path, overlay.relative_path);
+    if (!isContained(workspace.source_path, target) || target === workspace.source_path) {
+        throw new Error("workspace_overlay_path_escape");
+    }
+    let cursor = workspace.source_path;
+    for (const segment of segments.slice(0, -1)) {
+        cursor = join(cursor, segment);
+        try {
+            const info = await lstat(cursor);
+            if (info.isSymbolicLink() || !info.isDirectory())
+                throw new Error("workspace_overlay_parent_unsafe");
+        }
+        catch (error) {
+            if (error.code !== "ENOENT")
+                throw error;
+            await mkdir(cursor, { mode: 0o700 });
+        }
+    }
+    try {
+        const info = await lstat(target);
+        if (info.isSymbolicLink() || !info.isFile())
+            throw new Error("workspace_overlay_target_unsafe");
+    }
+    catch (error) {
+        if (error.code !== "ENOENT")
+            throw error;
+    }
+    const parent = await realpath(dirname(target));
+    const source = await realpath(workspace.source_path);
+    if (!isContained(source, parent))
+        throw new Error("workspace_overlay_parent_escape");
+    const bytes = Buffer.from(overlay.content, "utf8");
+    const digest = createHash("sha256").update(bytes).digest("hex");
+    if (digest !== overlay.digest)
+        throw new Error("workspace_overlay_digest_mismatch");
+    await writeFile(target, bytes, { mode: 0o600 });
+    await chmod(target, 0o600);
+    const storedDigest = createHash("sha256").update(await readFile(target)).digest("hex");
+    if (storedDigest !== overlay.digest)
+        throw new Error("workspace_overlay_write_mismatch");
+}
 function verifierReason(code) {
     return `verifier_${code}`;
 }