cowork-harness 1.13.2 → 1.15.0

package/.claude/skills/cowork-harness/SKILL.md

@@ -3,8 +3,8 @@ name: cowork-harness
 description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. Also when iterating or hardening a skill across fixes, or grounding a skill's self-critique against its own run evidence — including a document-analysis skill (cap table, deck, financial model, transcript) that needs an uploaded file attached to be critiqued at all. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
 metadata:
   author: cowork-harness
-  version: 1.13.2
-  tracks-harness: cowork-harness 1.13.2 (baseline desktop-1.24012.9)
+  version: 1.15.0
+  tracks-harness: cowork-harness 1.15.0 (baseline desktop-1.24012.9)
 ---
 
 # cowork-harness
@@ -22,7 +22,7 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
 allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
 the highest-value part. Read it.
 
-> **Version note:** the facts and `file:line` pointers here track `cowork-harness 1.13.2` (baseline
+> **Version note:** the facts and `file:line` pointers here track `cowork-harness 1.15.0` (baseline
 > `desktop-1.24012.9`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
 > `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
 
@@ -39,7 +39,7 @@ Before the first command, confirm the CLI is reachable and **fail loud** (never
 
 - **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
 - **Replay-only? Skip `doctor`.** Replaying committed cassettes needs no Docker, no staged agent, and no token — and every tier's `doctor` validates the auth token (the live tiers also Docker + the staged agent), so a ✗ there is expected, not a blocker. Go straight to `cowork-harness replay <cassette>`.
-- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 1.13.2**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=1.13.2" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=1.13.2"`. **Pin `@>=1.13.2`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
+- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 1.15.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=1.15.0" <cmd>` (Node ≥ 22), or install once with `npm i -g "cowork-harness@>=1.15.0"`. **Pin `@>=1.15.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
 
   This skill documents the CURRENT surface, not release history. If `cowork-harness --version` is
   OLDER than the floor, the per-release record of what you are missing is [CHANGELOG.md](https://github.com/yaniv-golan/cowork-harness/blob/main/CHANGELOG.md)
@@ -284,12 +284,22 @@ quoting, an egress assert on `protocol` fidelity, `transcript_no_host_path` on `
 (ERROR — fails by design at those tiers; WARN on `fidelity: cowork`, whose tier resolves per the
 baseline's host-loop gate), non-empty `requires_capabilities` on `protocol` without
 `allow_missing_capability` (ERROR — the capability probe can't run there, so the run hard-fails as
-unverifiable), a container-only assertion key (`no_scratchpad_leak`/`present_files_called`) off the
-`container` tier (ERROR on `protocol`/`microvm`/`hostloop`, WARN on `cowork`), a `controlOut`-gated key on a non-`controlOut` replay, mixed-class assertion items,
+unverifiable), `no_scratchpad_leak` off `container` (ERROR on `protocol`/`microvm`/`hostloop` — hostloop's
+`present_files` passes a validated path through without promoting, so there is no scratch→outputs copy
+to leak; WARN on `cowork`, whose tier resolves per the baseline gate) or `present_files_called` on
+`protocol`/`microvm` (ERROR — served only at `container`/`hostloop`), or `present_files_called`/`no_scratchpad_leak`/`user_visible_artifact` on `lane: remote` (ERROR — the runtime rejects those at scenario load time, so the tier rules are suppressed there), a `controlOut`-gated key on a non-`controlOut` replay, mixed-class assertion items,
 and hallucinated schema (`assertions:` vs `assert:`, unknown keys). Exit code is non-zero on errors
 (CI-friendly). `scaffold` auto-upgrades the tier if you ask for egress on `protocol`, so it never
 emits a scenario `lint` would reject.
 
+**`lint` is the LENIENT check — the loader is the strict one.** An unknown top-level key is a ⚠ WARN in
+`lint` (exit 0) but a **hard error** in the runtime (`Unrecognized key: "<k>"`, exit 2) — so a scenario
+that lints with warnings may still not run. To check whether a scenario actually loads, without spending:
+`cowork-harness record <file.yaml> --dry-run` (exit 2 on a schema error; a directory reports each
+`✗ broken:` file and exits 1). Corollary: **a key from a newer harness fails LOUD on an older CLI**, never
+silently — `lane:` on a pre-1.14.0 CLI does not load rather than defaulting to `lane: local`. That is why a
+new scenario key ships with a version floor.
+
 ## Part II — RUN, RECORD & LOCK
 
 You have an authored scenario. This Part runs it, reads the verdict, locks it into a
@@ -368,7 +378,12 @@ cassette — has its own recipe:
    input + result are captured, not just errored ones. When iterating, tag generations with `--label` and
    pair a critique only with a `result.json` whose `fingerprint.skillHash` **matches** the skill that
    produced it (`inspect`/the run-index row surface a short `skillHash` prefix; `verify-run` warns when a
-   kept run predates the current skill). **Multi-skill plugin caveat (post-1.7.0 CLIs with `--skill`):**
+   kept run predates the current skill). **The hazard is general, not critique-specific:** repeated
+   `run`/`skill` invocations of one scenario accumulate in the SAME scenario directory regardless of skill
+   version, so a plain `stats <scenario>` silently averages pre-fix and post-fix runs together. Compare
+   generations with **`stats <scenario> --group-by skill-hash`** (or narrow with `--skill-hash <prefix>` /
+   `--label <tag>`); an un-split window spanning more than one generation now warns.
+   **Multi-skill plugin caveat (post-1.7.0 CLIs with `--skill`):**
    skillHash keys the whole MOUNTED plugin, so on a multi-skill plugin the hash alone cross-pairs
    critiques of DIFFERENT skills — pair by the report's `(gradedSkillHash, gradedSkill)` pair. **On a pre-1.5.0 CLI the `skill` lane emits no `fingerprint.skillHash` at all**, so a
    pairing step there silently groups on an absent key instead of erroring — check the field is present, or
@@ -406,6 +421,16 @@ Recognize these before "fixing" a non-bug:
   covers the residual (a mid-message `?`, or tool work after the last gate that still ended asking). Read
   the final message before acting — a legitimate question-posing answer that wrote a file never fires.
   Assert `allow_stall: true` if ending on a question is the intended terminal state.
+- **`undelivered_deliverables`** (`WARN`) — the skill produced file(s) **outside every user-visible root**
+  and never delivered them. On a **remote** Cowork session the workspace is reclaimed at session end, so
+  they are destroyed; on a **local** one they persist but stay invisible to the user. Either way the user
+  does not get them. It fires with no assertion written — `present_files_called` covers the positive case
+  only when you thought to ask for it, and the runs that most need this are the ones where nobody did.
+  **Silent when the evidence cannot answer the question** (no workspace walk, or a tier that runs no
+  scratchpad walk, absent delivery telemetry, or a resumed turn) — "cannot tell" never reads as "clean".
+  Fix by writing deliverables under `outputs/` or a connected folder, or delivering them explicitly; assert
+  **`allow_undelivered_deliverables: true`** when the leftovers are intentional (intermediates, caches,
+  downloaded inputs) rather than a delivery gap.
 - **`host_path_leak`** — skipped at **`hostloop` and `protocol`** fidelity (the agent runs on real host
   paths there, so a host path in model-visible text is expected, not a leak); it is *armed* at
   `container`/`microvm`, but only *fires* on an actual scanned leak with no authored
@@ -424,7 +449,7 @@ Recognize these before "fixing" a non-bug:
   `RunResult.scan` is undefined and the host-path + outputs-delete guards **did not run this run**. Not a
   pass or a defect — assert `no_delete_in_outputs` / `transcript_no_host_path` to hard-fail on it instead.
 
-The full 14-code signal table (severity + per-signal opt-out) is in
+The full 17-code signal table (severity + per-signal opt-out) is in
 [`references/scenario-schema.md`](./references/scenario-schema.md); [`docs/scenario.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/scenario.md) (repo-only) carries
 the fuller narrative.
 
@@ -440,7 +465,10 @@ harness writes/updates throughout the run's lifecycle (including a crash-safety
 error/`SIGTERM`, AND staleness detection for a hard `SIGKILL`/OOM-kill that no exit handler can catch —
 either way you get `"error"`/`stale` instead of a permanently-trusted `"running"`), so liveness is
 checkable regardless of PID namespace. The harness prints `[status] <outDir>` to stderr as soon as the
-run starts, so capture stderr to get the exact directory — but `<dir>` also accepts the run-dir root
+run starts, so capture stderr to get the exact directory — **unless you passed `--compact` (or `--demo`,
+which implies it), which suppress that line** (it is a raw, un-tildeified host path, exactly what those shareable-output
+modes exist to withhold; `status.json` is still written either way, so `status` still works) — but
+`<dir>` also accepts the run-dir root
 passed to `--run-dir` (a directory without its own `status.json`): it scans up to two levels down for the
 newest session's `status.json` and reads that. `--follow` fails loud on a timeout/staleness
 rather than hanging forever. (Fuller recipe in [`docs/run-status.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/run-status.md) — repo-only, not in the installed
@@ -508,9 +536,28 @@ decide which assertions from *Assertions: two orthogonal axes* are worth adding)
   call-count/timing table, the sub-agent dispatch tree, the gate lifecycle, the tool/error rollups, …);
   bare `trace` digests the whole run. The view set is actively being extended — run `trace --help` for
   the current list rather than relying on a fixed enumeration here.
+- **`lane: local|remote`** (scenario key, default `local`) — which Cowork lane's DELIVERY CONTRACT the run
+  is held to. Cowork picks the lane per session ("Run this task: In the cloud / On your computer") and
+  cloud is the default for new sessions; the lanes disagree about what *delivered* means. On `remote`,
+  location delivers nothing (a remote container has no auto-delivering outputs dir and is reclaimed at
+  session end), `present_files` is NOT served, and `user_visible_artifact` /
+  `present_files_called` / `no_scratchpad_leak` are rejected at LOAD time as unable to pass. Reach for it
+  to check a skill's delivery survives the lane most new sessions get. Orthogonal to `fidelity` — a
+  `lane: remote` scenario still runs locally.
 - **`cowork-harness stats [--metric <m>]`** — aggregate across the run index: `cost`, `duration`,
-  `tokens`, `cache-tokens`, `model-cost`, `turns`, `pass-rate`.
-- **`result.json` carries the raw fields** the assertions read: `verdict`, `toolDurations`, `models`, `toolErrors`,
+  `tokens`, `cache-tokens`, `model-cost`, `turns`, `pass-rate`. Filters: `--since`/`--baseline`/`--branch`,
+  plus `--skill-hash <prefix>`/`--label <tag>` to narrow to ONE skill generation and
+  `--group-by scenario|skill-hash|label|fidelity` to split per generation — or per effective fidelity
+  tier — instead of aggregating across them (a window spanning >1 generation warns — see Gotcha 6;
+  >1 tier warns too, independently, with `--group-by fidelity` as its own remedy). `--runs` lists the
+  individual runs behind each summary with their `skillHash`/`runLabel`, so
+  you can tell which arm a run belonged to without opening its `result.json`. `--last <n>` windows per group.
+- **`result.json` carries the raw fields** the assertions read: `verdict`, `lane` (which Cowork delivery
+  contract the run was held to — see Gotcha 24), `scratchpadEvidenceComplete` (did a COMPLETE scratchpad
+  walk observe this run — what distinguishes "nothing was left undelivered" from "cannot tell"), `cost` (`cost.usd` = the SDK's
+  `total_cost_usd` for the run — the authoritative single-run spend; NOT the same source as summing
+  `modelUsage[].costUSD`, which is what `trace --view usage` reports, so the two can differ),
+  `usage` (`input_tokens`/`output_tokens`/`turns`), `toolDurations`, `models`, `toolErrors`,
   `redundantToolCalls`, `modelUsage`, `thinking`, `skillActivity`, `subagents[]` (prompt/`dispatchModel`/
   `resolvedModel`/output/`attributedSkillId`, `outputTruncated`, `referencesRead`, `reasoning`/`reasoningElided`),
   `context` (tools/mcpServers/availableSkills), `tasks`,
@@ -539,7 +586,7 @@ decide which assertions from *Assertions: two orthogonal axes* are worth adding)
   cost spike from fan-out reads as `trace --view dispatches` (how many, which agent) against that model's
   per-model usage — the harness doesn't line-item each sub-agent's tokens.
 - **Debugging a wrong Cowork UI panel.** Each panel is reconstructed in `result.json`: **Progress** =
-  `tasks[]`, **Working folder** = `workspaceFiles[]` (classified output/mount/input, with a
+  `tasks[]`, **Working folder** = `workspaceFiles[]` (classified output/mount/input/scratchpad — the last being the agent's working area outside every user-visible root, with a
   `trace --view files` diff), **Context / Connectors** = `context` (tools / mcpServers / availableSkills),
   **Scratch-pad → outputs** = `presentedFiles[]`. If a panel looks wrong in a run, read its field. An
   **absent** `workspaceFiles`/`artifacts` (a replay result, or a run whose workspace root was missing at
@@ -723,10 +770,14 @@ repeats the assertion/replay-relevant ones alongside the schema (a scoped subset
     isolated** (its own egress sidecar network + proxy, its own per-session run dir), so parallel records don't
     cross-talk — the concurrency bound exists only for the Docker address pool + API rate limits, not correctness.
 
-17. **Editing `scenarios/*.yaml` `assert:` does NOT change a plain `replay`.** *Why:* `replay` evaluates the
-    assertions **frozen in the cassette** by default — it is byte-deterministic and ignores the working tree (so
-    a committed cassette can't silently re-interpret against an uncommitted YAML). This is *loud* rather than a *silent*
-    no-op; now plain `replay` prints a `::notice::` when a sibling's `assert:` differs and points you at the fix.
+17. **Editing `scenarios/*.yaml` does NOT change a plain `replay` — the WHOLE scenario is frozen, not just
+    `assert:`.** *Why:* a cassette captures every key (`lane:`, `fidelity:`, `baseline:`, `prompt:`, `skills:` …)
+    and `replay` evaluates all of them from that frozen copy — byte-deterministic, ignoring the working tree (so
+    a committed cassette can't silently re-interpret against an uncommitted YAML). **Only `assert:`
+    (+`expect_denied:`) can be opted back to disk; any other edited key reaches a replay only by re-recording.**
+    This is *loud* rather than a *silent* no-op: plain `replay` prints a `::notice::` when a sibling's
+    `assert:`/`prompt:` differs, and when the sibling **fails to load** at all (a typo'd or too-new key) — and
+    points you at the fix.
     *Fix:* to re-check token-free against the edited block, `replay --assert-from <scenario.yaml>` (or
     `--reassert`). That opt-in path is safe by construction for the authored fields — it **hard-fails** if
     `prompt`/`answers`/`baseline`/`fidelity`/`skills`/`requires_capabilities` or the skill content (when a
@@ -767,11 +818,14 @@ repeats the assertion/replay-relevant ones alongside the schema (a scoped subset
 
 22. **`lint` floods CI with INFO advisories that don't apply to you.** *Why:* two rules —
     `manifest-needs-snapshot` and `gate-needs-controlout` — fire on the mere presence of manifest/gate
-    assertion keys, unconditionally. The linter is **static**: it never reads your cassettes, so it cannot
-    know whether yours already carry an `artifacts` manifest and `controlOut` (a current cassette does).
-    On a healthy fleet every one of those lines is a false alarm. *Fix:* `lint --min-severity WARN` in CI
-    (≥1.11.0) — the INFO advisories stay one flag away for interactive use. `--strict --min-severity ERROR`
-    behaves as a plain lint, not a contradiction.
+    assertion keys. The linter is **static**: it never reads your cassettes, so it cannot know whether
+    yours already carry an `artifacts` manifest and `controlOut` (a current cassette does). On a healthy
+    fleet every one of those lines is a false alarm. One exception: `manifest-needs-snapshot` is
+    suppressed for `user_visible_artifact` on `lane: remote` — the only manifest-backed key that lane also
+    rejects outright (`lane-remote-incompatible-key`, an ERROR), so the INFO would be redundant advice
+    about a key the scenario can never even load with. `gate-needs-controlout` has no such exception. *Fix:*
+    `lint --min-severity WARN` in CI (≥1.11.0) — the INFO advisories stay one flag away for interactive use.
+    `--strict --min-severity ERROR` behaves as a plain lint, not a contradiction.
 23. **`verify-cassettes`/`replay` report a `discovery-surface` note on cassettes you just recorded fine.**
     *Why:* the cassette froze its `system/init` tool inventory from before the skills/plugins discovery
     servers existed at that tier (added 1.10.0). It is a non-gating **note**, never a finding — it cannot
@@ -779,6 +833,18 @@ repeats the assertion/replay-relevant ones alongside the schema (a scoped subset
     `mcp__skills__*`/`mcp__plugins__*`; then re-record. It stays silent at `microvm`/`protocol`, where
     re-recording would never produce those tools anyway.
 
+24. **Never name the file-delivery tool in a `SKILL.md`.** *Why:* Cowork has **two**, one per product
+    lane, and an agent only sees the one for the surface it is on. The desktop-local sandbox this harness
+    emulates is served `mcp__cowork__present_files` (`{files:[{file_path}]}`); **remote** cloud-container
+    Cowork instead gives the agent the native `SendUserFile` (`files: string[]`, required `status`,
+    optional `caption`/`display`). A skill that hardcodes either name works on one lane and fails on the
+    other — and probing a remote session makes this harness look like it emulates the wrong tool under the
+    wrong schema. It doesn't; the lanes genuinely disagree. *Fix:* describe the **outcome** ("deliver the
+    file to the user") and let the model pick its surface's tool. The `no_scratchpad_leak` /
+    `present_files_called` assertion keys are harness-side names and stay valid either way.
+    ([`docs/fidelity-gaps.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/fidelity-gaps.md)
+    → "File delivery" has the binary-verified detail; repo-only.)
+
 For the assertion catalog, the YAML schema, the fidelity/answer tables, and the CI recipe, read the
 files in `references/` (the gotchas above are the full list; the references repeat only the
 assertion/replay-relevant ones).

package/.claude/skills/cowork-harness/references/ci-recipe.md

@@ -1,6 +1,6 @@
 # CI recipe — replay vs live lanes
 
-Self-contained reference. Tracks `cowork-harness 1.13.2` (baseline `desktop-1.24012.9`).
+Self-contained reference. Tracks `cowork-harness 1.15.0` (baseline `desktop-1.24012.9`).
 
 **Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
 job-summary reporter (verdict table, staleness findings, cost/turns when available):
@@ -13,7 +13,7 @@ job-summary reporter (verdict table, staleness findings, cost/turns when availab
 ```
 
 The Action's `version` input defaults to `latest` — intentional so a copy-pasted recipe tracks the current
-release; pin an exact version (e.g. `version: "1.13.2"`) for reproducible CI.
+release; pin an exact version (e.g. `version: "1.15.0"`) for reproducible CI.
 
 Reach for the manual multi-step form below only when you need per-step control the Action's inputs don't
 cover (a custom flag combination, a different runner matrix per step, or `lint`/`verify-cassettes` gated
@@ -58,7 +58,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
 GitHub-hosted runners, no token/Docker/agent:
 
 ```yaml
-- run: npm i -g "cowork-harness@>=1.13.2"
+- run: npm i -g "cowork-harness@>=1.15.0"
 - run: cowork-harness lint scenarios/*.yaml          # no silent false-greens
 - run: cowork-harness verify-cassettes cassettes/    # privacy + staleness
 - run: cowork-harness replay cassettes/              # token-free content/structure
@@ -107,8 +107,10 @@ The split is not just about tokens — it decides **where each lane can run**:
   one (see [docs/cassette.md](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/cassette.md)). This is your **always-on PR gate**. With `--output-format json`, read each
   `results[].verdict.{pass,signals}` for **per-cassette** pass/fail and the reason (the top-level `ok`
   collapses the whole batch) — e.g. a `stalled` signal fails a cassette whose assertions all passed.
-  The PR gate evaluates the assertions **frozen in the cassette**; to re-check against an edited on-disk
-  `assert:` without a paid re-record, use `replay --assert-from <scenario.yaml>` (or `--reassert`).
+  The PR gate evaluates the **whole scenario frozen in the cassette** (`lane:`/`fidelity:`/`baseline:` too,
+  not only `assert:`) — a YAML edit cannot move this gate's verdict. To re-check against an edited on-disk
+  `assert:` without a paid re-record, use `replay --assert-from <scenario.yaml>` (or `--reassert`); any other
+  edited key needs a re-record.
 - **`run` / `record` (live).** Spawns the real agent in a sandbox: real model tokens + Docker **+ the
   staged Claude Code agent ELF**, bind-mounted from a local Claude Desktop install or pointed to via
   `COWORK_AGENT_BINARY`. Nothing is bundled, and **the agent binary is not redistributable** — a clean
@@ -219,10 +221,10 @@ jobs:
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-node@v4
-        with: { node-version: '20' }
+        with: { node-version: '24' }
       - uses: actions/setup-python@v5
         with: { python-version: '3.x' }                                       # python3 only — PyYAML is bundled with the linter
-      - run: npm i -g "cowork-harness@>=1.13.2"
+      - run: npm i -g "cowork-harness@>=1.15.0"
       - run: cowork-harness lint scenarios/*.yaml                              # no-silent-false-green (needs python3; PyYAML bundled)
       - run: cowork-harness verify-cassettes cassettes/ --output-format json   # privacy + staleness gate
       - run: cowork-harness replay cassettes/ --output-format json             # token-free content/structure
@@ -251,7 +253,7 @@ jobs:
             echo "live=true" >> "$GITHUB_OUTPUT"
           fi
       - if: steps.guard.outputs.live == 'true'
-        run: npm i -g "cowork-harness@>=1.13.2"
+        run: npm i -g "cowork-harness@>=1.15.0"
       - if: steps.guard.outputs.live == 'true'
         run: cowork-harness run scenarios/ --output-format json
         env:

package/.claude/skills/cowork-harness/references/critique.md

@@ -1,6 +1,6 @@
 # Critique — the facts a plugin install can't otherwise reach
 
-Tracks `cowork-harness 1.13.2` (baseline `desktop-1.24012.9`). This is **not** a trim of the full
+Tracks `cowork-harness 1.15.0` (baseline `desktop-1.24012.9`). This is **not** a trim of the full
 [`docs/critique.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/critique.md) (repo-only —
 flags, cost, reproduction discipline, known limitations all live there). This file covers exactly what a
 plugin install cannot otherwise discover: the run-dir artifact a harvester actually reads, the report's
@@ -30,7 +30,8 @@ turn rows carry `critiqueRole:"task"` / `"reflection"`.
 
 Two things a harvester needs: roll-ups are excluded from `stats` aggregation (they carry no verdict —
 counting them adds a phantom run and drags `passRate` toward 1), so **filter them out of any pass-rate
-computed over raw rows**; and a roll-up with `result:"error"` had an unpriced workload, so its totals
+computed over raw rows** — that exclusion also governs `stats --group-by skill-hash`, so a per-generation
+**total spend** still needs the raw rows (only the roll-up carries the evaluator passes); and a roll-up with `result:"error"` had an unpriced workload, so its totals
 UNDERCOUNT. The index is the only cost record that survives run-dir pruning.
 
 ## The report's item shape — no `title`, no `summary`

package/.claude/skills/cowork-harness/references/fidelity-and-answers.md

@@ -1,6 +1,6 @@
 # Fidelity tiers & answer paths
 
-Self-contained reference. Tracks `cowork-harness 1.13.2` (baseline `desktop-1.24012.9`).
+Self-contained reference. Tracks `cowork-harness 1.15.0` (baseline `desktop-1.24012.9`).
 
 ## Fidelity tiers (`fidelity:` in the scenario)
 
@@ -211,14 +211,18 @@ up often enough to spell out:
   (`staleness[]` flags skill/baseline drift as a hint; only a live `run` re-confirms current
   behavior). See [`docs/cassette.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/cassette.md) § "Still skipped on replay" and [`docs/scenario.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/scenario.md) § "Which
   assertions survive replay."
-- **Container-only assertions can't verify off the `container` tier.** `no_scratchpad_leak` and
-  `present_files_called` check the `present_files` delivery path, which is served **only** on
-  `container` — not `hostloop`/`microvm`. Asserting them off-container hard-fails at runtime (a red
-  run, not a false green), so you won't be fooled if you write the assertion. The quieter trap is a
-  scenario that runs at `hostloop`/`microvm`/`protocol` and simply omits these assertions: a green
-  run there proves **nothing** about scratchpad-leak safety or present_files delivery, because that
-  tier never exercises the delivery path the assertions would check. Use `fidelity: container` for
-  present_files/scratchpad-delivery coverage.
+- **`present_files` assertions can't verify off the tiers that serve the tool.** `no_scratchpad_leak` and
+  `present_files_called` check the `present_files` delivery path — the desktop-local lane's tool; remote
+  Cowork delivers via the agent-native `SendUserFile` instead, so never hardcode a delivery tool name in
+  a SKILL.md (SKILL.md Gotcha 24). **The harness** serves `present_files` on `container` **and `hostloop`**
+  — not `microvm`/`protocol`. `present_files_called` works at both; `no_scratchpad_leak` stays
+  `container`-only, because hostloop's handler passes a validated path through without promoting, so
+  there is no scratch→outputs copy to leak. Asserting either key off the tiers that serve it hard-fails at
+  runtime (a red run, not a false green), so you won't be fooled if you write the assertion. The quieter
+  trap is a scenario that runs at `microvm`/`protocol` and simply omits these assertions: a green run
+  there proves **nothing** about scratchpad-leak safety or present_files delivery, because that tier
+  never exercises the delivery path the assertions would check. Use `fidelity: container` or `hostloop`
+  for present_files-delivery coverage; `no_scratchpad_leak` still needs `container` specifically.
 - **The harness doesn't observe rendered-artifact interactions, browser downloads, or human
   clicks.** It runs the agent headless — no webview, no browser, no person clicking "Submit." A
   class of Cowork bug (a client-side write-back to a relative URL that resolves-but-fails against

package/.claude/skills/cowork-harness/references/scenario-schema.md

@@ -1,6 +1,6 @@
 # Scenario & session schema, assertion catalog, web_fetch, full gotchas
 
-Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 1.13.2`
+Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 1.15.0`
 (baseline `desktop-1.24012.9`). If your checkout is newer, prefer the live [`docs/scenario.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/scenario.md),
 [`docs/session.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/session.md), and `SPEC.md`.
 
@@ -37,6 +37,20 @@ fidelity: container                 # protocol | container | microvm | hostloop
 execution: local                    # OPTIONAL — orthogonal to fidelity (a privilege/sandbox tier, all
                                     # local): local (default) | cloud-describe (RESERVED — no runner
                                     # exists yet; authoring it is a load-time error, not a silent no-op)
+lane: local                         # OPTIONAL — which Cowork lane's DELIVERY CONTRACT the run is held to:
+                                    # local (default) | remote. On `remote`, location delivers NOTHING (a
+                                    # remote container has no auto-delivering outputs dir and is reclaimed
+                                    # at session end) and `present_files` is NOT served (a local MCP server
+                                    # can't reach a remote session) — so `user_visible_artifact` and
+                                    # present_files_called/no_scratchpad_leak are REJECTED AT SCENARIO-LOAD
+                                    # TIME (before the run starts, before any spend), not left to fail
+                                    # unverifiable/can't-verify at assertion time. Orthogonal to fidelity
+                                    # and execution — a `lane: remote` scenario still runs locally.
+                                    # Delivery semantics only; the remote device bridge is deliberately
+                                    # unmodeled.
+                                    # NEEDS >= 1.14.0: on an older CLI a scenario carrying `lane:` does NOT
+                                    # load (`Unrecognized key: "lane"`, exit 2) — it is NOT reinterpreted as
+                                    # `lane: local`. Adopting the key means raising your floor.
 on_unanswered: fail                 # policy for unscripted gates: fail | prompt | first | llm — run rejects prompt
                                     # ("agent" is retired — no longer a valid value)
 
@@ -298,8 +312,8 @@ same set live from the schema.
 | `all_tasks_completed: true` | every task in `RunResult.tasks[]` reached status `"completed"` — **requires ≥1 task** (a zero-task run fails; assert `task_count_min` for presence); evidence-unavailable if tasks telemetry is absent |
 | `task_count_min: <N>` | at least N tasks were created (`RunResult.tasks.length >= N`) — the presence companion for task assertions |
 | `task_status: {match, status}` | a task whose `subject` OR `id` matches the regex `match` reached `status` — evidence-unavailable if tasks telemetry is absent; also fails **malformed** when a TaskCreate result was unparseable (corrupt task telemetry), mirroring the guard `all_tasks_completed`/`task_count_min` already had |
-| `no_scratchpad_leak: true` | every file presented via `present_files` that was in the scratchpad was successfully promoted to `mnt/outputs` (none left behind) — vacuously passes if nothing was presented (pair with a presence check to require a delivery); content-class: both the `present_files` tool_use and its own tool_result live in the ordinary events stream, so `RunResult.presentedFiles` re-derives identically on replay (meaningfully replay-checkable, same as `skill_triggered`); evidence-unavailable if `presentedFiles` telemetry is absent (an older run predating the feature). **`fidelity: container` only** — `present_files` is not served on hostloop/microvm, so a scratchpad-delivered file is neither promoted to `mnt/outputs` nor detected there (a skill that delivers via write-to-cwd→`present_files` will false-red `user_visible_artifact` on those tiers; use `container`, or write directly to `outputs/`). **Only `true` is valid** |
-| `present_files_called: true` | at least one file was actually delivered via the `present_files` tool (`RunResult.presentedFiles` is non-empty) — the presence companion to `no_scratchpad_leak` (which passes vacuously when nothing was presented). Pair them to require a delivery **and** require it not to leak. Content-class (re-derives identically on replay). **`fidelity: container` only** — `present_files` is not served on hostloop/microvm. **Only `true` is valid** |
+| `no_scratchpad_leak: true` | every file presented via `present_files` that was in the scratchpad was successfully promoted to `mnt/outputs` (none left behind) — vacuously passes if nothing was presented (pair with a presence check to require a delivery); content-class: both the `present_files` tool_use and its own tool_result live in the ordinary events stream, so `RunResult.presentedFiles` re-derives identically on replay (meaningfully replay-checkable, same as `skill_triggered`); evidence-unavailable if `presentedFiles` telemetry is absent (an older run predating the feature). **Container-only on the merits**: hostloop serves `present_files` but never promotes (its handler passes a validated path through), so there is no scratch→outputs copy that could leak. On microvm/protocol a scratchpad-delivered file is neither promoted to `mnt/outputs` nor detected there (a skill that delivers via write-to-cwd→`present_files` will false-red `user_visible_artifact` on those tiers; use `container`, or write directly to `outputs/`). Hostloop is not one of them: the agent's cwd there already *is* the outputs dir, so a write-to-cwd delivery is under a user-visible root from the start and `user_visible_artifact` passes. **The tool name is lane-specific:** `present_files` is the desktop-local lane's tool (the one this harness emulates); remote Cowork delivers via the agent-native `SendUserFile` instead, so a skill should describe the delivery outcome rather than naming either tool — this key asserts the harness-side delivery record either way. **Only `true` is valid** |
+| `present_files_called: true` | at least one file was actually delivered via the `present_files` tool (`RunResult.presentedFiles` is non-empty) — the presence companion to `no_scratchpad_leak` (which passes vacuously when nothing was presented). Pair them to require a delivery **and** require it not to leak. Content-class (re-derives identically on replay). **`fidelity: container` or `hostloop`** — the harness serves `present_files` at both (hostloop via a handler mirroring production's own host-loop branch: validate the path, pass it through, no promotion). `protocol` and `microvm` report cannot-verify. See the `no_scratchpad_leak` row, which stays container-only for a different reason; and see the lane note there — the tool name differs on remote Cowork. **Only `true` is valid** |
 | `question_asked: <regex>` | the agent asked an AskUserQuestion whose text matches |
 | `questions_count_max: <N>` | at most N **sub-questions** asked — a bundled `AskUserQuestion` with K sub-questions counts as K, not 1; `trace --view questions`'s footer total uses the same definition |
 | `gate_answers_delivered: true` | every answered gate's answer reached the model (observed `tool_result`; unobserved = fail); **zero gates fired passes vacuously** — pair with `gate_answer_count_min` to also require a gate |
@@ -314,6 +328,7 @@ same set live from the schema.
 | `allow_missing_capability: true` | verdict modifier — suppresses the default-fail when the (partial "core") agent image omits a capability the skill used but real Cowork ships (OCR/LibreOffice/markitdown/opencv/PDF-tables). Assert only when the skill's fallback is genuinely equivalent; otherwise rebuild full parity (`--build-arg COWORK_FULL_PARITY=1`). Also opts out of the `requires_capabilities` declared-need check. Live tiers only |
 | `allow_l0_plugin_divergence: true` | verdict modifier — opt into L0/protocol plugin divergence: suppresses the default-fail when a plugin behaves differently at `protocol` (L0) fidelity than under a sandboxed tier. Live tiers only |
 | `allow_stall: true` | verdict modifier — suppresses the `stalled` default-fail when a run ends on a question having done no productive tool work after its last gate (the agent asked for input and stopped — incl. re-asking in plain text after answering an `AskUserQuestion`); assert only when ending on a question is intended, else script the answer (`answer:` / `--answer` / a decider) |
+| `allow_undelivered_deliverables: true` | verdict modifier — suppresses the `undelivered_deliverables` WARN. Working in the scratchpad is Cowork's designed pattern, so a skill that legitimately leaves intermediates, caches or downloaded inputs behind can say so instead of carrying permanent noise. The signal is warn-only and never fails a run on its own; reach for this when the scratch activity is intentional, not to silence a real delivery gap |
 | `transcript_no_host_path: true` | no host path (`/Users/`, `/opt/cowork/`, `/home/`, `/root/`) leaked into model-visible text — **incompatible with `hostloop` AND `protocol`**: hostloop's native file tools legitimately expose real host paths, and protocol (L0) runs the agent's file tools on the real host cwd with no sealed filesystem, so this fails BY DESIGN on both (the harness warns at run start if asserted anyway); use `container`/`microvm` for this check |
 | `egress_denied: <host>` | the host was blocked by the egress proxy |
 | `egress_allowed: <host>` | the host was allowed through |
@@ -358,6 +373,7 @@ codes (`VerdictSignal["code"]` in `src/run/verdict.ts`):
 | `prompt_asset_missing` | warn | The run proceeded with a missing prompt asset (`COWORK_HARNESS_ALLOW_MISSING_PROMPT=1`); fidelity is degraded |
 | `scan_unavailable` | warn | Post-run scan evidence unavailable (`RunResult.scan` undefined) — the host-path and outputs-delete guards did not run this run |
 | `ended_with_question` | warn | Live-lane heuristic: the final answer contains a question and the run wrote no deliverable to `outputs/` — the lenient sibling of `stalled` (covers a mid-message `?`, or tool work after the last gate that still ended asking). Opt out: `allow_stall` |
+| `undelivered_deliverables` | warn | The skill produced file(s) OUTSIDE every user-visible root and never delivered them — on a remote Cowork session the workspace is reclaimed at session end; on a local one they stay invisible to the user. Fires on every run without opting in, because the scenarios that most need it are the ones whose author never considered delivery. Silent when the evidence cannot answer the question (no workspace walk, a tier that runs no scratchpad walk, absent delivery telemetry, or a resumed turn) — never a vacuous clean. Opt out: `allow_undelivered_deliverables` |
 | `exec_infra_error` | warn | Host-loop: one or more container `exec` calls failed for infrastructure reasons, so those tool calls returned an error to the agent. Warns rather than fails because the run's other evidence is intact — unlike `infra_error`, where a dead supervisor contaminates everything. Caveat: if *every* exec failed, the agent ran nothing and this still only warns — check `result.infraErrors` |
 
 A **fail**-severity signal does not change `result.result` (still `"success"`), but it DOES fail the
@@ -369,9 +385,12 @@ overall run verdict and exit code — `assert result: success` alone won't catch
 A cassette (`record`/`replay`) has **no filesystem and no network**. `replay` re-evaluates only the
 **content** assertions. The authoritative list is `ALWAYS_CONTENT_KEYS`/`QUESTION_GATE_KEYS`/`MANIFEST_KEYS` (composed) in `src/run/cassette.ts`.
 
-**Assertion source — frozen by default, on-disk by opt-in.** A plain `replay` evaluates the `assert:` block
-**frozen in the cassette** (byte-deterministic, ignores the working tree); editing `scenarios/<name>.yaml` does
-not change it — replay only prints a `::notice::` when a sibling's `assert:` differs. `replay --assert-from
+**Scenario source — the WHOLE scenario is frozen; only `assert:` can be opted back to disk.** A cassette
+captures every key (`name`/`prompt`/`session`/`baseline`/`fidelity`/`lane`/`skills`/`answers`/`execution`/
+`requires_capabilities`/`expect_denied`/`assert`), and a plain `replay` evaluates **all** of them from that
+frozen copy (byte-deterministic, ignores the working tree); editing `scenarios/<name>.yaml` does not change
+it — replay only prints a `::notice::` when a sibling's `assert:`/`prompt:` differs, or when it fails to
+load at all. An edited `lane:`/`fidelity:`/`baseline:` reaches a replay ONLY by re-recording. `replay --assert-from
 <scenario.yaml>` / `--reassert` is the opt-in token-free re-check against the on-disk block; it **hard-fails**
 on recording-shaping drift (`prompt`/`answers`/`baseline`/`skills`) or skill-content staleness (it implies
 `--fail-on-skill-drift`). `expect_denied`/filesystem/egress keys are sourced from on-disk but stay live-only —

package/.claude/skills/cowork-harness/references/task-recipes.md

@@ -2,7 +2,7 @@
 
 Each recipe composes facts that live scattered across SKILL.md and the other references into one
 decision path. Every one answers a question a real fleet owner had to work out the hard way.
-Tracks `cowork-harness 1.13.2` (baseline `desktop-1.24012.9`), same as SKILL.md's front-matter. Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
+Tracks `cowork-harness 1.15.0` (baseline `desktop-1.24012.9`), same as SKILL.md's front-matter. Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
 Recipe 3's `init-redact` shipped in 0.24.0 and are part of the current feature set — no version gate
 needed if your CLI meets SKILL.md's version floor.
 
@@ -118,7 +118,8 @@ have:
 4. **Replay-class note:** `max_turns` / `tool_calls_max` / `dispatch_count_max` re-evaluate
    token-free on replay; `questions_count_max` needs `controlOut` (Recipe 1's tree applies).
    `max_cost_usd` / `max_tokens` on replay assert the FROZEN recording's spend — near-zero signal
-   as a regression gate; if you need a cost gate, put it on the live lane via `stats`.
+   as a regression gate; if you need a cost gate, put it on the live lane via `stats` (history) or
+   `--max-budget-usd` (a pre-flight refusal before the run spends).
 
 ## Recipe 5 — Evaluate your skill's ANSWER QUALITY (semantic regression gate)
 
@@ -174,7 +175,14 @@ Hardening a skill is a loop: run → read what it did → fix → run again. Two
    output. **Reproduce before acting on a finding:** `cowork-harness skill <folder> "<prompt>" --repeat 5 --label gen-1`
    runs the same skill+prompt N times (2-100) and prints a variance rollup instead of a single pass/fail —
    `--repeat` works on the `skill` lane, not just `run`. A single green run proves it passed *once*.
-   Companions: `--min-pass-rate`, `--stop-on-diverge`, `--max-budget-usd`, `--allow-budget-stop`. It rejects `--session-id`/
+   Companions: `--min-pass-rate`, `--stop-on-diverge`, `--max-budget-usd`, `--allow-budget-stop`.
+   (`--max-budget-usd` also works WITHOUT `--repeat`, where it pre-flights a single run against that
+   scenario's cost history and refuses before spending; the others are batch-only.
+   **`record` accepts it too** — there it is CUMULATIVE across the batch (a `dir/` or `--rerecord-stale`
+   sweep is the least predictable spend in the CLI), refusing up front if the summed history exceeds the
+   cap. At `--concurrency 1` a running total also stops the batch once the cap is reached; above that the
+   stop is disabled and the tool says so, because with N runs in flight the total is only known after an
+   overshoot is already paid for. Unpriced scenarios contribute $0 and are named as a LOWER BOUND.) It rejects `--session-id`/
    `--resume` (both pin one run dir) and `--decider-cmd`/`--decider-dir` (a driving agent x N is not a
    measurement).
    The full loop (harvest -> reproduce -> fix -> prove freshness -> compare) is written out end-to-end in

package/.claude/skills/cowork-harness/scripts/assertion-keys.json

@@ -6,6 +6,7 @@
     "allow_missing_capability",
     "allow_permissive_auto_allow",
     "allow_stall",
+    "allow_undelivered_deliverables",
     "artifact_json",
     "compaction_occurred",
     "computer_links_resolve",
@@ -80,6 +81,7 @@
     "execution",
     "expect_denied",
     "fidelity",
+    "lane",
     "name",
     "on_unanswered",
     "prompt",
@@ -92,6 +94,7 @@
     "allow_l0_plugin_divergence",
     "allow_missing_capability",
     "allow_permissive_auto_allow",
-    "allow_stall"
+    "allow_stall",
+    "allow_undelivered_deliverables"
   ]
 }