cowork-harness 2.0.0 → 2.0.1

package/.claude/skills/cowork-harness/SKILL.md

@@ -3,8 +3,8 @@ name: cowork-harness
 description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. Also when iterating or hardening a skill across fixes, or grounding a skill's self-critique against its own run evidence — including a document-analysis skill (cap table, deck, financial model, transcript) that needs an uploaded file attached to be critiqued at all. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
 metadata:
   author: cowork-harness
-  version: 2.0.0
-  tracks-harness: cowork-harness 2.0.0 (baseline desktop-1.34493.1)
+  version: 2.0.1
+  tracks-harness: cowork-harness 2.0.1 (baseline desktop-1.34493.1)
 ---
 
 # cowork-harness
@@ -22,7 +22,7 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
 allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
 the highest-value part. Read it.
 
-> **Version note:** the facts and `file:line` pointers here track `cowork-harness 2.0.0` (baseline
+> **Version note:** the facts and `file:line` pointers here track `cowork-harness 2.0.1` (baseline
 > `desktop-1.34493.1`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
 > `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
 
@@ -39,7 +39,7 @@ Before the first command, confirm the CLI is reachable and **fail loud** (never
 
 - **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
 - **Replay-only? Skip `doctor`.** Replaying committed cassettes needs no Docker, no staged agent, and no token — and every tier's `doctor` validates the auth token (the live tiers also Docker + the staged agent), so a ✗ there is expected, not a blocker. Go straight to `cowork-harness replay <cassette>`.
-- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 2.0.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=2.0.0" <cmd>` (Node ≥ 22), or install once with `npm i -g "cowork-harness@>=2.0.0"`. **Pin `@>=2.0.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
+- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 2.0.1**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@^2.0.1" <cmd>` (Node ≥ 22), or install once with `npm i -g "cowork-harness@^2.0.1"`. **Pin `@^2.0.1`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
 
   This skill documents the CURRENT surface, not release history. If `cowork-harness --version` is
   OLDER than the floor, the per-release record of what you are missing is [CHANGELOG.md](https://github.com/yaniv-golan/cowork-harness/blob/main/CHANGELOG.md)
@@ -220,9 +220,14 @@ run's echoed `--answer "<q>=<choice>"` footer lines into the scenario's `answers
 being unattended. Skip the transcribe step only for one-off/exploratory runs.
 <!-- answer-channels:end -->
 
-**Never hand-write the `req-N.json`/`resp-N.json` files.** `gates` and `answer` wrap the protocol — the
-atomic temp+rename, the `{id, answers}` envelope, the multiSelect array shape. Hand-rolling a Monitor over
-the raw files is the single most common mistake on this channel.
+**For a QUESTION gate, never hand-write the `req-N.json`/`resp-N.json` files.** `gates` and `answer` wrap
+the protocol — the atomic temp+rename, the `{id, answers}` envelope, the multiSelect array shape.
+Hand-rolling a Monitor over the raw files is the single most common mistake on this channel.
+
+`answer` writes `{id, answers}` and nothing else, so it covers **question gates only**. The channel also
+carries **permission**, **dialog** and **elicit** gates, whose replies need `{behavior}` / `{action}` — for
+those, write `resp-N.json` yourself, following the `reply_with` template the gate's own `req-N.json`
+advertises (it spells out the exact shape, e.g. `{"id":"…","behavior":"allow|deny"}`).
 
 Exact accepted values (teach precisely): `--on-unanswered` takes `fail|prompt|first` on `skill`,
 only `fail|first` on `run`. **`llm` is NOT an `--on-unanswered` value** — the bare flag
@@ -416,7 +421,8 @@ missing was anything saying so while you could still act. Related: recording at
 (`protocol`/`hostloop`/`cowork`→hostloop) into a repo-visible path is refused outright (gotcha 25).
 The clean answer there is `fidelity: container` (sealed, `HOME=/tmp`, nothing to leak) — **not**
 redirecting `--out` outside the repo and moving the file in afterwards, which trades a loud refusal
-for a permanently unverifiable cassette.
+for a cassette that cannot verify staleness from its own location — recoverable only by passing
+`--session <file>` on every invocation thereafter.
 
 **Author answers WITHOUT re-paying — the cheap loop.** You don't need a fresh paid record to discover a
 scenario's gates or their labels: `--keep` ONE run, then `cowork-harness trace <run-dir> --view questions`
@@ -917,8 +923,11 @@ repeats the assertion/replay-relevant ones alongside the schema (a scoped subset
       `artifact_json` / `transcript_matches`), never just `result: success`.
     - `on_unanswered` governs **unanswered** `AskUserQuestion` gates; the `stalled` signal covers
       stalling *after* one is answered — two different failure modes.
-    - **Free-text aside:** a "type-it-in-notes" option has **no scripted deterministic answer** today
-      (the `OTHER:` directive works only on the LLM-decider path, not scripted `choose:`, and only on
+    - **Free-text aside:** the scripted key for a "type-it-in-notes" option is **`answer:`** — an
+      arbitrary string delivered verbatim, bypassing label validation by author intent (Cowork
+      auto-provides an "Other" free-text path on every gate). Mutually exclusive with `choose:`; setting
+      both fails loud. What has no scripted equivalent is the `OTHER:` *directive*
+      (it works only on the LLM-decider path, not scripted `choose:`, and only on
       **single-select** gates — a **multi-select** gate is index-only, so `OTHER:` fails loud there; on an
       options-bearing single-select gate a bare out-of-set LLM answer also fails loud (exit 2) — see the
       LLM-decider free-text note in `references/fidelity-and-answers.md`). An LLM decision answered via

package/.claude/skills/cowork-harness/references/ci-recipe.md

@@ -1,6 +1,6 @@
 # CI recipe — replay vs live lanes
 
-Self-contained reference. Tracks `cowork-harness 2.0.0` (baseline `desktop-1.34493.1`).
+Self-contained reference. Tracks `cowork-harness 2.0.1` (baseline `desktop-1.34493.1`).
 
 **Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
 job-summary reporter (verdict table, staleness findings, cost/turns when available):
@@ -13,7 +13,7 @@ job-summary reporter (verdict table, staleness findings, cost/turns when availab
 ```
 
 The Action's `version` input defaults to `latest` — intentional so a copy-pasted recipe tracks the current
-release; pin an exact version (e.g. `version: "2.0.0"`) for reproducible CI.
+release; pin an exact version (e.g. `version: "2.0.1"`) for reproducible CI.
 
 Reach for the manual multi-step form below only when you need per-step control the Action's inputs don't
 cover (a custom flag combination, a different runner matrix per step, or `lint`/`verify-cassettes` gated
@@ -30,15 +30,19 @@ jobs:
     runs-on: [self-hosted, linux, arm64]   # needs Docker + this staged ELF; not a stock GitHub-hosted runner
     steps:
       - uses: actions/checkout@v4
-      - name: Stage the agent binary (official channel, sha256-verified — see https://github.com/yaniv-golan/cowork-harness/blob/main/docs/maintenance.md)
+      - name: Stage the agent binary (official channel, sha256-verified against the pinned baseline)
         run: |
           V=2.1.237   # match your scenario's pinned baseline's agentVersion
+          # The expected digest is baselines/desktop-<ver>.json -> agentBinary.sha256. Paste it here, or
+          # read it with jq if you vendor the baseline. An unverified download is an unverified agent:
+          # this step FAILS rather than staging one, which is the whole point of naming it "verified".
+          EXPECTED=<paste agentBinary.sha256 for $V>
           curl -fSL "https://downloads.claude.ai/claude-code-releases/$V/linux-arm64/claude" -o "$RUNNER_TEMP/claude-$V"
+          echo "$EXPECTED  $RUNNER_TEMP/claude-$V" | sha256sum -c -
           chmod +x "$RUNNER_TEMP/claude-$V"
-          # verify against the committed baseline's sha256 (baselines/desktop-*.json → agentBinary.sha256)
-          # before trusting it — see the "Agent-binary provenance" section of
-          # https://github.com/yaniv-golan/cowork-harness/blob/main/docs/maintenance.md
           echo "COWORK_AGENT_BINARY=$RUNNER_TEMP/claude-$V" >> "$GITHUB_ENV"
+          # Background on the provenance chain: the "Agent-binary provenance" section of
+          # https://github.com/yaniv-golan/cowork-harness/blob/main/docs/maintenance.md
       - uses: yaniv-golan/cowork-harness@main
         with:
           command: run
@@ -58,7 +62,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
 GitHub-hosted runners, no token/Docker/agent:
 
 ```yaml
-- run: npm i -g "cowork-harness@>=2.0.0"
+- run: npm i -g "cowork-harness@^2.0.1"
 - run: cowork-harness lint scenarios/*.yaml --strict --min-severity WARN
                                                     # no silent false-greens. WITHOUT --strict this
                                                     # step cannot fail on a WARN-class rule (e.g.
@@ -323,7 +327,7 @@ jobs:
         with: { node-version: '24' }
       - uses: actions/setup-python@v5
         with: { python-version: '3.x' }                                       # python3 only — PyYAML is bundled with the linter
-      - run: npm i -g "cowork-harness@>=2.0.0"
+      - run: npm i -g "cowork-harness@^2.0.1"
       - run: cowork-harness lint scenarios/*.yaml                              # no-silent-false-green (needs python3; PyYAML bundled)
       - run: cowork-harness verify-cassettes cassettes/ --output-format json   # privacy + staleness gate
       - run: cowork-harness replay cassettes/ --output-format json             # token-free content/structure
@@ -352,7 +356,7 @@ jobs:
             echo "live=true" >> "$GITHUB_OUTPUT"
           fi
       - if: steps.guard.outputs.live == 'true'
-        run: npm i -g "cowork-harness@>=2.0.0"
+        run: npm i -g "cowork-harness@^2.0.1"
       - if: steps.guard.outputs.live == 'true'
         run: cowork-harness run scenarios/ --output-format json
         env:
@@ -374,10 +378,14 @@ sandbox).
 ## Reading results in CI
 
 `--output-format json` emits a machine envelope on stdout (human output goes to stderr):
-`{tool, version, command, ok, results[], error}` — one `RunResult` per scenario. Overall pass for a
-scenario = `result === "success" && assertions.every(pass)`. Exit code is non-zero if any assertion
-fails or a run errors, so a plain `cowork-harness run scenarios/` is already CI-ready without parsing
-JSON.
+`{tool, version, command, ok, results[], error}` — one `RunResult` per scenario. **Overall pass for a
+scenario is `verdict.pass`** (envelope-wide: `ok`), and it is strictly stronger than
+`result === "success" && assertions.every(pass)`: the verdict also carries ~20 signal codes that fail a run
+with no failing assertion at all — `stalled`, `outputs_delete`, `mount_delete`, `host_path_leak`,
+`undelivered_deliverables`, `missing_capability`, `permissive_auto_allow`, `ended_with_question`,
+`infra_error`, and more. A parser that reimplements the shorter formula greens through every one of them.
+Read `ok` / `verdict.pass`, or just use the exit code — a plain `cowork-harness run scenarios/` is already
+CI-ready without parsing JSON.
 
 **Telling *why* a run failed, without scraping stderr.** Each result carries a `verdict` whose
 `failures[]` is the one place every failure reason is enumerated, in one shape (the same object lands

package/.claude/skills/cowork-harness/references/critique.md

@@ -1,6 +1,6 @@
 # Critique — the facts a plugin install can't otherwise reach
 
-Tracks `cowork-harness 2.0.0` (baseline `desktop-1.34493.1`). This is **not** a trim of the full
+Tracks `cowork-harness 2.0.1` (baseline `desktop-1.34493.1`). This is **not** a trim of the full
 [`docs/critique.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/critique.md) (repo-only —
 flags, cost, reproduction discipline, known limitations all live there). This file covers exactly what a
 plugin install cannot otherwise discover: the run-dir artifact a harvester actually reads, the report's

package/.claude/skills/cowork-harness/references/fidelity-and-answers.md

@@ -1,6 +1,6 @@
 # Fidelity tiers & answer paths
 
-Self-contained reference. Tracks `cowork-harness 2.0.0` (baseline `desktop-1.34493.1`).
+Self-contained reference. Tracks `cowork-harness 2.0.1` (baseline `desktop-1.34493.1`).
 
 ## Fidelity tiers (`fidelity:` in the scenario)
 
@@ -210,7 +210,7 @@ up often enough to spell out:
 - **A green `replay` proves "same as when recorded," not "correct today."** `replay` never touches
   a filesystem or network — it re-evaluates assertions from the frozen cassette. A fixed set of
   keys is live-only and **skipped outright** on replay (absent from `assertions[]`, not vacuously
-  passed): `file_absent`, `no_delete_in_outputs`, `self_heal_ran`, `transcript_no_host_path`, `egress_denied`,
+  passed): `file_absent`, `no_delete_in_outputs`, `no_delete_in_mounts`, `self_heal_ran`, `transcript_no_host_path`, `egress_denied`,
   `egress_allowed`, `no_mcp_error`, `max_peak_rss_bytes`, `semantic_matches`, `no_lost_write_back`, and `expect_denied`.
   Everything else that *is* evaluated is checked against the **recording**, not fresh behavior — a
   green replay says the skill produced these events when it was recorded, not that it still does

package/.claude/skills/cowork-harness/references/scenario-schema.md

@@ -1,6 +1,6 @@
 # Scenario & session schema, assertion catalog, web_fetch, full gotchas
 
-Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 2.0.0`
+Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 2.0.1`
 (baseline `desktop-1.34493.1`). If your checkout is newer, prefer the live [`docs/scenario.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/scenario.md),
 [`docs/session.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/session.md), and `SPEC.md`.
 

package/.claude/skills/cowork-harness/references/task-recipes.md

@@ -2,7 +2,7 @@
 
 Each recipe composes facts that live scattered across SKILL.md and the other references into one
 decision path. Every one answers a question a real fleet owner had to work out the hard way.
-Tracks `cowork-harness 2.0.0` (baseline `desktop-1.34493.1`), same as SKILL.md's front-matter. Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
+Tracks `cowork-harness 2.0.1` (baseline `desktop-1.34493.1`), same as SKILL.md's front-matter. Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
 Recipe 3's `init-redact` shipped in 0.24.0 and are part of the current feature set — no version gate
 needed if your CLI meets SKILL.md's version floor.
 
@@ -49,11 +49,11 @@ what production would do. Two layers of defense:
 
 ### Cassette anatomy (what you're looking at when you open one)
 
-Top-level fields of a `*.cassette.json` (schema [`schema/cassette.v11.json`](https://github.com/yaniv-golan/cowork-harness/blob/main/schema/cassette.v11.json)):
+Top-level fields of a `*.cassette.json` (schema [`schema/cassette.v12.json`](https://github.com/yaniv-golan/cowork-harness/blob/main/schema/cassette.v12.json)):
 
 | Field | What it is |
 |---|---|
-| `$schema`, `generator`, `cassetteVersion` | Provenance: schema URL, producing tool, format version — the MINIMUM a reader needs for this scenario, not the recorder's version (current max: 11; a `lane: remote` scenario stamps 11, nearly everything else stays 10) |
+| `$schema`, `generator`, `cassetteVersion` | Provenance: schema URL, producing tool, format version — the MINIMUM a reader needs for this scenario, not the recorder's version (current max: 12 — the hash-format epoch floors every stamp there, so a fresh recording stamps 12 whatever its `lane:`) |
 | `scenario` | The embedded scenario snapshot at record time |
 | `events` | The recorded agent event stream (the replay source) |
 | `controlOut` | Driver→agent control responses — presence unlocks gate asserts on replay |

package/.claude/skills/cowork-harness/scripts/scenario.py

@@ -409,6 +409,67 @@ def _is_positional_choose(choose):
     return any(isinstance(v, str) and (v == "first" or v.isdigit()) for v in vals)
 
 
+# Single-segment absolute paths that legitimately appear in prompt prose. A `/word` from this set is a
+# path, not a slash command, so it never raises the not-leading warning below.
+_SLASH_PATH_WORDS = frozenset(
+    {"outputs", "mnt", "tmp", "home", "users", "var", "etc", "usr", "bin", "dev", "opt", "workspace", "root", "srv"}
+)
+# A `/`-prefixed token at a word start (start-of-string or whitespace), captured WHOLE up to the next
+# space. An opening bracket or quote also counts as a word start, so "(/deck-review)" is seen; `and/or`,
+# `8/22` and `https://x` still cannot match, because their slash follows a letter, digit or colon. The
+# token is then classified in Python rather than by a lookahead: an earlier lookahead-based pattern
+# BACKTRACKED, matching `/mn` inside `/mnt/uploads` because a shorter prefix satisfied the lookahead.
+_SLASH_TOKEN_RE = re.compile(r"""(?:^|[\s(\[{"'])/(\S+)""")
+# What a slash command may look like once trailing sentence punctuation is stripped: a bare name, or a
+# plugin-qualified `plugin:skill`. Anchored, so any residual `/` or `.` disqualifies it as a path/filename.
+_SLASH_CMD_NAME_RE = re.compile(r"^[A-Za-z][A-Za-z0-9_-]*(?::[A-Za-z][A-Za-z0-9_-]*)?$")
+
+
+def _lint_prompt_slash(doc, path):
+    """W: `prompt:` names a slash command somewhere other than position 0.
+
+    The agent binary resolves a slash command only when the TRIMMED prompt starts with `/` — its parser
+    trims, then requires `startsWith("/")` (verified against agent 2.1.239 in the harness's own spawn
+    shape: `-p --input-format stream-json --output-format stream-json --setting-sources user`). A slash
+    named mid-sentence is never expanded. It reaches the model as ordinary prose, and the model may then
+    reach for the `Skill` tool on its own — the model-invocation path, i.e. exactly the unreliable
+    auto-trigger a slash is normally used to bypass. The scenario still runs and can still pass, so the
+    failure mode is a scenario that silently tests something other than what it reads as.
+
+    Deliberately silent when the prompt DOES start with `/`: that is the working case. Registration is not
+    checkable statically (it depends on how the skill is staged), so an unresolvable leading name is left
+    to the run itself, where it shows up as `Unknown command: /x` with `num_turns: 0`.
+    """
+    findings = []
+    prompt = doc.get("prompt")
+    if not isinstance(prompt, str) or prompt.lstrip().startswith("/"):
+        return findings
+    seen = []
+    for m in _SLASH_TOKEN_RE.finditer(prompt):
+        # Trailing sentence punctuation is not part of the name, so "use /deck-review." still counts.
+        name = m.group(1).rstrip(".,;:!?)]}\"'")
+        if not _SLASH_CMD_NAME_RE.match(name):
+            continue  # a path (`/mnt/uploads`), a filename (`/deck.pdf`), or not command-shaped
+        if name.lower() in _SLASH_PATH_WORDS or name in seen:
+            continue
+        seen.append(name)
+    for name in seen:
+        findings.append(
+            Finding(
+                "WARN",
+                "prompt-slash-not-leading",
+                f"`prompt:` names `/{name}` but does not START with it. A slash command is expanded only "
+                "when the trimmed prompt begins with `/`, so here it reaches the model as ordinary prose "
+                "and the skill is NOT preloaded — the model may or may not reach for it on its own, which "
+                "is the auto-trigger path a slash is normally used to bypass.",
+                f'Put the command first — `prompt: "/{name} <args>"` — or drop the slash if the scenario '
+                "means to test auto-triggering from a natural request.",
+                path,
+            )
+        )
+    return findings
+
+
 def lint_doc(doc, path, raw_lines):
     findings = []
     if not isinstance(doc, dict):
@@ -456,6 +517,9 @@ def lint_doc(doc, path, raw_lines):
                 )
             )
 
+    # W: a slash command named mid-prompt is never expanded — see _lint_prompt_slash.
+    findings.extend(_lint_prompt_slash(doc, path))
+
     # W: unknown assertion keys inside assert items (e.g. invented file_not_empty, kind, path)
     unknown_assert = sorted(assert_keys - ASSERT_KEYS)
     for k in unknown_assert:
@@ -1834,11 +1898,16 @@ def build_scenario(args):
         )
         content_lines.append("  - gate_answers_delivered: true   # the steered answers actually reached the model")
 
-    live_lines = []
+    # Two buckets, not one. `file_exists`/`user_visible_artifact` are MANIFEST_KEYS above — they DO
+    # evaluate on replay whenever the cassette carries an artifacts manifest, which `record` has
+    # snapshotted since 0.24. Filing them under a "LIVE-only" heading taught the reader the opposite of
+    # what this file's own taxonomy says, and of what the `manifest-needs-snapshot` INFO tells them.
+    manifest_lines = []
     for p in (args.file or []):
-        live_lines.append(f"  - file_exists: {p}")
+        manifest_lines.append(f"  - file_exists: {p}")
     for p in (args.artifact or []):
-        live_lines.append(f"  - user_visible_artifact: {p}")
+        manifest_lines.append(f"  - user_visible_artifact: {p}")
+    live_lines = []
     if args.no_delete:
         live_lines.append("  - no_delete_in_outputs: true")
     for h in (args.egress_allowed or []):
@@ -1850,12 +1919,16 @@ def build_scenario(args):
     L.append("assert:")
     L.append("  # --- content / structure: evaluate on the token-free replay PR gate AND live ---")
     L.extend(content_lines)
+    if manifest_lines:
+        L.append("  # --- artifacts: replay-checkable WHEN the cassette carries an artifacts manifest ---")
+        L.extend(manifest_lines)
     if live_lines:
-        L.append("  # --- filesystem / egress: LIVE-only (skipped on replay, with a loud warning) ---")
+        L.append("  # --- egress / deletes: LIVE-only (skipped on replay, with a loud warning) ---")
         L.extend(live_lines)
-    else:
-        L.append("  # TODO add filesystem/egress checks (file_exists / user_visible_artifact /")
-        L.append("  #      egress_denied / no_delete_in_outputs) — they run on the LIVE lane only.")
+    if not manifest_lines and not live_lines:
+        L.append("  # TODO add artifact checks (file_exists / user_visible_artifact — these replay from")
+        L.append("  #      the cassette's artifacts manifest) and filesystem/egress checks")
+        L.append("  #      (egress_denied / no_delete_in_outputs — LIVE lane only).")
 
     if args.web_fetch:
         notes.append(