cowork-harness 1.24.0 → 2.0.0

package/.claude/skills/cowork-harness/SKILL.md

@@ -3,8 +3,8 @@ name: cowork-harness
 description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. Also when iterating or hardening a skill across fixes, or grounding a skill's self-critique against its own run evidence — including a document-analysis skill (cap table, deck, financial model, transcript) that needs an uploaded file attached to be critiqued at all. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
 metadata:
   author: cowork-harness
-  version: 1.24.0
-  tracks-harness: cowork-harness 1.24.0 (baseline desktop-1.32352.0)
+  version: 2.0.0
+  tracks-harness: cowork-harness 2.0.0 (baseline desktop-1.34493.1)
 ---
 
 # cowork-harness
@@ -22,8 +22,8 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
 allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
 the highest-value part. Read it.
 
-> **Version note:** the facts and `file:line` pointers here track `cowork-harness 1.24.0` (baseline
-> `desktop-1.32352.0`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
+> **Version note:** the facts and `file:line` pointers here track `cowork-harness 2.0.0` (baseline
+> `desktop-1.34493.1`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
 > `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
 
 ## Preflight — make sure the harness can actually run
@@ -39,7 +39,7 @@ Before the first command, confirm the CLI is reachable and **fail loud** (never
 
 - **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
 - **Replay-only? Skip `doctor`.** Replaying committed cassettes needs no Docker, no staged agent, and no token — and every tier's `doctor` validates the auth token (the live tiers also Docker + the staged agent), so a ✗ there is expected, not a blocker. Go straight to `cowork-harness replay <cassette>`.
-- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 1.24.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=1.24.0" <cmd>` (Node ≥ 22), or install once with `npm i -g "cowork-harness@>=1.24.0"`. **Pin `@>=1.24.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
+- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 2.0.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=2.0.0" <cmd>` (Node ≥ 22), or install once with `npm i -g "cowork-harness@>=2.0.0"`. **Pin `@>=2.0.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
 
   This skill documents the CURRENT surface, not release history. If `cowork-harness --version` is
   OLDER than the floor, the per-release record of what you are missing is [CHANGELOG.md](https://github.com/yaniv-golan/cowork-harness/blob/main/CHANGELOG.md)
@@ -407,7 +407,7 @@ That choice is permanent: the cassette rewrites `scenario.session` and `scenario
 its own directory** at record time, so moving the file later — a different `--out`, a `git mv`, a copy
 into another repo — leaves those unresolvable and
 `verify-cassettes` reports `unverifiable-skill` ("can't verify ⇒ not green", exit 3) until you
-re-record at the new location. **`record` now says so BEFORE it spends:** a pre-flight — at the same
+re-record at the new location — or point `replay`/`verify-cassettes` at the session with `--session <file>`, which resolves it without a re-record. Since 2.0.0 a bare `replay` FAILS on this class rather than warning. **`record` now says so BEFORE it spends:** a pre-flight — at the same
 pre-spend point as the host-inventory refusal, and in `record --dry-run`, so the rehearsal is free —
 warns when the cassette would be written outside the scenario's tree, or when `session:` itself lives
 outside it (an absolute or `~` path: the mirror case, invisible to a check that only looks at where the
@@ -584,7 +584,9 @@ discovery source removed, so the agent answers from its own priors. **It is ONE
 experiment**: this invocation is the control. Run the same prompt a second time *without* the flag for
 the treatment arm and compare them yourself. Composed with `--repeat 5` it produces **5 ablated runs
 and 0 treatment runs** — N samples of the control, which is the intended reading and is not an A/B.
-Every ablated run is stamped `ablated: true` in `result.json`; a run that isn't stamped is a real run.
+The rollup says so on its verdict line: `repeat "<skill>": PASS [ABLATED — control arm] — 5/5 passed`.
+Every ablated run is stamped `ablated: true` in `result.json` and carries `ablated=true` on its
+`[provenance]` footer line; a run that isn't stamped is a real run.
 What the harness gives you here is the run execution and the control arm — designing the comparison
 (scrubbing giveaways, shuffling, judging blind, unblinding only after grading) is still yours.
 
@@ -1026,8 +1028,14 @@ repeats the assertion/replay-relevant ones alongside the schema (a scoped subset
 25. **Two distinct host-inventory consent flags — a record-time one and a verify-time one.** `record
     --allow-host-inventory-fixture` is the boolean consent to proceed recording a host-inheriting
     (`protocol`/`hostloop`/`cowork`-resolving-to-hostloop) cassette into a repo-visible path — otherwise
-    `record` refuses (freezing this machine's MCP servers/agents/account into a committed fixture is the
-    risk). `verify-cassettes --allow-host-inventory <regex>` is unrelated: a per-finding suppressor for the
+    `record` refuses before it spends (freezing this machine's MCP servers/agents/account into a committed
+    fixture is the risk). That pre-spend check **warns rather than refuses when the cassette already
+    exists** — refusing would fire on every `--rerecord-stale` pass — and it reads the tier and the
+    destination path, never the bytes. So `record` also scans the FINISHED recording, after redaction and
+    before the write: a `host-inventory`/`machine-inventory` finding on a repo-visible path is
+    **quarantined** to `<runs-root>/quarantine/` with a `.findings.txt` naming what leaked, and the command
+    fails without writing the path you asked for (the recording is not discarded — you paid for it).
+    `verify-cassettes --allow-host-inventory <regex>` is unrelated: a per-finding suppressor for the
     scanner's `host-inventory` class on an already-committed cassette. Passing one where the other command
     wants it fails as an unrecognized flag — they don't interchange. Depth: `references/ci-recipe.md`.
 

package/.claude/skills/cowork-harness/references/ci-recipe.md

@@ -1,6 +1,6 @@
 # CI recipe — replay vs live lanes
 
-Self-contained reference. Tracks `cowork-harness 1.24.0` (baseline `desktop-1.32352.0`).
+Self-contained reference. Tracks `cowork-harness 2.0.0` (baseline `desktop-1.34493.1`).
 
 **Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
 job-summary reporter (verdict table, staleness findings, cost/turns when available):
@@ -13,7 +13,7 @@ job-summary reporter (verdict table, staleness findings, cost/turns when availab
 ```
 
 The Action's `version` input defaults to `latest` — intentional so a copy-pasted recipe tracks the current
-release; pin an exact version (e.g. `version: "1.24.0"`) for reproducible CI.
+release; pin an exact version (e.g. `version: "2.0.0"`) for reproducible CI.
 
 Reach for the manual multi-step form below only when you need per-step control the Action's inputs don't
 cover (a custom flag combination, a different runner matrix per step, or `lint`/`verify-cassettes` gated
@@ -32,7 +32,7 @@ jobs:
       - uses: actions/checkout@v4
       - name: Stage the agent binary (official channel, sha256-verified — see https://github.com/yaniv-golan/cowork-harness/blob/main/docs/maintenance.md)
         run: |
-          V=2.1.229   # match your scenario's pinned baseline's agentVersion
+          V=2.1.237   # match your scenario's pinned baseline's agentVersion
           curl -fSL "https://downloads.claude.ai/claude-code-releases/$V/linux-arm64/claude" -o "$RUNNER_TEMP/claude-$V"
           chmod +x "$RUNNER_TEMP/claude-$V"
           # verify against the committed baseline's sha256 (baselines/desktop-*.json → agentBinary.sha256)
@@ -58,7 +58,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
 GitHub-hosted runners, no token/Docker/agent:
 
 ```yaml
-- run: npm i -g "cowork-harness@>=1.24.0"
+- run: npm i -g "cowork-harness@>=2.0.0"
 - run: cowork-harness lint scenarios/*.yaml --strict --min-severity WARN
                                                     # no silent false-greens. WITHOUT --strict this
                                                     # step cannot fail on a WARN-class rule (e.g.
@@ -77,7 +77,20 @@ GitHub-hosted runners, no token/Docker/agent:
 - run: cowork-harness replay cassettes/              # token-free content/structure
 ```
 
-Both lines matter: `replay` alone **warns** on a stale recording and exits 0, so dropping the
+If a cassette has MOVED (a `git mv`, a repo reorg, a copy between projects), staleness becomes
+unverifiable — `verify-cassettes` exits 3 and says the skill dirs are not resolvable. Recover with
+`--session <file>` on either command rather than re-recording:
+
+```yaml
+- run: cowork-harness verify-cassettes cassettes/moved.cassette.json --session sessions/default.yaml
+```
+
+It takes a session (not skill dirs) so `staleness.hash_ignore` survives, refuses a directory target,
+and echoes the dirs it resolved. Full contract:
+[docs/cassette.md](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/cassette.md).
+
+Both lines matter: for the CONTENT-drift classes (`skill`, `shared-root`) `replay` alone **warns** and
+exits 0, so dropping the
 `verify-cassettes` step means a skill edit silently stops being tested. (One command instead of two:
 `replay --fail-on-skill-drift`.)
 
@@ -210,9 +223,23 @@ dollar figures). In a skill repo these cassettes get **committed**. So:
 - **Host-inheriting record refused by default — `--allow-host-inventory-fixture` is the consent.** A
   `protocol`/`hostloop`/`cowork`-resolving-to-hostloop record into a repo-visible cassette path would
   freeze THIS machine's MCP server names, agents, and account metadata into a committed fixture, so
-  `record` refuses outright rather than warn. Pass `--allow-host-inventory-fixture` only when the
+  `record` refuses **before the paid spawn**. Pass `--allow-host-inventory-fixture` only when the
   recording session genuinely has no personal MCP servers or plugins to leak — it is a per-record
   boolean consent, not a pattern.
+
+  Two details that matter for a re-record loop. The pre-spend check **warns rather than refuses when the
+  cassette already exists**, deliberately: refusing there would fire on every `--rerecord-stale` pass and
+  make the escape flag reflexive. And it is a **prediction** — it reads the tier and the destination path,
+  never the resulting bytes, so it can be wrong in both directions.
+
+  So `record` also checks the **evidence**. After redaction and before the write, the finished cassette is
+  scanned; a `host-inventory` or `machine-inventory` finding on a repo-visible path is **quarantined** —
+  written to `<runs-root>/quarantine/` (honouring `--run-dir`/`COWORK_HARNESS_RUNS_DIR`) with a
+  `.findings.txt` sibling naming what leaked, and the command fails without writing the path you asked for.
+  The recording is not discarded; you paid for it. Only the machine-identity classes trigger this —
+  `email`/`currency`/`domain`/`path` are frequently legitimate scenario content, and a gate that fires on
+  those just teaches you to pass the escape flag. Outside a git repo it warns instead: nothing there
+  publishes the file by accident.
 - **Always-on scan gate** — `verify-cassettes` flags email / currency / bare-domain / local-path /
   machine-inventory matches it finds in the committed cassettes and **exits non-zero**, so "no leak" is
   a gate, not discipline. Non-zero is not one thing, though: exit `1` means verification RAN and found a
@@ -222,6 +249,13 @@ dollar figures). In a skill repo these cassettes get **committed**. So:
   $? -ne 0 ]` tripwire treats both the same — if you need to tell "the gate caught something" apart from
   "the gate couldn't run", branch on the exit code (or parse `--output-format json`'s per-file
   `findings`/`staleness` vs `unverifiable`/`version`/`error` buckets).
+
+  **Do not read `error` as "this file was never scanned".** The privacy scan needs a readable *transcript*
+  (an `events` array of strings), not a *valid* cassette — so a file that fails shape validation is still
+  scanned, and reports its findings **and** its `error`. Each result carries **`privacyScanned`**, which
+  answers that question directly. A gate that must not treat "could not verify" as "verified clean" should
+  key on `privacyScanned === false`, where `findings: []` is an absence of evidence rather than evidence of
+  absence. `--skip-privacy` also reports `false`, for the same reason.
   Suppress synthetic / public reference names (NVCA, Cooley GO, …) with `--allow <regex>`. (Multi-word
   proper names are NOT a default class — too noisy to gate on; add a pattern via config if your corpus
   needs it.)
@@ -289,7 +323,7 @@ jobs:
         with: { node-version: '24' }
       - uses: actions/setup-python@v5
         with: { python-version: '3.x' }                                       # python3 only — PyYAML is bundled with the linter
-      - run: npm i -g "cowork-harness@>=1.24.0"
+      - run: npm i -g "cowork-harness@>=2.0.0"
       - run: cowork-harness lint scenarios/*.yaml                              # no-silent-false-green (needs python3; PyYAML bundled)
       - run: cowork-harness verify-cassettes cassettes/ --output-format json   # privacy + staleness gate
       - run: cowork-harness replay cassettes/ --output-format json             # token-free content/structure
@@ -318,7 +352,7 @@ jobs:
             echo "live=true" >> "$GITHUB_OUTPUT"
           fi
       - if: steps.guard.outputs.live == 'true'
-        run: npm i -g "cowork-harness@>=1.24.0"
+        run: npm i -g "cowork-harness@>=2.0.0"
       - if: steps.guard.outputs.live == 'true'
         run: cowork-harness run scenarios/ --output-format json
         env:
@@ -357,13 +391,19 @@ in `result.json` and in the stdout envelope, by construction). Each entry is
 - **`staleness`** — on `replay --strict` / `--assert-from` / `--reassert`, skill or baseline drift,
   which those modes escalate to a hard failure on purpose (frozen events must not green an edited
   assert against a skill whose current source produces something else).
-- **`cassette-format`** — the cassette is too new for this build to interpret.
+- **`cassette-format`** — the cassette itself cannot be interpreted: too new a version for this build,
+  OR corrupt (duplicate/malformed control frames, a truncated recording). Before 1.25.0 the corrupt
+  cases were mis-reported as `assertion`, i.e. as if you had written them.
 - **`coverage`** — a `verify-run` answer-coverage miss: a gate the run fired that your `answers:` block
   does not cover.
 
-So `jq '[.verdict.failures[] | select(.kind=="assertion")]'` answers "did MY assertions pass?" and
-`select(.kind=="staleness")` answers "is the cassette stale?", from the envelope alone. Do not infer
-either from the exit code: every kind lands on exit 1.
+So `jq '[.results[]? | .verdict.failures[]? | select(.kind=="assertion")] | length'` answers "did MY
+assertions pass?" and swapping in `select(.kind=="staleness")` answers "is the cassette stale?", from
+the envelope alone. Do not infer either from the exit code: every kind lands on exit 1.
+
+Keep the `.results[]?` hop and the `?` operators. `.results[0]` silently ignores every scenario after
+the first when you pass a directory, and a bare `.verdict` does not exist at the envelope root at all —
+both read as "no failures" against a run that failed.
 
 > **Do not filter on whether `assertion` is present.** That was the only discriminator before `kind`
 > existed and it never worked in both directions: `coverage` entries carry a key too (an internal
@@ -400,19 +440,21 @@ evaluation), not present-and-passing. A CI script that counts assertions will se
 on replay vs live — compare by assertion identity / pass-fail, not by total count. The count of
 skipped live-only assertions is reported on each replay result as `skippedAssertions: {full, partial}`.
 
-## Staleness does NOT fail a replay by default — read it from the JSON
+## Staleness mostly does NOT fail a replay — read it from the JSON
 
-A plain `replay` **warns** on a stale cassette (skill/baseline drift) but stays `ok:true` — a green replay
-does **not** imply the recording is still valid. Each replay result carries `staleness[]`, an array of
+A plain `replay` **warns** on a DRIFTED cassette (skill/baseline drift) but stays `ok:true` — a green
+replay does **not** imply the recording is still valid. **Since 2.0.0 there is one exception:**
+`unverifiable-skill` — staleness that could not be checked at all, most often a cassette that moved —
+FAILS a bare `replay`. Recover with `--session <file>` rather than re-recording. Each replay result carries `staleness[]`, an array of
 `{class, message}`, so a token-free gate can act on it without `ok` being the whole story:
 
 | `class` | meaning | concern |
 |---|---|---|
 | `baseline` | platform baseline moved since record | low (format-compatible) |
 | `skill` / `shared-root` | the skill source the assertions validate drifted | **high** (assertions may validate dead code) |
-| `format` | recorded under an older hash format | re-record once |
+| `format` | the git/raw file-set mode or agent-scope differs from the recording | re-record under the same setting (waivable) |
 | `unverifiable-baseline` | the latest baseline couldn't be loaded | couldn't verify (env, not skill) |
-| `unverifiable-skill` | skill dirs unresolvable — skill staleness couldn't be checked | couldn't verify the skill |
+| `unverifiable-skill` | skill dirs unresolvable, **or** the cassette predates the hash-format epoch (v12) so its digest is not comparable | couldn't verify the skill. **Fails a bare `replay`.** For the epoch case try `rehash` first — it migrates without a re-record where it can prove the content unchanged (`rehash <file> --session <s.yaml>` if the cassette moved) |
 | `resolved-tier` | a `fidelity: cowork` cassette's recorded `effectiveFidelity` no longer matches what the baseline resolves to today (the host-loop gate flipped) | **high** (the recording exercises the wrong tier) |
 | `unverifiable-tier` | tier check couldn't run for a `fidelity: cowork` cassette (no recorded `effectiveFidelity`, or its pinned baseline failed to load) | couldn't verify the tier — re-record |
 
@@ -427,8 +469,11 @@ fail it either way.)
 To gate in CI, pick the severity you want:
 
 - `replay --strict` — fail (exit 1) on **any** staleness class.
-- `replay --fail-on-skill-drift` — fail only on skill-source drift (`skill` / `shared-root` / `unverifiable-skill`);
+- `replay --fail-on-skill-drift` — fail on the skill-source DRIFT classes (`skill` / `shared-root`); `unverifiable-skill` needs no flag, it fails the default verdict since 2.0.0;
   baseline / format / `unverifiable-baseline` stay non-failing warnings.
+  Note `--allow-failing` waives this gate wholesale, including the copy `--assert-from` turns on for you:
+  `replay --assert-from … --write --allow-failing` will persist an assert block validated against a
+  recording whose skill sources have since moved. Re-record when the drift is real.
 - Or read `results[].staleness[].class` yourself and decide.
 
 Both flags realize the gate as failing assertions, so the verdict / `ok` / exit code stay consistent with the

package/.claude/skills/cowork-harness/references/critique.md

@@ -1,6 +1,6 @@
 # Critique — the facts a plugin install can't otherwise reach
 
-Tracks `cowork-harness 1.24.0` (baseline `desktop-1.32352.0`). This is **not** a trim of the full
+Tracks `cowork-harness 2.0.0` (baseline `desktop-1.34493.1`). This is **not** a trim of the full
 [`docs/critique.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/critique.md) (repo-only —
 flags, cost, reproduction discipline, known limitations all live there). This file covers exactly what a
 plugin install cannot otherwise discover: the run-dir artifact a harvester actually reads, the report's

package/.claude/skills/cowork-harness/references/fidelity-and-answers.md

@@ -1,6 +1,6 @@
 # Fidelity tiers & answer paths
 
-Self-contained reference. Tracks `cowork-harness 1.24.0` (baseline `desktop-1.32352.0`).
+Self-contained reference. Tracks `cowork-harness 2.0.0` (baseline `desktop-1.34493.1`).
 
 ## Fidelity tiers (`fidelity:` in the scenario)
 

package/.claude/skills/cowork-harness/references/scenario-schema.md

@@ -1,7 +1,7 @@
 # Scenario & session schema, assertion catalog, web_fetch, full gotchas
 
-Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 1.24.0`
-(baseline `desktop-1.32352.0`). If your checkout is newer, prefer the live [`docs/scenario.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/scenario.md),
+Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 2.0.0`
+(baseline `desktop-1.34493.1`). If your checkout is newer, prefer the live [`docs/scenario.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/scenario.md),
 [`docs/session.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/session.md), and `SPEC.md`.
 
 **Minimal scenario** — `prompt` is the only required field:
@@ -107,7 +107,7 @@ form a relocatable bundle. `~` expands to home.
 > references relative to **its own** directory at record time (`scenario.session` and
 > `scenarioSource`), so moving it afterwards — a different `--out`, a
 > `git mv`, a copy into another repo — leaves them unresolvable and `verify-cassettes` reports
-> `unverifiable-skill` ("can't verify ⇒ not green", exit 3) until you re-record at the new location.
+> `unverifiable-skill` ("can't verify ⇒ not green", exit 3) until you re-record at the new location, or pass `--session <file>`.
 > Decide where a cassette will live *before* you record it.
 
 ## Session YAML

package/.claude/skills/cowork-harness/references/task-recipes.md

@@ -2,7 +2,7 @@
 
 Each recipe composes facts that live scattered across SKILL.md and the other references into one
 decision path. Every one answers a question a real fleet owner had to work out the hard way.
-Tracks `cowork-harness 1.24.0` (baseline `desktop-1.32352.0`), same as SKILL.md's front-matter. Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
+Tracks `cowork-harness 2.0.0` (baseline `desktop-1.34493.1`), same as SKILL.md's front-matter. Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
 Recipe 3's `init-redact` shipped in 0.24.0 and are part of the current feature set — no version gate
 needed if your CLI meets SKILL.md's version floor.
 
@@ -182,7 +182,7 @@ degrade the advice. It is real work to calibrate; these steps are the traps that
    **that one invocation**, so the agent answers from its own priors, and stamps the result
    `ablated: true`. It is the **control arm only** — run the same prompt again *without* the flag for
    the treatment arm; `--ablate-skill --repeat 5` gives you 5 control runs and 0 treatment runs, not an
-   A/B. (Inspecting an organically not-invoked rep works too, but is not a substitute: that rep may
+   A/B — and the rollup labels it `PASS [ABLATED — control arm]` so you cannot bank it as one. (Inspecting an organically not-invoked rep works too, but is not a substitute: that rep may
    differ for other reasons.) If the answer still scores high without the skill, that claim is
    answerable from priors and tests the model, not your skill — strengthen it (a skill-specific fact) or
    drop it. Everything past "run both arms" — scrubbing giveaways, shuffling, judging blind, unblinding