cowork-harness 0.30.0 → 0.32.0

package/.claude/skills/cowork-harness/SKILL.md

@@ -3,8 +3,8 @@ name: cowork-harness
 description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
 metadata:
   author: cowork-harness
-  version: 0.30.0
-  tracks-harness: cowork-harness 0.30.0 (baseline desktop-1.20186.1)
+  version: 0.32.0
+  tracks-harness: cowork-harness 0.32.0 (baseline desktop-1.20186.1)
 ---
 
 # cowork-harness
@@ -22,8 +22,8 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
 allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
 the highest-value part. Read it.
 
-> **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.30.0` (baseline
-> `desktop-1.20186.0`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
+> **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.32.0` (baseline
+> `desktop-1.20186.1`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
 > `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
 
 ## Preflight — make sure the harness can actually run
@@ -38,7 +38,7 @@ cowork-harness skill ./my-skill "do X"      # run the skill once against the sta
 Before the first command, confirm the CLI is reachable and **fail loud** (never fake a pass) when a tier's dependencies are missing:
 
 - **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
-- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.30.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=0.30.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=0.30.0"`. **Pin `@>=0.30.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published. (≥ 0.30.0 is what gates the commands/assertions this skill teaches: `assertions --list`, `scaffold <run-id>`, `trace --view dispatches`, `artifact_json` incl. the `in:` operator (passes when the resolved value deep-equals one of the listed members — value ∈ your list, not the reverse), `verify-cassettes`, batch `record <dir>`/`--rerecord-stale`, `record --concurrency <N>`, record-time redaction, multiSelect/`answer:`, `verify-run` answer-coverage, `record --max-artifact-bytes`, live record-time deciders, `verify-cassettes --allow-domain`/`--allow-email`/`--allow-path`/`--allow-patterns-file` (`path` — local absolute filesystem paths — is the scanner's 4th class, new in 0.21.0; `--allow-patterns-file <path>` is a FILE of patterns, one regex per line — not a path to allow, unlike `--allow <regex>`), scenario `skills:` staleness scoping with `COWORK_HARNESS_AGENT_SCOPE=skill`, `chat --plugin`, `/help` in the REPL, `hostloop`'s native host/VM process split with its `allow_host_writes:` consent field, `computer_links_resolve` (new in 0.22.0), `semantic_matches` (an LLM judge grades a fixed `rubric` of claims against the run's answer — **live-only**, so it is evidence-unavailable / skipped-loud on replay, never a vacuous pass; new in 0.27.0), **glob-matched** `tool_called`/`tool_not_called`/`subagent_tool_used`/`subagent_tool_absent` (a pattern like `mcp__workspace__*` matches any tool in the family; exact names match exactly; new in 0.28.0), the five path-gate assertion keys `no_vm_path_file_op`/`vm_path_denied`/`path_denied`/`no_path_denied`/`subagent_file_write`, the session-level `agent_env` knob, and resolved sub-agent identity on dispatch records (`resolvedAgentType`/`resolvedModel`; new in 0.30.0).)
+- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.32.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=0.32.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=0.32.0"`. **Pin `@>=0.32.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published. (≥ 0.32.0 is what gates the commands/assertions this skill teaches: `assertions --list`, `scaffold <run-id>`, `trace --view dispatches`, `artifact_json` incl. the `in:` operator (passes when the resolved value deep-equals one of the listed members — value ∈ your list, not the reverse), `verify-cassettes`, batch `record <dir>`/`--rerecord-stale`, `record --concurrency <N>`, record-time redaction, multiSelect/`answer:`, `verify-run` answer-coverage, `record --max-artifact-bytes`, live record-time deciders, `verify-cassettes --allow-domain`/`--allow-email`/`--allow-path`/`--allow-patterns-file` (`path` — local absolute filesystem paths — is the scanner's 4th class, new in 0.21.0; `--allow-patterns-file <path>` is a FILE of patterns, one regex per line — not a path to allow, unlike `--allow <regex>`), scenario `skills:` staleness scoping with `COWORK_HARNESS_AGENT_SCOPE=skill`, `chat --plugin`, `/help` in the REPL, `hostloop`'s native host/VM process split with its `allow_host_writes:` consent field, `computer_links_resolve` (new in 0.22.0), `semantic_matches` (an LLM judge grades a fixed `rubric` of claims against the run's answer — **live-only**, so it is evidence-unavailable / skipped-loud on replay, never a vacuous pass; new in 0.27.0), **glob-matched** `tool_called`/`tool_not_called`/`subagent_tool_used`/`subagent_tool_absent` (a pattern like `mcp__workspace__*` matches any tool in the family; exact names match exactly; new in 0.28.0), the five path-gate assertion keys `no_vm_path_file_op`/`vm_path_denied`/`path_denied`/`no_path_denied`/`subagent_file_write`, the session-level `agent_env` knob, and resolved sub-agent identity on dispatch records (`resolvedAgentType`/`resolvedModel`; new in 0.30.0), the `lint-skill` (static host-loop footgun + `subagent_type` resolution linter) / `analyze-skill` (advisory `/sessions`-path static scan, `--strict`, `analyze-skill: ignore` marker) / `probe-dispatch` (single-dispatch mechanics probe) commands, `status --latest-for` (resolve a scenario's newest run dir by run time, not directory mtime), the `subagent_dispatch_healthy` composite assertion, and the persisted `result.json` fields `verdict`, `subagents[].referencesRead`/`subagents[].reasoning`, plus the `toolCounts`/`toolErrors`/`toolDurations` shape distinction (all new in 0.31.0), `analyze-skill`'s directory scan now covering a skill/plugin's full contract surface (recursive `agents/`/`references/`/`commands/`, plugin-root-aware, symlink-following) with line/block-scoped `analyze-skill: ignore-next-line`/`ignore-start`/`ignore-end` markers and multi-path/glob input, and `lint-skill`'s provable in-plugin `subagent_type` typo now a WARN that gates under `--strict` (new in 0.32.0).)
 - **Agent binary (sandboxed live tiers — `container`/`microvm`/`hostloop`/`cowork`).** The staged Claude Code agent is **bind-mounted** from a local Claude Desktop install, or point `COWORK_AGENT_BINARY` at a `claude-code-vm/<ver>/claude` ELF. Nothing is bundled. `protocol` (L0) and `replay` need no staged agent; for the sandboxed tiers, no agent → no run; report that, don't skip silently.
 - **Docker / Lima.** Only `--fidelity protocol` (L0) runs without them. `container` / `microvm` / `hostloop` / `cowork` need Docker (Lima for L2). If they're absent, drop to `--fidelity protocol` and **say so** — a green that never exercised the sandbox is not a sandbox pass.
 - **Auth.** `CLAUDE_CODE_OAUTH_TOKEN` (preferred) or `ANTHROPIC_API_KEY`, via env or `.env`. Minting an OAuth token needs the **`claude` CLI** (`npm i -g @anthropic-ai/claude-code`, then `claude setup-token`).
@@ -63,13 +63,14 @@ reproducible regression (Part II), and **debug** a run that misbehaved or greene
   the path, `trace <run-id>` finds it). **Localize the failure post-hoc** from that evidence:
   `cowork-harness trace <run-dir>`'s views + the emitted `result.json` to see what the run actually did,
   then `verify-run` to re-check a suspect assertion — all token-free, no Docker, no re-record. This is
-  the loop 0.30.0's observability is built for; the *Triage* and *Inspecting a run's observability
+  the loop 0.32.0's observability is built for; the *Triage* and *Inspecting a run's observability
   output* sections in **Part III — Debug** are the detail (the fuller human-facing map lives in
   `docs/debugging.md` — repo-only, not shipped with the installed skill).
 - **Multi-turn / interactive reproduction** → `cowork-harness chat` (interactive; gates answered at the
   TTY, **not** an asserted test — see *Debugging with `chat`* in **Part III — Debug**).
 
 Full command set: `skill · run · chat · record · replay · verify-cassettes · rehash · prune · lint ·
+lint-skill · analyze-skill · probe-dispatch ·
 verify-run · trace · inspect · diff · stats · decide · gates · answer · scaffold · assertions --list · sync ·
 list · boundary-check · status · vm <init|status|delete|prune> · doctor · init-redact`. Always check `cowork-harness <cmd> --help`.
 
@@ -421,9 +422,10 @@ decide which assertions from *Assertions: two orthogonal axes* are worth adding)
   the current list rather than relying on a fixed enumeration here.
 - **`cowork-harness stats [--metric <m>]`** — aggregate across the run index: `cost`, `duration`,
   `tokens`, `cache-tokens`, `model-cost`, `turns`, `pass-rate`.
-- **`result.json` carries the raw fields** the assertions read: `toolDurations`, `models`, `toolErrors`,
+- **`result.json` carries the raw fields** the assertions read: `verdict`, `toolDurations`, `models`, `toolErrors`,
   `redundantToolCalls`, `modelUsage`, `thinking`, `skillActivity`, `subagents[]` (prompt/`dispatchModel`/
-  `resolvedModel`/output/`attributedSkillId`, `outputTruncated`), `context` (tools/mcpServers/availableSkills), `tasks`,
+  `resolvedModel`/output/`attributedSkillId`, `outputTruncated`, `referencesRead`, `reasoning`/`reasoningElided`),
+  `context` (tools/mcpServers/availableSkills), `tasks`,
   `workspaceFiles`, `presentedFiles`, `hookEvents`, `mcpErrors`, `contextEvents`, `resources`
   (`probeFailures` distinguishes a failed sample from a tier that was never sampleable). Provenance/
   evidence-health fields: `command` (`run`/`skill`/`record`/`chat`/`replay` — finer than `mode`),
@@ -431,8 +433,11 @@ decide which assertions from *Assertions: two orthogonal axes* are worth adding)
   `bySource` histogram), `evidenceErrors` (dropped/malformed telemetry lines per stream, incl.
   `egressParse`), `fingerprint.frozen` (replay only — marks the shown staleness fingerprint as the
   cassette's record-time value, not a fresh recompute), and `assertTextTruncated` (companion to
-  `outputTruncated` on a matched tool result). (Full per-field semantics: the README's "Observability
-  fields" section — repo-only; `schema/run-result.json` is the machine source.)
+  `outputTruncated` on a matched tool result). Three separately-shaped rollups, easy to conflate in a
+  `jq` recipe: `toolCounts` is a flat `{tool: number}` call-count map, `toolErrors` is
+  `{tool: {calls, errors}}`, and `toolDurations` is `{tool: {calls, totalMs, maxMs}}`. (Full per-field
+  semantics: the README's "Observability fields" section — repo-only; `schema/run-result.json` is the
+  machine source.)
 - **Opaque failure?** A failed run also records **`errorSource`** (where the failure originated) and
   **`stderrLogPath`** (the captured agent stderr) — read those and `trace <run-dir>` *before* re-running;
   a re-record rarely tells you more than the captured stderr already does. Also check
@@ -674,8 +679,9 @@ assertion/replay-relevant ones).
 - `references/fidelity-and-answers.md` — fidelity tiers, answer paths, the determinism contract.
 - `references/ci-recipe.md` — the packaged GitHub Action, replay-vs-live lane split, and the four-stage
   GitHub Actions pipeline.
-- `scripts/scenario.py` — `scaffold` a valid scenario skeleton and `lint` scenarios for the
-  no-silent-false-green invariants (both usable as CI steps).
+- `scripts/scenario.py` — `scaffold` a valid scenario skeleton, `lint` scenarios for the
+  no-silent-false-green invariants (both usable as CI steps), and `resolve-agent-types <plugin-dir>`
+  (validates a pinned `subagent_type` against the plugin's own `plugin.json` + `agents/*.md`).
 - Checking a background run's status without `ps aux` — covered in *Checking whether a background run is
   alive* (Part II) above; the fuller recipe is in `docs/run-status.md` (repo-only, not shipped with the
   installed skill).

package/.claude/skills/cowork-harness/references/ci-recipe.md

@@ -1,6 +1,6 @@
 # CI recipe — replay vs live lanes
 
-Self-contained reference. Tracks `cowork-harness 0.30.0` (baseline `desktop-1.20186.0`).
+Self-contained reference. Tracks `cowork-harness 0.32.0` (baseline `desktop-1.20186.1`).
 
 **Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
 job-summary reporter (verdict table, staleness findings, cost/turns when available):
@@ -54,7 +54,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
 GitHub-hosted runners, no token/Docker/agent:
 
 ```yaml
-- run: npm i -g "cowork-harness@>=0.30.0"
+- run: npm i -g "cowork-harness@>=0.32.0"
 - run: cowork-harness lint scenarios/*.yaml          # no silent false-greens
 - run: cowork-harness verify-cassettes cassettes/    # privacy + staleness
 - run: cowork-harness replay cassettes/              # token-free content/structure
@@ -139,13 +139,19 @@ dollar figures). In a skill repo these cassettes get **committed**. So:
   universal net (container-tier recordings can trip it too).
 - **Always-on scan gate** — `verify-cassettes` flags email / currency / bare-domain / local-path /
   machine-inventory matches it finds in the committed cassettes and **exits non-zero**, so "no leak" is
-  a gate, not discipline.
+  a gate, not discipline. Non-zero is not one thing, though: exit `1` means verification RAN and found a
+  real finding (a PII match, a genuine staleness drift, or scenario-prompt drift); exit `3` means
+  verification could NOT complete (an `unverifiable-*`-class staleness finding, a cassette written by a
+  newer harness than this one understands, or a malformed/unreadable cassette). A plain `|| true` or `[
+  $? -ne 0 ]` tripwire treats both the same — if you need to tell "the gate caught something" apart from
+  "the gate couldn't run", branch on the exit code (or parse `--output-format json`'s per-file
+  `findings`/`staleness` vs `unverifiable`/`version`/`error` buckets).
   Suppress synthetic / public reference names (NVCA, Cooley GO, …) with `--allow <regex>`. (Multi-word
   proper names are NOT a default class — too noisy to gate on; add a pattern via config if your corpus
   needs it.)
 
 ```bash
-cowork-harness verify-cassettes cassettes/                       # privacy scan + staleness, exit 1 on a finding
+cowork-harness verify-cassettes cassettes/                       # privacy scan + staleness — exit 1 = verified & failed, exit 3 = could not verify
 cowork-harness verify-cassettes cassettes/ --allow 'NVCA|Cooley GO|Acme'
 cowork-harness verify-cassettes cassettes/ --skip-privacy        # staleness only (skip the privacy scan); both run by default
 ```
@@ -188,7 +194,7 @@ jobs:
         with: { node-version: '20' }
       - uses: actions/setup-python@v5
         with: { python-version: '3.x' }                                       # python3 only — PyYAML is bundled with the linter
-      - run: npm i -g "cowork-harness@>=0.30.0"
+      - run: npm i -g "cowork-harness@>=0.32.0"
       - run: cowork-harness lint scenarios/*.yaml                              # no-silent-false-green (needs python3; PyYAML bundled)
       - run: cowork-harness verify-cassettes cassettes/ --output-format json   # privacy + staleness gate
       - run: cowork-harness replay cassettes/ --output-format json             # token-free content/structure
@@ -217,7 +223,7 @@ jobs:
             echo "live=true" >> "$GITHUB_OUTPUT"
           fi
       - if: steps.guard.outputs.live == 'true'
-        run: npm i -g "cowork-harness@>=0.30.0"
+        run: npm i -g "cowork-harness@>=0.32.0"
       - if: steps.guard.outputs.live == 'true'
         run: cowork-harness run scenarios/ --output-format json
         env:
@@ -245,9 +251,12 @@ fails or a run errors, so a plain `cowork-harness run scenarios/` is already CI-
 JSON.
 
 `verify-cassettes` emits its **own** envelope (`{command, ok, coverage, results[]}` with per-file
-`findings`/`staleness`/`notes`/`version`/`error`), published as `schema/verify-cassettes.json` in the
-npm package. Both envelope schemas are covered 1.0 contract surfaces (SPEC §12) — parse the JSON, not
-the human-readable text (which is explicitly NOT stable).
+`findings`/`staleness`/`unverifiable`/`notes`/`version`/`error`), published as
+`schema/verify-cassettes.json` in the npm package. `ok:false` doesn't say *why* — read the buckets, or
+the exit code (`1` = `findings`/`staleness`/`scenarioDrift` populated, a real problem verified & found;
+`3` = `unverifiable`/`version`/`error` populated, verification could not complete; a real finding wins
+`1` if both are present). Both envelope schemas are covered 1.0 contract surfaces (SPEC §12) — parse the
+JSON, not the human-readable text (which is explicitly NOT stable).
 
 A run writes to `~/.cowork-harness/runs/<name>/<sessionId>/` by default — outside any working tree. In CI,
 set `COWORK_HARNESS_RUNS_DIR` (or pass `--run-dir`) to a workspace-relative path (e.g. `runs`) so an
@@ -280,8 +289,11 @@ does **not** imply the recording is still valid. Each replay result carries `sta
 
 (A pre-`effectiveFidelity` cassette with an **explicit** tier is statically knowable — it passes the tier
 check with a non-failing informational note in the `verify-cassettes` envelope's per-file `notes[]`, a
-`·`-prefixed row in text output. On `verify-cassettes` every staleness *finding* above is a hard fail —
-the gate is class-blind; notes never fail it.)
+`·`-prefixed row in text output. On `verify-cassettes` every staleness *finding* above still fails the
+gate (`ok:false`) — but it's no longer class-blind on the EXIT CODE: a `baseline`/`skill`/`shared-root`/
+`format`/`resolved-tier` class lands in the envelope's `staleness[]` (verified & failed — exit `1`),
+while an `unverifiable-*` class lands in `unverifiable[]` (could not verify — exit `3`). Notes never
+fail it either way.)
 
 To gate in CI, pick the severity you want:
 

package/.claude/skills/cowork-harness/references/fidelity-and-answers.md

@@ -1,6 +1,6 @@
 # Fidelity tiers & answer paths
 
-Self-contained reference. Tracks `cowork-harness 0.30.0` (baseline `desktop-1.20186.0`).
+Self-contained reference. Tracks `cowork-harness 0.32.0` (baseline `desktop-1.20186.1`).
 
 ## Fidelity tiers (`fidelity:` in the scenario)
 

package/.claude/skills/cowork-harness/references/scenario-schema.md

@@ -1,7 +1,7 @@
 # Scenario & session schema, assertion catalog, web_fetch, full gotchas
 
-Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.30.0`
-(baseline `desktop-1.20186.0`). If your checkout is newer, prefer the live `docs/scenario.md`,
+Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.32.0`
+(baseline `desktop-1.20186.1`). If your checkout is newer, prefer the live `docs/scenario.md`,
 `docs/session.md`, and `SPEC.md`.
 
 **Minimal scenario** — `prompt` is the only required field:
@@ -262,6 +262,7 @@ same set live from the schema.
 | `subagent_tool_absent: <glob>` | no sub-agent used a tool matching this glob (same rejection) |
 | `no_vm_path_file_op: true` | **`fidelity: hostloop` only** — NO gated file tool attempted a `/sessions`(-prefixed) path (`RunResult.fileToolAttempts`) — content-class, replay-checkable without `controlOut`; any other tier FAILS "cannot verify" (`/sessions/...` is valid there). **Only `true` is valid** |
 | `subagent_file_write: {path?, path_suffix?, tool?}` | a sub-agent-origin write attempt whose raw path equals `path` (exact) or ends with `path_suffix` has a paired non-error tool_result — the causal half of a delivery probe; requires one of `path`/`path_suffix`; `tool` defaults to Write/Edit/MultiEdit; content-class; tier-agnostic |
+| `subagent_dispatch_healthy: {type?, delivered?, path?, path_suffix?, no_vm_paths?}` | **`fidelity: hostloop` only** — composite: selects dispatch(es) via `type` (same matching as `subagent_dispatched`; omit to require every dispatch) and, for EACH selected dispatch, checks it (not just any sub-agent) delivered a paired non-error write (`delivered`, default true — narrowed by `path`/`path_suffix`, same exact-vs-suffix precedence as `subagent_file_write`) and made no `/sessions` VM-path attempt (`no_vm_paths`, default true) — both scoped to that dispatch's OWN `parentToolUseId`, the per-dispatch correlation `subagent_file_write` (which matches ANY sub-agent write) cannot express; a `type` that matches no dispatch FAILS; content-class (`RunResult.fileToolAttempts` + `RunResult.toolResults`); any non-hostloop tier FAILS "cannot verify" |
 | `subagent_dispatched: <regex>` | a sub-agent whose `dispatchAgentType`, binary-*resolved* `resolvedAgentType`, **or dispatch description** matches |
 | `subagent_declared_but_unused: <Tool>` | a sub-agent declared the tool but never used **that** tool (even if it used others) |
 | `subagent_output_contains: {match?, contains}` | a dispatched sub-agent's own output contains the substring `contains` — `match` (optional regex over `dispatchAgentType`/`resolvedAgentType`/`description`) narrows to specific dispatch(es); omitted, checks whether ANY dispatch's output contains it (existence check, not "all"); a miss against an output that was **truncated at the assert cap** reports evidence-unavailable instead of a proven absence — the substring could lie past the cut |

package/.claude/skills/cowork-harness/references/task-recipes.md

@@ -2,7 +2,7 @@
 
 Each recipe composes facts that live scattered across SKILL.md and the other references into one
 decision path. Every one answers a question a real fleet owner had to work out the hard way. Facts track the harness version in SKILL.md's
-front-matter (currently 0.29.0). Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
+front-matter (currently 0.32.0). Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
 Recipe 3's `init-redact` shipped in 0.24.0 and are part of the current feature set — no version gate
 needed if your CLI meets SKILL.md's version floor.
 

package/.claude/skills/cowork-harness/scripts/assertion-keys.json

@@ -45,6 +45,7 @@
     "skill_tool_used",
     "skill_triggered",
     "subagent_declared_but_unused",
+    "subagent_dispatch_healthy",
     "subagent_dispatched",
     "subagent_file_write",
     "subagent_output_contains",