cowork-harness 0.26.0 → 0.28.0

package/.claude/skills/cowork-harness/SKILL.md

@@ -3,8 +3,8 @@ name: cowork-harness
 description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
 metadata:
   author: cowork-harness
-  version: 0.26.0
-  tracks-harness: cowork-harness 0.26.0 (baseline desktop-1.18286.0)
+  version: 0.28.0
+  tracks-harness: cowork-harness 0.28.0 (baseline desktop-1.19367.0)
 ---
 
 # cowork-harness
@@ -22,11 +22,11 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
 allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
 the highest-value part. Read it.
 
-> **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.26.0` (baseline
-> `desktop-1.18286.0`). If your checkout is newer, prefer the live `--help`, `SPEC.md`, and
-> `docs/*.md` over this snapshot, and re-run the bundled linter.
+> **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.28.0` (baseline
+> `desktop-1.19367.0`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
+> `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
 
-## 0. Preflight — make sure the harness can actually run
+## Preflight — make sure the harness can actually run
 
 The 10-second inner loop, once the CLI is on PATH:
 
@@ -38,31 +38,54 @@ cowork-harness skill ./my-skill "do X"      # run the skill once against the sta
 Before the first command, confirm the CLI is reachable and **fail loud** (never fake a pass) when a tier's dependencies are missing:
 
 - **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
-- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.26.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=0.26.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=0.26.0"`. **Pin `@>=0.26.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published. (≥ 0.26.0 is what gates the commands/assertions this skill teaches: `assertions --list`, `scaffold <run-id>`, `trace --view dispatches`, `artifact_json` incl. the `in:` operator, `verify-cassettes`, batch `record <dir>`/`--rerecord-stale`, `record --concurrency <N>`, record-time redaction, multiSelect/`answer:`, `verify-run` answer-coverage, `record --max-artifact-bytes`, live record-time deciders, `verify-cassettes --allow-domain`/`--allow-email`/`--allow-path`/`--allow-patterns-file` (`path` — local absolute filesystem paths — is the scanner's 4th class, new in 0.21.0; `--allow-patterns-file <path>` is a FILE of patterns, one regex per line — not a path to allow, unlike `--allow <regex>`), scenario `skills:` staleness scoping with `COWORK_HARNESS_AGENT_SCOPE=skill`, `chat --plugin`, `/help` in the REPL, `hostloop`'s native host/VM process split with its `allow_host_writes:` consent field, and `computer_links_resolve` (new in 0.22.0).)
+- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.28.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=0.28.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=0.28.0"`. **Pin `@>=0.28.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published. (≥ 0.28.0 is what gates the commands/assertions this skill teaches: `assertions --list`, `scaffold <run-id>`, `trace --view dispatches`, `artifact_json` incl. the `in:` operator (passes when the resolved value deep-equals one of the listed members — value ∈ your list, not the reverse), `verify-cassettes`, batch `record <dir>`/`--rerecord-stale`, `record --concurrency <N>`, record-time redaction, multiSelect/`answer:`, `verify-run` answer-coverage, `record --max-artifact-bytes`, live record-time deciders, `verify-cassettes --allow-domain`/`--allow-email`/`--allow-path`/`--allow-patterns-file` (`path` — local absolute filesystem paths — is the scanner's 4th class, new in 0.21.0; `--allow-patterns-file <path>` is a FILE of patterns, one regex per line — not a path to allow, unlike `--allow <regex>`), scenario `skills:` staleness scoping with `COWORK_HARNESS_AGENT_SCOPE=skill`, `chat --plugin`, `/help` in the REPL, `hostloop`'s native host/VM process split with its `allow_host_writes:` consent field, `computer_links_resolve` (new in 0.22.0), and `semantic_matches` (an LLM judge grades a fixed `rubric` of claims against the run's answer — **live-only**, so it is evidence-unavailable / skipped-loud on replay, never a vacuous pass; new in 0.27.0), and **glob-matched** `tool_called`/`tool_not_called`/`subagent_tool_used`/`subagent_tool_absent` (a pattern like `mcp__workspace__*` matches any tool in the family; exact names match exactly; new in 0.28.0).)
 - **Agent binary (sandboxed live tiers — `container`/`microvm`/`hostloop`/`cowork`).** The staged Claude Code agent is **bind-mounted** from a local Claude Desktop install, or point `COWORK_AGENT_BINARY` at a `claude-code-vm/<ver>/claude` ELF. Nothing is bundled. `protocol` (L0) and `replay` need no staged agent; for the sandboxed tiers, no agent → no run; report that, don't skip silently.
 - **Docker / Lima.** Only `--fidelity protocol` (L0) runs without them. `container` / `microvm` / `hostloop` / `cowork` need Docker (Lima for L2). If they're absent, drop to `--fidelity protocol` and **say so** — a green that never exercised the sandbox is not a sandbox pass.
 - **Auth.** `CLAUDE_CODE_OAUTH_TOKEN` (preferred) or `ANTHROPIC_API_KEY`, via env or `.env`. Minting an OAuth token needs the **`claude` CLI** (`npm i -g @anthropic-ai/claude-code`, then `claude setup-token`).
+- **`--dotenv` is a GLOBAL flag — put it BEFORE the subcommand.** `cowork-harness --dotenv .env record …`, never `cowork-harness record … --dotenv .env`. Every *other* flag is subcommand-level, so muscle memory fights this one; the harness rejects the misplaced form with an exact-fix error, but placing it first avoids the round-trip.
 
-## 1. Pick the loop
+## Orient — the three loops
+
+Everything you do with the harness is one of **three loops**, and the rest of this skill is organized
+into three Parts to match: **author** a scenario (Part I), **run / record / lock** it into a
+reproducible regression (Part II), and **debug** a run that misbehaved or greened when it shouldn't
+(Part III — reachable straight from here in one hop). Pick the loop you're in:
 
 - **"Is it even alive?"** (inner loop) → `cowork-harness skill <folder> "<prompt>"`. Fastest; no
   scenario file.
 - **Repeatable, asserted regression** → author a `scenarios/*.yaml` and run `cowork-harness run`.
   This is the CI-grade path and most of this skill.
-- **Multi-turn debugging** → `cowork-harness chat` (interactive; gates answered at the TTY, **not** an
-  asserted test — see *Debugging with `chat`* in `docs/scenario.md`, or `docs/debugging.md` for the full
-  debugging map).
+- **Regression-test your skill's ANSWER quality** (not just its behavior — does its guidance still lead to
+  correct answers after you edit it?) → author `semantic_matches` scenarios and gate on the per-claim
+  profile. See **Recipe 5** in `references/task-recipes.md` (validity, N≥3, discrimination — the traps).
+- **A run failed — or greened and you don't trust it** (the debugging loop) → don't re-run and hope.
+  The run already wrote its evidence to a **kept run dir** (`~/.cowork-harness/runs/…`; `--keep` prints
+  the path, `trace <run-id>` finds it). **Localize the failure post-hoc** from that evidence:
+  `cowork-harness trace <run-dir>`'s views + the emitted `result.json` to see what the run actually did,
+  then `verify-run` to re-check a suspect assertion — all token-free, no Docker, no re-record. This is
+  the loop 0.28.0's observability is built for; the *Triage* and *Inspecting a run's observability
+  output* sections in **Part III — Debug** are the detail (the fuller human-facing map lives in
+  `docs/debugging.md` — repo-only, not shipped with the installed skill).
+- **Multi-turn / interactive reproduction** → `cowork-harness chat` (interactive; gates answered at the
+  TTY, **not** an asserted test — see *Debugging with `chat`* in **Part III — Debug**).
 
 Full command set: `skill · run · chat · record · replay · verify-cassettes · rehash · prune · lint ·
 verify-run · trace · inspect · diff · stats · decide · gates · answer · scaffold · assertions --list · sync ·
-list · boundary-check · status · vm <init|status|delete|prune> · doctor`. Always check `cowork-harness <cmd> --help`.
+list · boundary-check · status · vm <init|status|delete|prune> · doctor · init-redact`. Always check `cowork-harness <cmd> --help`.
 
 **Two different `scaffold` tools — don't confuse them.** The native `cowork-harness scaffold <run-id>`
 above turns an already-*recorded* run into a scenario (needs a run to exist first). The bundled
-`scripts/scenario.py scaffold --name … --skill …` in §9 builds a scenario from flags alone, no run
-required. Passing §9's flag set to the native command fails with `unknown flag: --name` (exit 2).
+`scripts/scenario.py scaffold --name … --skill …` — see *Scaffold a valid scenario, then lint before
+you push* in **Part I** — builds a scenario from flags alone, no run required. Passing that section's
+flag set to the native command fails with `unknown flag: --name` (exit 2).
+
+## Part I — AUTHOR a scenario
+
+Everything below composes one deterministic, asserted `scenarios/*.yaml`: the session/scenario split,
+how the skill mounts, the fidelity tier, the answer path, the two assertion axes, `web_fetch`
+provenance, and the scaffold/lint tools that keep the YAML honest.
 
-## 2. Two files: session vs scenario
+### Two files: session vs scenario
 
 - **`sessions/*.yaml`** — pre-prompt setup: `model`, mounts (`folders`), and discovery
   (marketplaces / plugins / skills / mcp). One session is reused by many scenarios. A scenario that
@@ -73,7 +96,7 @@ This split matters: release ground truth (`baseline:` / `baselines/`, produced b
 **separate** from authored setup (`session:` / `sessions/`). "profile" is retired vocabulary — do
 not use it. See `references/scenario-schema.md` for every field.
 
-## 3. Discovery: how the skill-under-test gets mounted
+### Discovery: how the skill-under-test gets mounted
 
 The skill is **copied fresh into the sandbox each run**. Wire it via `plugins.local_plugins` +
 `plugins.enabled: [<plugin>@local]` in the session (or `--marketplace` / `--plugin` flags on
@@ -83,13 +106,17 @@ the folder basename (collision-resolved); there is no `to:` override. See `refer
 
 > **`git add` a brand-new skill before testing it.** Inside a git repo the harness stages the
 > **git-tracked** files (the fidelity boundary — real Cowork installs from a repo and sees only committed
-> files). An **all-untracked** skill folder used to mount *empty* and the agent reported "the skill isn't
+> files). *Tracked* means **in the git index** (committed **or** `git add`-staged); the **content** staged
+> is your **working tree**, so an uncommitted edit to an already-tracked file *is* tested — you needn't
+> commit to iterate. Only brand-new (untracked) files must be `git add`-ed to appear. Commit before you
+> record the **locking cassette**, though: real Cowork ships the *committed* tree, so a green on
+> uncommitted edits isn't yet a green on what installs. An **all-untracked** skill folder used to mount *empty* and the agent reported "the skill isn't
 > installed" then did the work itself — a green-looking run where the skill never loaded. That now
 > **hard-fails** (`BoundaryError`, exit 3) naming the dir, and a partially-tracked folder emits a loud
 > `::notice:: [stage]` listing the excluded files. Fix: `git add` the skill, or `COWORK_HARNESS_GITSET=0`
 > to copy untracked files (won't reflect what ships). A folder **outside** any repo is copied raw (no guard).
 
-## 4. Choose a fidelity tier
+### Choose a fidelity tier
 
 | Tier | What it gives you | Use when |
 |---|---|---|
@@ -102,7 +129,7 @@ Set the tier in the **scenario's `fidelity:` field**, not a flag — `run` rejec
 (it's a `skill`/`chat` flag; `run` takes fidelity only from the scenario). See
 `references/fidelity-and-answers.md`.
 
-## 5. Choose an answer path (gates: AskUserQuestion + tool-permission)
+### Choose an answer path (gates: AskUserQuestion + tool-permission)
 
 Default to **deterministic**: scripted `answers:` + `on_unanswered: fail`. Anything that brings a
 live model into answering flags the run `nonDeterministic` — keep those out of deterministic
@@ -131,7 +158,7 @@ anchor. (When label *order* is stable but the text drifts, a positional `choose`
 linter flags positional `choose` as order-dependent, so use it deliberately.) The caution stands: `first`
 *masks* an unanswered gate, so don't use it for a gate you actually need answered a specific way.
 
-### External deciders and the "first" shorthand
+#### External deciders and the "first" shorthand
 
 When using `--decider-cmd` or `--decider-dir`, the helper's output is passed through
 `coerceLabel` **with the "first" shorthand disabled**. This means a helper that returns the literal
@@ -142,22 +169,7 @@ The `"first"` shorthand remains active only for the built-in `--on-unanswered fi
 write an external helper, return a label name or option index — never the bare word `"first"` unless
 your gate actually has a label called `"first"`.
 
-## 5a. Checking whether a background run is alive
-
-Never use `ps aux` to check on a `cowork-harness` run you launched in the background — it only sees
-processes in your OWN PID namespace, which is frequently NOT the harness process's namespace (e.g. when
-you're a sandboxed subagent). An empty `ps aux` match tells you nothing about whether the run is still
-going.
-
-Use **`cowork-harness status <dir> [--follow]`** instead — reads `<outDir>/status.json`, a file the
-harness writes/updates throughout the run's lifecycle (including a crash-safety net for a thrown
-error/`SIGTERM`, AND staleness detection for a hard `SIGKILL`/OOM-kill that no exit handler can catch —
-either way you get `"error"`/`stale` instead of a permanently-trusted `"running"`), so liveness is
-checkable regardless of PID namespace. The harness prints `[status] <outDir>` to stderr as soon as the
-run starts, so capture stderr to get the exact directory. `--follow` fails loud on a timeout/staleness
-rather than hanging forever. See `docs/run-status.md` for the full recipe.
-
-## 6. Assertions: two orthogonal axes
+### Assertions: two orthogonal axes
 
 Conflating these is the **biggest landmine**. An assertion key has two independent properties:
 
@@ -173,11 +185,40 @@ Conflating these is the **biggest landmine**. An assertion key has two independe
 
 Getting Axis B wrong means a check that **does nothing in CI** — the harness warns loudly when it skips
 (an `::warning::` annotation, not a silent no-op — see the Axis B bullet above), and the bundled linter
-catches it before you push — run it (§9).
+catches it before you push — run it (see *Scaffold a valid scenario, then lint before you push* below).
 
 See `references/scenario-schema.md` for the full assertion catalog with each key's replay class.
 
-## 7. web_fetch (fail-closed, two-path)
+#### Which assertion for which question (goal → key)
+
+Beyond the outcome/content keys most scenarios reach for first (`result`, `transcript_*`,
+`file_exists`/`user_visible_artifact`, `artifact_json`), the harness surfaces the agent's *behavior*
+— tool health, sub-agent work, panels, skill attribution, resources — as assertable keys. Reach for
+them by what you're trying to prove:
+
+| You want to check that… | Reach for |
+|---|---|
+| the skill didn't error out of a tool | `tool_no_error: <regex>`, `max_tool_errors: <N>` |
+| it didn't waste repeated identical calls | `max_redundant_tool_calls: <N>` |
+| a deliverable reached the user | `user_visible_artifact: <path>` (+ `no_scratchpad_leak: true` if it delivers via `present_files` — **`container` only**) |
+| a to-do workflow finished | `all_tasks_completed: true`, `task_status: {match, status}` |
+| a skill / connector / tool was **offered** | `skill_available`, `connector_available`, `tool_available` (all `<regex>`) |
+| a skill actually **ran** (or must NOT) | `skill_triggered: <regex>`, `no_skill_triggered: <regex>` |
+| a tool ran **inside** a skill's scope | `skill_tool_used: {skill, tool}` |
+| a sub-agent did the work | `subagent_output_contains: {contains}`, `subagent_dispatched: <regex>`, `dispatch_count_max: <N>` |
+| a pre-existing input wasn't mutated | `input_unmodified: [<glob>]` (live/verify-run; not microvm) |
+| a resource ceiling held | `max_peak_rss_bytes: <N>` (**live-only**) |
+| a hook blocked / didn't block a tool | `hook_blocked: <regex>`, `no_hook_blocked: true` (replay needs a `controlOut` cassette) |
+| every MCP round-trip succeeded | `no_mcp_error: true` (**live-only**) |
+| a context compaction happened | `compaction_occurred: true` |
+
+Every one of these still obeys the two axes above — several are live-only or need a `controlOut`
+cassette on replay, so check the catalog's replay class before putting one on a PR gate.
+`cowork-harness assertions --list` prints the full, always-current key set with one-line semantics
+straight from the schema — treat it (and the catalog) as the source of truth; this map is a
+goal-oriented index into it, not a second catalog.
+
+### web_fetch (fail-closed, two-path)
 
 `web_fetch` behaves unlike `curl`. A URL is gated by **provenance**, not the egress allowlist:
 
@@ -193,7 +234,47 @@ See `references/scenario-schema.md` for the full assertion catalog with each key
 Surprise to remember: adding a host to `egress.extra_allow` is a **no-op** for a provenanced fetch.
 Full model in `references/scenario-schema.md`.
 
-## 8. Run, then lock determinism
+### Scaffold a valid scenario, then lint before you push
+
+Don't hand-write the YAML from memory — that's how invented keys (`assertions:` vs `assert:`,
+`json_file`, `answer_policy`) creep in. Start from the bundled generator, which emits the
+known-good skeleton (right tier, scripted `answers:` + `on_unanswered: fail`, content assertions
+separated from live-only ones, one concern per item) and **self-lints its own output**:
+
+```bash
+S="${CLAUDE_PLUGIN_ROOT}/scripts/scenario.py"
+# Working from a repo checkout instead of an installed plugin?
+# S=".claude/skills/cowork-harness/scripts/scenario.py"
+python3 "$S" scaffold --name report-check --skill ./skills/report-gen \
+  --prompt "Generate the weekly report to outputs/report.md." \
+  --content 'weekly report' --artifact outputs/report.md \
+  --egress-allowed api.weather.example.com --out scenarios/report-check.yaml
+```
+
+Then lint every scenario — it encodes the no-silent-false-green invariants. Use the CLI wrapper
+`cowork-harness lint` (it runs the same bundled `scenario.py lint`):
+
+```bash
+cowork-harness lint scenarios/*.yaml
+```
+
+`lint` flags: filesystem/egress-only assertions on a `replay` gate (silent no-op), bad regex
+quoting, an egress assert on `protocol` fidelity, `transcript_no_host_path` on `hostloop`/`protocol`
+(ERROR — fails by design at those tiers; WARN on `fidelity: cowork`, whose tier resolves per the
+baseline's host-loop gate), non-empty `requires_capabilities` on `protocol` without
+`allow_missing_capability` (ERROR — the capability probe can't run there, so the run hard-fails as
+unverifiable), a `controlOut`-gated key on a non-`controlOut` replay, mixed-class assertion items,
+and hallucinated schema (`assertions:` vs `assert:`, unknown keys). Exit code is non-zero on errors
+(CI-friendly). `scaffold` auto-upgrades the tier if you ask for egress on `protocol`, so it never
+emits a scenario `lint` would reject.
+
+## Part II — RUN, RECORD & LOCK
+
+You have an authored scenario. This Part runs it, reads the verdict, locks it into a
+byte-deterministic cassette, checks a background run's liveness, and places the assertions in the
+right CI lane.
+
+### Run, then lock determinism
 
 Read the verdict and the inline failing transcript. To pin a flaky-because-stochastic gate, paste
 the echoed `--answer "<q>=<choice>"` footer lines back into the scenario's `answers:` for a
@@ -218,7 +299,7 @@ launched from a repo root never drops sensitive skill inputs/outputs into it. Pa
 `COWORK_HARNESS_RUNS_DIR`) to relocate; in CI point it at a workspace path so an artifact-upload step can
 collect the runs.
 
-### Validate a skill against real documents (not a cassette)
+#### Validate a skill against real documents (not a cassette)
 
 The loops above build **deterministic regressions**. A different job — drive a skill against *real* input
 documents to judge whether it actually does the work (extraction, analysis), with no intent to record a
@@ -254,7 +335,7 @@ cassette — has its own recipe:
    proceeds. Rebuild with `--build-arg COWORK_FULL_PARITY=1` and point `COWORK_AGENT_IMAGE` at it for those
    skills.
 
-### Interpreting verdict signals
+#### Interpreting verdict signals
 
 The run verdict may include `WARN`-severity signals in addition to pass/fail. One to watch for:
 
@@ -263,51 +344,111 @@ The run verdict may include `WARN`-severity signals in addition to pass/fail. On
   the run can still green. If you see it, fix the asset path — a green with a missing asset is
   not a valid pass.
 
-## 9. Scaffold a valid scenario, lint before you push, then place in CI
-
-Don't hand-write the YAML from memory — that's how invented keys (`assertions:` vs `assert:`,
-`json_file`, `answer_policy`) creep in. Start from the bundled generator, which emits the
-known-good skeleton (right tier, scripted `answers:` + `on_unanswered: fail`, content assertions
-separated from live-only ones, one concern per item) and **self-lints its own output**:
+### Checking whether a background run is alive
 
-```bash
-S="${CLAUDE_PLUGIN_ROOT}/scripts/scenario.py"
-# Working from a repo checkout instead of an installed plugin?
-# S=".claude/skills/cowork-harness/scripts/scenario.py"
-python3 "$S" scaffold --name report-check --skill ./skills/report-gen \
-  --prompt "Generate the weekly report to outputs/report.md." \
-  --content 'weekly report' --artifact outputs/report.md \
-  --egress-allowed api.weather.example.com --out scenarios/report-check.yaml
-```
+Never use `ps aux` to check on a `cowork-harness` run you launched in the background — it only sees
+processes in your OWN PID namespace, which is frequently NOT the harness process's namespace (e.g. when
+you're a sandboxed subagent). An empty `ps aux` match tells you nothing about whether the run is still
+going.
 
-Then lint every scenario — it encodes the no-silent-false-green invariants. Use the CLI wrapper
-`cowork-harness lint` (it runs the same bundled `scenario.py lint`):
+Use **`cowork-harness status <dir> [--follow]`** instead — reads `<outDir>/status.json`, a file the
+harness writes/updates throughout the run's lifecycle (including a crash-safety net for a thrown
+error/`SIGTERM`, AND staleness detection for a hard `SIGKILL`/OOM-kill that no exit handler can catch —
+either way you get `"error"`/`stale` instead of a permanently-trusted `"running"`), so liveness is
+checkable regardless of PID namespace. The harness prints `[status] <outDir>` to stderr as soon as the
+run starts, so capture stderr to get the exact directory. `--follow` fails loud on a timeout/staleness
+rather than hanging forever. (Fuller recipe in `docs/run-status.md` — repo-only, not in the installed
+payload; `cowork-harness status --help` has the flags.)
 
-```bash
-cowork-harness lint scenarios/*.yaml
-```
+**A multi-minute `record`/`run` outlives a short-lived wrapper.** Don't launch a long record from a
+subagent that returns before it finishes — the returning agent tears down its process tree and kills the
+in-flight run mid-artifact-write. Run it foreground, or detached from any process that will exit first.
+(The `status.json` liveness above is exactly what surfaces such a teardown as `"error"`/`stale` rather
+than a stuck `"running"`.)
 
-`lint` flags: filesystem/egress-only assertions on a `replay` gate (silent no-op), bad regex
-quoting, an egress assert on `protocol` fidelity, `transcript_no_host_path` on `hostloop`/`protocol`
-(ERROR — fails by design at those tiers; WARN on `fidelity: cowork`, whose tier resolves per the
-baseline's host-loop gate), non-empty `requires_capabilities` on `protocol` without
-`allow_missing_capability` (ERROR — the capability probe can't run there, so the run hard-fails as
-unverifiable), a `controlOut`-gated key on a non-`controlOut` replay, mixed-class assertion items,
-and hallucinated schema (`assertions:` vs `assert:`, unknown keys). Exit code is non-zero on errors
-(CI-friendly). `scaffold` auto-upgrades the tier if you ask for egress on `protocol`, so it never
-emits a scenario `lint` would reject.
+### Place assertions in the right CI lane
 
 CI placement: a **token-free `replay` PR gate** (content/structure only) + a **nightly live `run`**
 (filesystem/egress). Fastest setup: `uses: yaniv-golan/cowork-harness@v1` (a packaged GitHub Action with a
 PR job-summary reporter). See `references/ci-recipe.md` for the Action, the manual step-by-step form, and
 the four-stage pipeline.
 
-## 10. Debugging with `chat`
+## Part III — Debug
+
+A run misbehaved, or greened when you don't trust it. Debugging is a first-class loop, not an
+afterthought: the run already wrote its evidence, so you **localize the failure post-hoc** rather than
+re-run and hope. Start at the triage below, then use the observability output and, when you need to
+reproduce interactively, `chat`.
+
+### Triage — a run misbehaved, or a green looks wrong
+
+<!-- BEGIN triage-canonical -->
+Two situations need different tools — figure out which one you're in first, then reach for the tool
+instead of re-running and hoping. The run already wrote its evidence to a kept run dir (`--keep` prints
+the path; `trace <run-id>` finds it), and every tool below reads that evidence **token-free** — no
+Docker, no re-record.
+
+| Situation | Symptom | Reach for (in order) |
+|---|---|---|
+| **The skill misbehaved** | wrong output, an unexpected gate, a denied tool, an opaque crash | `inspect` — what did it produce? · `trace <run-dir> --view <view>` — what did it actually do (tools, gates, sub-agent tree)? · `verify-run` — re-assert cheaply when only an assertion is wrong · `diff <old-run> <new-run>` — what changed since it worked · `chat` — reproduce it by hand |
+| **A green you don't trust** | an assert that may have tested nothing, a stale cassette, an auto-answered or decided gate | `replay --explain` — the evidence trail behind each *passing* assert · `lint` — assertions on the wrong CI lane / mixed-class keys · `verify-cassettes` — privacy + staleness over committed cassettes · the Gotchas landmine catalog — how a check passes vacuously · `run --repeat N` — did it pass, or pass once? · `stats` — flaky or expensive over time |
+
+A failed run also records `errorSource` (where the failure originated) and `stderrLogPath` (the captured
+agent stderr) — read those before re-running; a re-record rarely tells you more than the captured stderr
+already does.
+<!-- END triage-canonical -->
+
+**Is it your skill's bug, or a known harness gap?** Before deep-debugging a wrong behavior, rule out a
+**deliberate fidelity gap** — the harness intentionally does *not* reproduce a few real-Cowork behaviors,
+so a "bug" you see here that real Cowork also has isn't yours to fix. The tier semantics are in
+`references/fidelity-and-answers.md` (shipped); the specific deltas vs. real Cowork and the sandbox
+boundary model live in `docs/fidelity-gaps.md` / `docs/boundary.md` (repo-only, not in the installed
+payload). If the behavior is on that gap list, it's expected — stop debugging your skill.
+
+### Inspecting a run's observability output
+
+A verdict is only the top of what a run records, and the run dir persists after the verdict
+(`~/.cowork-harness/runs/…`). Beyond pass/fail, every `run`/`skill`/`chat` writes a `result.json` and a
+trace you read back without a re-record — the debugging loop is *localize the failure from that
+already-written evidence*, not re-run-and-hope. Use them to diagnose a failure (and, secondarily, to
+decide which assertions from *Assertions: two orthogonal axes* are worth adding):
+
+- **`cowork-harness trace <run-dir> --view <view>`** — focuses one of the run's rollups (the per-tool
+  call-count/timing table, the sub-agent dispatch tree, the gate lifecycle, the tool/error rollups, …);
+  bare `trace` digests the whole run. The view set is actively being extended — run `trace --help` for
+  the current list rather than relying on a fixed enumeration here.
+- **`cowork-harness stats [--metric <m>]`** — aggregate across the run index: `cost`, `duration`,
+  `tokens`, `cache-tokens`, `model-cost`, `turns`, `pass-rate`.
+- **`result.json` carries the raw fields** the assertions read: `toolDurations`, `models`, `toolErrors`,
+  `redundantToolCalls`, `modelUsage`, `thinking`, `skillActivity`, `subagents[]` (prompt/model/output/
+  `attributedSkillId`), `context` (tools/mcpServers/availableSkills), `tasks`, `workspaceFiles`,
+  `presentedFiles`, `hookEvents`, `mcpErrors`, `contextEvents`, `resources`. (Full per-field semantics:
+  the README's "Observability fields" section — repo-only; `schema/run-result.json` is the machine
+  source.)
+- **Opaque failure?** A failed run also records **`errorSource`** (where the failure originated) and
+  **`stderrLogPath`** (the captured agent stderr) — read those and `trace <run-dir>` *before* re-running;
+  a re-record rarely tells you more than the captured stderr already does.
+- **Attributing cost to sub-agent work.** `subagents[]` gives the dispatch tree — each sub-agent's
+  `model`, `toolsUsed`, `prompt`/`output`, and `attributedSkillId` — but **not** its own token/cost;
+  aggregate cost is per-**model** in `modelUsage` (and `trace --view usage`), not per-sub-agent. So a
+  cost spike from fan-out reads as `trace --view dispatches` (how many, which agent) against that model's
+  per-model usage — the harness doesn't line-item each sub-agent's tokens.
+- **Debugging a wrong Cowork UI panel.** Each panel is reconstructed in `result.json`: **Progress** =
+  `tasks[]`, **Working folder** = `workspaceFiles[]` (classified output/mount/input, with a
+  `trace --view files` diff), **Context / Connectors** = `context` (tools / mcpServers / availableSkills),
+  **Scratch-pad → outputs** = `presentedFiles[]`. If a panel looks wrong in a run, read its field.
+
+### Debugging with `chat`
 
 `cowork-harness chat` opens an interactive multi-turn REPL against a live Cowork session. It is
 **not** an asserted test — no `assert:` block, no cassette. Use it to explore behavior, reproduce a
 bug interactively, or test a prompt before committing it to a scenario.
 
+Each session still writes an informational `result.json` (`mode: "chat"`, no `assertions`) plus a
+trace and index row under its run dir — the same telemetry (tool durations, model usage, resources,
+etc.) that `run`/`skill` produce — so `cowork-harness trace <chat-run-dir>` / `stats` work on a chat
+session too, even though it never yields a verdict.
+
 **`--plugin <dir>` flag (repeatable).** Load additional skill folders alongside the primary session
 plugin. Each `--plugin <dir>` appends the folder to `local_plugins`. Useful when the skill-under-test
 depends on a sibling plugin:
@@ -331,8 +472,8 @@ The startup banner now reads `type your message (/help for commands)` as a remin
 
 ## Gotchas — the "✓ passed ≠ correct" landmines
 
-Stated as *symptom → why → fix*. The full catalog (with `file:line`) is in the references; these
-are the ones that bite hardest.
+Stated as *symptom → why → fix*. **This is the full landmine catalog;** `references/scenario-schema.md`
+repeats the assertion/replay-relevant ones alongside the schema (a scoped subset, not a fuller list).
 
 1. **An assertion passed but tested nothing on the PR gate.** *Why:* on a manifest-less cassette
    `replay` skips filesystem/egress keys (`file_exists`, `user_visible_artifact`, `artifact_json`,
@@ -369,9 +510,9 @@ are the ones that bite hardest.
 6. **`dispatch_count_max` is an author-chosen budget, not a production cap.** It's a post-hoc count
    assertion: passing means "happened to dispatch ≤N this run," nothing more. Cowork imposes **no**
    in-conversation `Task`-dispatch cap — gate `1648655587`'s `{perTask:1, global:3}` governs the
-   separate scheduled/cron-task session scheduler, not the `Task` tool (binary-verified; see SPEC
-   §10). So there is no "skip-on-cap" for the harness to reproduce; use this key only to catch a
-   fan-out you don't want.
+   separate scheduled/cron-task session scheduler, not the `Task` tool (binary-verified; details in
+   `SPEC.md` §10 — repo-only). So there is no "skip-on-cap" for the harness to reproduce; use this key
+   only to catch a fan-out you don't want.
 
 7. **`protocol` is rejected (not silently passed) if the scenario asserts egress** — boundary
    assertions need a sandboxed tier (`container`+). Good: this one fails loud by design.
@@ -394,13 +535,14 @@ are the ones that bite hardest.
     content assertion and gate on `file_exists` on the live lane instead.
 
 11. **An external decider returning `"first"` does not select option 1.** The `"first"` keyword
-    shorthand is disabled for `--decider-cmd` / `--decider-dir` helpers (see §5). If your helper
+    shorthand is disabled for `--decider-cmd` / `--decider-dir` helpers (see *Choose an answer path*
+    → External deciders). If your helper
     accidentally emits `"first"` and no label named `"first"` exists, the gate fails — it does
     **not** silently pick the first option. This is intentional: a helper bug should fail loud, not
     green wrong. *Fix:* have helpers return a label name or numeric index.
 
 12. **`prompt_asset_missing` is a WARN, not a hard failure — greens can hide it.** The
-    `prompt_asset_missing` verdict signal (see §8) does not block a green verdict. Scan the verdict
+    `prompt_asset_missing` verdict signal (see *Interpreting verdict signals*) does not block a green verdict. Scan the verdict
     signals section after every run; a run that greened with this signal ran against an incomplete
     prompt. *Fix:* treat `prompt_asset_missing` as a blocking error in CI by checking the signals
     array.
@@ -444,6 +586,11 @@ are the ones that bite hardest.
     any failure) — a failing scenario does NOT abort the batch. So a single `cowork-harness record cassettes/
     --rerecord-stale` surfaces ALL stale anchors in one pass (add `--concurrency <N>` to parallelize); a shell
     wrapper that loops one cassette at a time with `set -e` defeats this and rediscovers stale anchors serially.
+    Two durability properties make the batch safe to trust: each cassette is written **atomically** (a
+    same-directory temp file + rename), so an interrupted or OOM-killed batch never leaves a partial/corrupt
+    cassette — a failed scenario simply produces none; and under `--concurrency <N>` each scenario runs **fully
+    isolated** (its own egress sidecar network + proxy, its own per-session run dir), so parallel records don't
+    cross-talk — the concurrency bound exists only for the Docker address pool + API rate limits, not correctness.
 
 17. **Editing `scenarios/*.yaml` `assert:` does NOT change a plain `replay`.** *Why:* `replay` evaluates the
     assertions **frozen in the cassette** by default — it is byte-deterministic and ignores the working tree (so
@@ -479,8 +626,17 @@ are the ones that bite hardest.
     `RunResult.artifacts`). A `mode: rw`/`rwd` folder's contents are captured with a full body, same as
     `outputs/`.
 
-For the complete gotcha list, the assertion catalog, the YAML schema, the fidelity/answer tables,
-and the CI recipe, read the files in `references/`.
+21. **A `fidelity: cowork` cassette can go stale in a way `skill`/`format` drift won't catch.** Its recorded
+    `effectiveFidelity` field pins which concrete tier (`hostloop` or `container`) the baseline resolved to
+    AT RECORD TIME. If a later Desktop baseline flips that resolution, `verify-cassettes` reports it as a
+    `resolved-tier` finding (re-record — the recording now exercises the wrong tier); a cassette with no
+    `effectiveFidelity` at all, or an unloadable pinned `baseline:`, reports `unverifiable-tier` instead
+    (couldn't check — also re-record). Both are `fidelity: cowork`-only; an explicit-tier scenario never
+    produces them. (Details: `docs/cassette.md` § tier staleness — repo-only.)
+
+For the assertion catalog, the YAML schema, the fidelity/answer tables, and the CI recipe, read the
+files in `references/` (the gotchas above are the full list; the references repeat only the
+assertion/replay-relevant ones).
 
 ## References
 
@@ -489,10 +645,13 @@ and the CI recipe, read the files in `references/`.
   redaction before the first hostloop/protocol record, derive budget assertions without a
   two-pass record. Start here when the question is "how do I do X", not "what does flag Y mean".
 - `references/scenario-schema.md` — scenario/session YAML schema, full assertion catalog (with each
-  key's replay class), the web_fetch model, and the complete gotcha list.
+  key's replay class), the web_fetch model, and an assertion/replay-scoped gotcha subset (the full
+  landmine catalog lives in this SKILL's Gotchas section above).
 - `references/fidelity-and-answers.md` — fidelity tiers, answer paths, the determinism contract.
 - `references/ci-recipe.md` — the packaged GitHub Action, replay-vs-live lane split, and the four-stage
   GitHub Actions pipeline.
 - `scripts/scenario.py` — `scaffold` a valid scenario skeleton and `lint` scenarios for the
   no-silent-false-green invariants (both usable as CI steps).
-- `../../../docs/run-status.md` — checking a background run's status without `ps aux`.
+- Checking a background run's status without `ps aux` — covered in *Checking whether a background run is
+  alive* (Part II) above; the fuller recipe is in `docs/run-status.md` (repo-only, not shipped with the
+  installed skill).

package/.claude/skills/cowork-harness/references/ci-recipe.md

@@ -1,6 +1,6 @@
 # CI recipe — replay vs live lanes
 
-Self-contained reference. Tracks `cowork-harness 0.26.0` (baseline `desktop-1.18286.0`).
+Self-contained reference. Tracks `cowork-harness 0.28.0` (baseline `desktop-1.19367.0`).
 
 **Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
 job-summary reporter (verdict table, staleness findings, cost/turns when available):
@@ -45,15 +45,16 @@ jobs:
 Why this is a step *you* write, not an Action input the harness provides for you: pulling Anthropic's
 binary is a call about your own relationship with their distribution terms — keeping it in your own
 version-controlled workflow keeps that decision and its execution yours, auditable, and outside any
-third-party action's code. See the [agent-binary provenance runbook](../../../../docs/maintenance.md) for
-the full recovery/verification story (including why `COWORK_AGENT_BINARY` substitutions are
+third-party action's code. The agent-binary provenance runbook (`docs/maintenance.md` in the repo —
+not shipped with the installed skill) has the full recovery/verification story (including why
+`COWORK_AGENT_BINARY` substitutions are
 sha256-*checked* but not hard-blocking on mismatch — it's advisory for an intentional substitution).
 
 **Minimal token-free PR gate (manual form)** — the smallest thing worth committing; runs on stock
 GitHub-hosted runners, no token/Docker/agent:
 
 ```yaml
-- run: npm i -g "cowork-harness@>=0.26.0"
+- run: npm i -g "cowork-harness@>=0.28.0"
 - run: cowork-harness lint scenarios/*.yaml          # no silent false-greens
 - run: cowork-harness verify-cassettes cassettes/    # privacy + staleness
 - run: cowork-harness replay cassettes/              # token-free content/structure
@@ -187,7 +188,7 @@ jobs:
         with: { node-version: '20' }
       - uses: actions/setup-python@v5
         with: { python-version: '3.x' }                                       # python3 only — PyYAML is bundled with the linter
-      - run: npm i -g "cowork-harness@>=0.26.0"
+      - run: npm i -g "cowork-harness@>=0.28.0"
       - run: cowork-harness lint scenarios/*.yaml                              # no-silent-false-green (needs python3; PyYAML bundled)
       - run: cowork-harness verify-cassettes cassettes/ --output-format json   # privacy + staleness gate
       - run: cowork-harness replay cassettes/ --output-format json             # token-free content/structure
@@ -216,7 +217,7 @@ jobs:
             echo "live=true" >> "$GITHUB_OUTPUT"
           fi
       - if: steps.guard.outputs.live == 'true'
-        run: npm i -g "cowork-harness@>=0.26.0"
+        run: npm i -g "cowork-harness@>=0.28.0"
       - if: steps.guard.outputs.live == 'true'
         run: cowork-harness run scenarios/ --output-format json
         env:

package/.claude/skills/cowork-harness/references/fidelity-and-answers.md

@@ -1,6 +1,6 @@
 # Fidelity tiers & answer paths
 
-Self-contained reference. Tracks `cowork-harness 0.26.0` (baseline `desktop-1.18286.0`).
+Self-contained reference. Tracks `cowork-harness 0.28.0` (baseline `desktop-1.19367.0`).
 
 ## Fidelity tiers (`fidelity:` in the scenario)
 
@@ -58,7 +58,7 @@ regressions.
 | **Spawned helper** | `--decider-cmd '<helper>'` | depends on the helper |
 | **In-band (driving agent)** | `--decider-dir <dir>` (+ a Monitor that writes responses) | depends |
 
-The answer paths are orthogonal — don't mix them on one run. The in-band path's Monitor drives two
+Scripted `answers:` and a terminal (`--decider-llm` / `--decider-cmd` / `--decider-dir` / `on_unanswered`) compose as a normal Chain — scripted rules resolve matched gates, the terminal answers whatever's left. What's mutually exclusive is picking **two terminals** at once (e.g. `--decider-llm` with `--decider-cmd`, or `--decider-llm` with an explicit `--on-unanswered`) — the CLI rejects that combination. The in-band path's Monitor drives two
 subcommands: `gates <dir>` (stream the pending questions) and `answer <dir> --gate N …` (reply to one).
 
 **The `--decider-llm` answer protocol.** The answering model is shown the gate's options **numbered** and