cowork-harness 4.7.0 → 4.7.1

package/.claude/skills/cowork-harness/SKILL.md

@@ -3,8 +3,8 @@ name: cowork-harness
 description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, asserting artifacts, egress, or sub-agent dispatch, measuring how long each tool call took (toolDurations / trace), or debugging a failed run or verdict from its result.json or transcript. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. Also when iterating or hardening a skill across fixes, or grounding a skill's self-critique against its own run evidence — including a document-analysis skill (cap table, deck, financial model, transcript) that needs an uploaded file attached to be critiqued at all. Also for comparing two versions of a skill before merging an edit — did it make answers worse? (`eval`: paired A/B, pinned models) — or improving one round by round (`hillclimb`, the `/claude-api hillclimb` runner). NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold / critique / stats / eval / hillclimb commands and the session-vs-scenario split.
 metadata:
   author: cowork-harness
-  version: 4.7.0
-  tracks-harness: cowork-harness 4.7.0 (baseline desktop-2.31226.1)
+  version: 4.7.1
+  tracks-harness: cowork-harness 4.7.1 (baseline desktop-2.31226.1)
 ---
 
 # cowork-harness
@@ -26,7 +26,7 @@ allowlist). This skill exists mostly to keep you out of those traps — the *Inv
 full landmine catalog in [`references/gotchas.md`](references/gotchas.md) are the highest-value part.
 Read them.
 
-> **Version note:** the facts and `file:line` pointers here track `cowork-harness 4.7.0` (baseline
+> **Version note:** the facts and `file:line` pointers here track `cowork-harness 4.7.1` (baseline
 > `desktop-2.31226.1`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
 > `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
 
@@ -43,7 +43,7 @@ Before the first command, confirm the CLI is reachable and **fail loud** (never
 
 - **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
 - **Replay-only? Skip `doctor`.** Replaying committed cassettes needs no Docker, no staged agent, and no token — and every tier's `doctor` validates the auth token (the live tiers also Docker + the staged agent), so a ✗ there is expected, not a blocker. Go straight to `cowork-harness replay <cassette>`.
-- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 4.7.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@^4.7.0" <cmd>` (Node ≥ 22), or install once with `npm i -g "cowork-harness@^4.7.0"`. **Pin `@^4.7.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
+- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 4.7.1**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@^4.7.1" <cmd>` (Node ≥ 22), or install once with `npm i -g "cowork-harness@^4.7.1"`. **Pin `@^4.7.1`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
 
   This skill documents the CURRENT surface, not release history. If `cowork-harness --version` is
   OLDER than the floor, the per-release record of what you are missing is [CHANGELOG.md](https://github.com/yaniv-golan/cowork-harness/blob/main/CHANGELOG.md)

package/.claude/skills/cowork-harness/references/assertion-catalog-agents-skills-budgets.md

@@ -1,6 +1,6 @@
 # Assertion catalog: sub-agents, skills and connectors, budgets, tasks, delivery
 
-Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). Part 2 of 3 of the per-key table. The conventions every key shares (an item with several
+Tracks `cowork-harness 4.7.1` (baseline `desktop-2.31226.1`). Part 2 of 3 of the per-key table. The conventions every key shares (an item with several
 keys is an AND; glob vs regex name matching) and the verdict-signal table are in
 [`assertion-catalog.md`](./assertion-catalog.md); the other parts are [`assertion-catalog-outcome-files-tools.md`](./assertion-catalog-outcome-files-tools.md), [`assertion-catalog-gates-hooks-modifiers.md`](./assertion-catalog-gates-hooks-modifiers.md).
 

package/.claude/skills/cowork-harness/references/assertion-catalog-gates-hooks-modifiers.md

@@ -1,6 +1,6 @@
 # Assertion catalog: gates, hooks, path denial, verdict modifiers, egress, judged keys
 
-Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). Part 3 of 3 of the per-key table. The conventions every key shares (an item with several
+Tracks `cowork-harness 4.7.1` (baseline `desktop-2.31226.1`). Part 3 of 3 of the per-key table. The conventions every key shares (an item with several
 keys is an AND; glob vs regex name matching) and the verdict-signal table are in
 [`assertion-catalog.md`](./assertion-catalog.md); the other parts are [`assertion-catalog-outcome-files-tools.md`](./assertion-catalog-outcome-files-tools.md), [`assertion-catalog-agents-skills-budgets.md`](./assertion-catalog-agents-skills-budgets.md).
 

package/.claude/skills/cowork-harness/references/assertion-catalog-outcome-files-tools.md

@@ -1,6 +1,6 @@
 # Assertion catalog: outcome, transcript, files and artifacts, tools, tool results
 
-Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). Part 1 of 3 of the per-key table. The conventions every key shares (an item with several
+Tracks `cowork-harness 4.7.1` (baseline `desktop-2.31226.1`). Part 1 of 3 of the per-key table. The conventions every key shares (an item with several
 keys is an AND; glob vs regex name matching) and the verdict-signal table are in
 [`assertion-catalog.md`](./assertion-catalog.md); the other parts are [`assertion-catalog-agents-skills-budgets.md`](./assertion-catalog-agents-skills-budgets.md), [`assertion-catalog-gates-hooks-modifiers.md`](./assertion-catalog-gates-hooks-modifiers.md).
 

package/.claude/skills/cowork-harness/references/assertion-catalog.md

@@ -1,6 +1,6 @@
 # Assertion catalog
 
-Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). Every `assert:` key with its semantics, and the
+Tracks `cowork-harness 4.7.1` (baseline `desktop-2.31226.1`). Every `assert:` key with its semantics, and the
 verdict-signal table. Which keys survive `replay` is in [`scenario-schema.md`](./scenario-schema.md#replay-class);
 the scenario and session YAML fields are there too.
 
@@ -47,7 +47,7 @@ codes (`VerdictSignal["code"]` in `src/run/verdict.ts`):
 |---|---|---|
 | `assertion` | fail | An authored `assert:` item failed |
 | `result_error` | fail | The run's SDK result was `"error"` |
-| `usage_limit` | fail | Usage/quota limit hit (not a skill failure) — retry after the limit resets. Emitted when `RunResult.resultErrorKind === "usage_limit"` |
+| `usage_limit` | fail | Usage/quota limit hit (not a skill failure) — retry after the limit resets. Emitted when `RunResult.resultErrorKind === "usage_limit"` — including when a sub-agent hit the limit and the main loop ended `success`, where the message names the sub-agent and says to check its model and the account's plan instead of promising a reset (`decisions[]` row `subagent_usage_limit`) |
 | `transport_error` | fail | The connection dropped mid/after-run |
 | `permissive_auto_allow` | fail | A cowork-parity auto-allow real Cowork would block (opt out: `allow_permissive_auto_allow`) |
 | `outputs_delete` | fail | An unauthorized delete touched `mnt/outputs` on a baseline that records outputs `rw` (not raised on `rwd`, Desktop 2.16120.0+, unless `no_delete_in_outputs` or `no_delete_in_mounts` (outputs not waived) is authored — with `no_delete_in_outputs` that assertion fails instead; with `no_delete_in_mounts` this signal itself fires, and its message names that key), confirmed: the per-turn filesystem diff proves it, a delete in command/call position has an `outputs/` path as its own operand, or the diff could not verify the turn. Authoring `no_delete_in_outputs` moves it into that assertion; `allow_outputs_delete` waives it |

package/.claude/skills/cowork-harness/references/assertions-guide.md

@@ -1,6 +1,6 @@
 # Assertions guide
 
-Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). Read it when choosing assertion keys: the two orthogonal axes and the goal → key map. The full catalog starts at `assertion-catalog.md`, which indexes the per-family files that hold every key's row.
+Tracks `cowork-harness 4.7.1` (baseline `desktop-2.31226.1`). Read it when choosing assertion keys: the two orthogonal axes and the goal → key map. The full catalog starts at `assertion-catalog.md`, which indexes the per-family files that hold every key's row.
 
 ### Assertions: two orthogonal axes
 

package/.claude/skills/cowork-harness/references/authoring.md

@@ -1,6 +1,6 @@
 # Authoring a scenario
 
-Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). Read it when composing a `scenarios/*.yaml`: session vs scenario, discovery, the fidelity tier, the answer path, `web_fetch`, and scaffold + lint.
+Tracks `cowork-harness 4.7.1` (baseline `desktop-2.31226.1`). Read it when composing a `scenarios/*.yaml`: session vs scenario, discovery, the fidelity tier, the answer path, `web_fetch`, and scaffold + lint.
 
 ## Part I — AUTHOR a scenario
 

package/.claude/skills/cowork-harness/references/ci-recipe.md

@@ -1,6 +1,6 @@
 # CI recipe — replay vs live lanes
 
-Self-contained reference. Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`).
+Self-contained reference. Tracks `cowork-harness 4.7.1` (baseline `desktop-2.31226.1`).
 
 **Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
 job-summary reporter (verdict table, staleness findings, cost/turns when available):
@@ -17,7 +17,7 @@ job-summary reporter (verdict table, staleness findings, cost/turns when availab
 CLI major reaches your workflow the moment it is promoted even though your `uses:` ref never changed — so a
 copy-pasted recipe that omits the input takes a major bump with no say in it. `^4` holds the major, needs no
 patch number to remember, and only wants a human decision at the next major. Pin an exact version
-(e.g. `version: "4.7.0"`) instead when you want byte-reproducible CI.
+(e.g. `version: "4.7.1"`) instead when you want byte-reproducible CI.
 
 Reach for the manual multi-step form below only when you need per-step control the Action's inputs don't
 cover (a custom flag combination, a different runner matrix per step, or `lint`/`verify-cassettes` gated
@@ -82,7 +82,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
 GitHub-hosted runners, no token/Docker/agent:
 
 ```yaml
-- run: npm i -g "cowork-harness@^4.7.0"
+- run: npm i -g "cowork-harness@^4.7.1"
 - run: cowork-harness lint scenarios/*.yaml --strict --min-severity WARN
                                                     # no silent false-greens. WITHOUT --strict this
                                                     # step cannot fail on a WARN-class rule (e.g.
@@ -409,7 +409,7 @@ jobs:
         with: { node-version: '24' }
       - uses: actions/setup-python@v5
         with: { python-version: '3.x' }                                       # python3 only — PyYAML is bundled with the linter
-      - run: npm i -g "cowork-harness@^4.7.0"
+      - run: npm i -g "cowork-harness@^4.7.1"
       - run: cowork-harness lint scenarios/*.yaml                              # no-silent-false-green (needs python3; PyYAML bundled)
       - run: cowork-harness verify-cassettes cassettes/ --output-format json   # privacy + staleness gate
       - run: cowork-harness replay cassettes/ --output-format json             # token-free content/structure
@@ -438,7 +438,7 @@ jobs:
             echo "live=true" >> "$GITHUB_OUTPUT"
           fi
       - if: steps.guard.outputs.live == 'true'
-        run: npm i -g "cowork-harness@^4.7.0"
+        run: npm i -g "cowork-harness@^4.7.1"
       - if: steps.guard.outputs.live == 'true'
         run: cowork-harness run scenarios/ --output-format json
         env:

package/.claude/skills/cowork-harness/references/critique.md

@@ -1,6 +1,6 @@
 # Critique — the facts a plugin install can't otherwise reach
 
-Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). This is **not** a trim of the full
+Tracks `cowork-harness 4.7.1` (baseline `desktop-2.31226.1`). This is **not** a trim of the full
 [`docs/critique.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/critique.md) (repo-only —
 flags, cost, reproduction discipline, known limitations all live there). This file covers exactly what a
 plugin install cannot otherwise discover: the run-dir artifact a harvester actually reads, the report's
@@ -49,12 +49,12 @@ carry that; read all three before touching anything.
 | `infraFailure` | the reason |
 | `infraFailurePhase` | `task turn` (the graded run) or `reflection turn` (critique's own protocol turn) |
 | `infraFailureKind` | why it failed — a harness `ErrCategory` (error envelope, exit 2/3) **or** a `resultErrorKind` (`usage_limit`/`transport`/`agent`) from a turn that RAN and errored (exit 1, top-level `error: null`). **Absent** = killed, or no envelope |
-| `gradedErrorReason` | on a `taskResult: "error"` run (still gradeable, exit 0): why the GRADED turn errored, so a quota exhaustion is not read as a skill defect |
+| `gradedErrorReason` | on a `taskResult: "error"` run (still gradeable, exit 0): why the GRADED turn errored, so a dropped connection is not read as a skill defect. A task turn that hit the usage limit (its own result, or a sub-agent's) is not graded: `infraFailurePhase: "task turn"`, `infraFailureKind: "usage_limit"`, exit 2 |
 
 **Do NOT read "has a kind" as "the instrument is fine".** The CLI's top-level catch turns every
 unexpected throw into category **`internal`** — Docker down, container start failure, missing staged
 agent, harness bug — and `runtime` carries a refused run dir. Only **`unanswered`, `usage`, `boundary`**
-and, from the result-row taxonomy, **`usage_limit`** (quota exhausted — retry after reset) and
+and, from the result-row taxonomy, **`usage_limit`** (the account's quota — the main loop's or a sub-agent's; on the task turn critique exits 2 with `RUN FAILED (task turn, usage_limit)` and spawns no reflection) and
 **`transport`** (a tail-end drop) are the caller's problem. `agent` is not: for critique's own protocol
 turn that IS the instrument breaking. The header encodes exactly that split and fails closed (an
 unrecognized kind renders as infrastructure):

package/.claude/skills/cowork-harness/references/debugging.md

@@ -1,6 +1,6 @@
 # Debugging a run
 
-Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). Read it when a run misbehaved or a green looks wrong: triage, the observability output, and `chat`.
+Tracks `cowork-harness 4.7.1` (baseline `desktop-2.31226.1`). Read it when a run misbehaved or a green looks wrong: triage, the observability output, and `chat`.
 
 ## Part III — Debug
 
@@ -148,8 +148,12 @@ decide which assertions from *Assertions: two orthogonal axes* in `assertions-gu
   **`stderrLogPath`** (the captured agent stderr) — read those and `trace <run-dir>` *before* re-running;
   a re-record rarely tells you more than the captured stderr already does. Also check
   **`resultErrorKind`** (`"transport" | "agent" | "usage_limit"`) before spending another paid run: a
-  `"usage_limit"` failure is a quota exhaustion, not a skill bug — retry after the limit resets rather
-  than debugging; `"transport"`/`"agent"` means something actually broke, worth localizing before
+  `"usage_limit"` failure is the account's quota, not a skill bug — don't debug the skill. It is also set
+  when a **sub-agent** hit the limit and the main loop ended `success` (then `errorSource: "agent"`): the
+  `usage_limit` verdict message names that sub-agent (dispatch id, type, limit text), and `decisions[]`
+  carries a `name: "subagent_usage_limit"` row (`detail: {toolUseId?, subagentType?, taskId?, error,
+  prior?}`). Retry once the quota allows; for a sub-agent's limit also check the model it is pinned to and
+  the account's plan — a model-scoped or credits limit does not reset on a timer; `"transport"`/`"agent"` means something actually broke, worth localizing before
   re-running.
 - **Attributing cost to sub-agent work.** `subagents[]` gives the dispatch tree — each sub-agent's
   `dispatchModel`/`resolvedModel`, `toolsUsed`, `prompt`/`output`, and `attributedSkillId` — but **not** its own token/cost;

package/.claude/skills/cowork-harness/references/eval.md

@@ -1,6 +1,6 @@
 # `eval` — paired before/after comparison of a skill edit (EXPERIMENTAL)
 
-Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). The full guide is
+Tracks `cowork-harness 4.7.1` (baseline `desktop-2.31226.1`). The full guide is
 [docs/eval.md](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/eval.md); this is the part you
 need while running it.
 
@@ -48,7 +48,9 @@ cowork-harness eval report <eval-dir>     # rebuild the report from the eval dir
 | `insufficient` | too few valid reps in an arm (4 of 5 needed by default) |
 
 A drop is a signal to investigate, not proof: open the run dirs the report links for that row. In order,
-first match wins: an infrastructure failure is excluded and reported — including a rep where no model
+first match wins: an infrastructure failure is excluded and reported — including a usage limit the run lane
+named (the main loop's or a sub-agent's, rule `kind_usage_limit`, even on a rep that otherwise timed out or hit
+`error_max_turns`), and a rep where no model
 answered (the agent's `Not logged in` / `Authentication required` reply, rule `auth`; a usage or
 spend limit as its final message, even on a nonzero exit after spend, rule `usage_limit`; or only
 `<synthetic>` models at $0, rule `no_model_answered`). An agent-caused failure (timeout, max turns,

package/.claude/skills/cowork-harness/references/fidelity-and-answers.md

@@ -1,6 +1,6 @@
 # Fidelity tiers & answer paths
 
-Self-contained reference. Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`).
+Self-contained reference. Tracks `cowork-harness 4.7.1` (baseline `desktop-2.31226.1`).
 
 > **This page vs. the repo docs.** This is the **offline snapshot** that ships inside the installed
 > plugin — it is self-contained on purpose. The repo carries four other fidelity views, each answering a

package/.claude/skills/cowork-harness/references/gotchas.md

@@ -1,6 +1,6 @@
 # Gotchas
 
-Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). The full "✓ passed ≠ correct" landmine catalog.
+Tracks `cowork-harness 4.7.1` (baseline `desktop-2.31226.1`). The full "✓ passed ≠ correct" landmine catalog.
 
 ## Gotchas — the "✓ passed ≠ correct" landmines
 

package/.claude/skills/cowork-harness/references/hillclimb-recipe.md

@@ -1,6 +1,6 @@
 # Recipe 7 — Climb a skill with `/claude-api hillclimb` and the harness as its runner
 
-Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). It needs a `cowork-harness` whose
+Tracks `cowork-harness 4.7.1` (baseline `desktop-2.31226.1`). It needs a `cowork-harness` whose
 `hillclimb --help` lists `--skill` (help goes to stderr). This page is the loop's procedure, step by step, in the order of the
 `/claude-api hillclimb` guide. Every mechanic (flags, refusals, the gate, `regrade`, `freeze-ref`, exit codes,
 row keys) is in [`hillclimb.md`](hillclimb.md); the setup and the full list of differences from the guide's own

package/.claude/skills/cowork-harness/references/hillclimb.md

@@ -1,6 +1,6 @@
 # `hillclimb` — the runner for a `/claude-api hillclimb` loop
 
-Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). It needs a `cowork-harness` whose
+Tracks `cowork-harness 4.7.1` (baseline `desktop-2.31226.1`). It needs a `cowork-harness` whose
 `hillclimb --help` lists `--skill` (help goes to stderr). The command reference is
 [docs/cli.md](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/cli.md); this is the part a loop needs
 while it runs. It covers `run`, `check`, `state-template`, `freeze-ref` and `regrade`.
@@ -382,6 +382,26 @@ Flags: `--flow DIR`, `--variant all|baseline|v<N>` (default `all`: every variant
   row, never by `--allow-doc-drift`); one whose assert has an edit inside a scrubbed literal (only a re-run applies
   it); and an open `judge_invalid` slot.
 
+## Cutting the cost per rep
+
+Every rep re-runs the whole scenario, so a case that tests the last step of a long pipeline pays for every step
+before it on every rep. Start such a case from a workspace fixture instead: a directory whose files are copied into
+the session's `outputs/` before turn 1, so the prompt asks for the late step alone
+([Starting from a saved workspace](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/scenario.md#starting-from-a-saved-workspace-workspace_fixture)).
+
+- **Make one from a kept run.** `cowork-harness fixture export <run-dir> --out <dir>`, then point the case's
+  `workspace_fixture:` at `<dir>` and commit both ([fixture export](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/cli.md#exporting-a-runs-outputs-as-a-fixture-fixture-export)).
+  Add `--session-paths` when the skill reads back paths it recorded (an outputs-dir probe, a sub-agent's output
+  path). `--exclude <path>` leaves out a file the step re-creates, such as a probe the skill re-runs.
+- **Limits.** A fixture with session-path tokens is refused on the `protocol` tier, and `--session-paths` refuses a
+  protocol run. Binary files are never rewritten; one that holds a token is refused. Any other host path in a text
+  file is still refused (`--allow-host-paths` accepts an ordinary one, never a path into a run dir).
+- **What the case then measures.** Only the steps after the fixture. An untouched fixture file is pre-run, not
+  authored, so a judge grades only what this step wrote, and the conversation starts fresh. The earlier steps are no
+  longer exercised: keep a full-pipeline case for them.
+- **The fixture is in the harness gate.** Its files are hashed with the scenario, so an edited or re-exported fixture
+  changes the sha and needs `--approve-harness` again.
+
 ## Exit codes
 
 - `run`: `0` every attempted (case, rep) was scored; `1` an attempt failed (an `errors.jsonl` row, or a scored

package/.claude/skills/cowork-harness/references/measurement.md

@@ -1,6 +1,6 @@
 # Measurement
 
-Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). Read it before comparing runs: `--repeat`, `--ablate-skill`, and the hygiene that keeps a batch valid.
+Tracks `cowork-harness 4.7.1` (baseline `desktop-2.31226.1`). Read it before comparing runs: `--repeat`, `--ablate-skill`, and the hygiene that keeps a batch valid.
 
 ### Measure — before/after, with/without (`--repeat`, `--ablate-skill`)
 

package/.claude/skills/cowork-harness/references/run-record-replay.md

@@ -1,6 +1,6 @@
 # Run, record and lock
 
-Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). Read it when running a scenario, recording or placing a cassette, reading verdict signals, checking a background run, or choosing CI lanes.
+Tracks `cowork-harness 4.7.1` (baseline `desktop-2.31226.1`). Read it when running a scenario, recording or placing a cassette, reading verdict signals, checking a background run, or choosing CI lanes.
 
 ## Part II — RUN, RECORD & LOCK
 

package/.claude/skills/cowork-harness/references/scenario-schema.md

@@ -1,6 +1,6 @@
 # Scenario & session schema, replay class, web_fetch, authoring gotchas
 
-Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 4.7.0`
+Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 4.7.1`
 (baseline `desktop-2.31226.1`). If your checkout is newer, prefer the live [`docs/scenario.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/scenario.md),
 [`docs/session.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/session.md), and `SPEC.md`.
 

package/.claude/skills/cowork-harness/references/semantic-judging.md

@@ -1,6 +1,6 @@
 # `semantic_matches` — the full semantics
 
-Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). The one-line summary is in
+Tracks `cowork-harness 4.7.1` (baseline `desktop-2.31226.1`). The one-line summary is in
 [assertion-catalog-gates-hooks-modifiers.md](assertion-catalog-gates-hooks-modifiers.md); this is the whole contract, split out so the catalog stays within the
 agent's single-read size.
 

package/.claude/skills/cowork-harness/references/task-recipes.md

@@ -2,7 +2,7 @@
 
 Each recipe composes facts that live scattered across SKILL.md and the other references into one
 decision path. Every one answers a question a real fleet owner had to work out the hard way.
-Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`), same as SKILL.md's front-matter. Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
+Tracks `cowork-harness 4.7.1` (baseline `desktop-2.31226.1`), same as SKILL.md's front-matter. Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
 Recipe 3's `init-redact` shipped in 0.24.0 and are part of the current feature set — no version gate
 needed if your CLI meets SKILL.md's version floor.
 

package/CHANGELOG.md

@@ -6,6 +6,57 @@ All notable changes to this project are documented here. The format is based on
 
 ## [Unreleased]
 
+## [4.7.1] — 2026-10-10
+
+### Upgrade notes
+
+- **`run`, `skill`, `record` and `replay` now fail a run whose sub-agent hit the account's usage limit** (exit 1,
+  `result: "error"`, `resultErrorKind: "usage_limit"`, outcome `errored`), even when the main loop carried on and
+  ended `success`. Such a run used to grade `delivered_clean`. `errorSource` is `agent` when the main loop itself ended
+  `success`; a source the run already had (`result`, `exit`, `timeout`, `no_result`, …) is kept, and an errored
+  run's kind becomes `usage_limit`. Replaying an older cassette recorded on such a run now fails it the same way;
+  `record` refuses to freeze one without `--allow-failing`, as for any failing run.
+- **`critique` whose task turn hits the account's usage limit now exits 2 with no critique.** That covers the main
+  loop's own limit and a sub-agent's. It used to grade the errored task and exit 0, then spawn the reflection turn on
+  the spent quota. The report reads `RUN FAILED (task turn, usage_limit)`, with `infraFailurePhase: "task turn"` and
+  `infraFailureKind: "usage_limit"`, and still carries `taskResult`, `gradedOutcome` and `gradedSkillHash`. Exit 2
+  keeps its meaning — no critique was produced. A task turn that errored any other way is still graded (exit 0).
+- **`errorSource: "agent"` can now mean a sub-agent's terminal quota error.** No field or enum value is added. The
+  `usage_limit` verdict message names the sub-agent (its dispatch id, its type and the limit text), and the run's
+  `decisions` gain a row named `subagent_usage_limit` (`decision: "error"`, `detail: {toolUseId?, subagentType?,
+  taskId?, error, prior?}`; `prior` is the error label the run had before the limit upgraded it).
+
+### Fixed
+
+- **A sub-agent that hits the account's usage limit no longer makes the run grade clean.** When a sub-agent (the
+  `Agent`/`Task` tool) failed on the account's quota, the main loop narrated the failure and ended its turn `success`,
+  and the harness classified a usage limit only on the main loop's own result, so the run read `delivered_clean`. The
+  run lane now reads the agent's own task-failure update, whose error is the one the sub-agent's run threw, never
+  text the model wrote. A model quoting the limit, a sub-agent that completed with the text in its reply, a web
+  search's own limit reply, a failure for another reason, a transient rate limit and a background shell task are not
+  flagged. The run fails even when the main loop recovers and delivers, since the stream cannot show that the lost
+  work was replaced. `eval` and `hillclimb` exclude such a rep as infrastructure (`errored_infra`, rule
+  `kind_usage_limit`) instead of grading it.
+- **`critique` no longer grades a task turn that ran out of quota, or spawns its reflection turn on that quota.** See
+  the upgrade note. Such a failure used to surface, if at all, as a failed "reflection turn".
+
+### Known limits
+
+- **Chat sessions** do not name the usage limit: a chat run whose sub-agent hit it reads `error` with no
+  `resultErrorKind`, as a main-loop limit already did there.
+- **The cloud lane** does not show a sub-agent's failure text, so a sub-agent's usage limit is not detected there.
+- **A background sub-agent that fails after the main loop's final result** may not be seen before the run ends.
+- **A run that stops at an unanswered question or a decider timeout** after a sub-agent's limit is reported by
+  that stop, not as a usage limit.
+- **Run dirs written before 4.7.1** keep their recorded result under `verify-run` and `regrade`, which read the
+  stored `result.json`; re-run or replay to apply the new reading.
+
+### Documentation
+
+- **Cutting the cost per rep.** `docs/eval.md`, `docs/hillclimb.md` and the skill's hillclimb reference describe
+  starting a late-step case from a workspace fixture exported from a kept run (`fixture export`, `--session-paths`,
+  `--exclude`), its limits, and that the case then measures only the steps after the fixture.
+
 ## [4.7.0] — 2026-10-10
 
 ### Upgrade notes

package/DESIGN.md

@@ -208,7 +208,7 @@ The policy that produces those `allow`/`deny` responses is the **Decider** seam
 > Cowork system-prompt fingerprint all unchanged from `1.20186.0`, with the staged VM ELF re-synced
 > 2.1.202 → 2.1.205 — and the live pass of that era was deliberately **not** restamped onto it.)
 
-> **Scope of that claim.** `2026-10-10 / desktop-2.31226.1` is the baseline carrying the latest live pass, run against agent **2.1.295** (the staged Linux ELF for `container`, and the staged native macOS build `9d6405864fdc` for `hostloop`, with no override of either); the `protocol` tier runs the HOST `claude`, **2.1.296** for this pass. No baselines have shipped since. **The 4.7.0 pass, 2026-10-10, on `desktop-2.31226.1` (agent 2.1.295; host `claude` 2.1.296 for `protocol`), on the release-prep branch at `2dba294f`** (the commits after it change docs only): (i) `vitest list --staticParse=false` listed all 24 live tests with no `SKIPPED` warning when the token was exported, and 6 with five warnings when none was resolvable; (ii) the live suite on `protocol`, `container` and `hostloop`: 24 passed, one pre-registered skip; (iii) `artifact_json` `schema:` on `container`: a whole-document schema passed, and a schema of `type: string` on the integer at `count` failed alone, naming the path and "must be string"; a schema using `format` was refused at load (exit 2) before any spend, and `lint` reported it as an error; (iv) `fixture export --session-paths` on `container`: without the flag, export refused the file holding the run's outputs path (`run_path`) and wrote nothing; with it, the path became `__COWORK_HARNESS_VM_SESSION_ROOT__/mnt/outputs`, and a next step staged from that fixture read its own new session's path, with no token left; (v) `record <dir/> --case`: the `--dry-run` budget check priced the named scenario alone and passed where the whole directory was refused, and a real `--case` record wrote one cassette for the named scenario only; it carries `hookFailureBlocks: {"events": []}` and stamps v12, as before; (vi) `onFailure: "block"` on `container`: two runs identical but for that field on a PreToolUse hook that exits 1 on the prompted Bash call. Both recorded the same failed hook frame (exit 1, `outcome: error`). With the field, the agent blocked the call, the run's inventory listed `PreToolUse`, and both forms of `no_hook_event_blocked` failed as evidence-unavailable; without it, the call ran, the inventory was empty, and both passed. Recorded with `--allow-failing`, the first run stamped v16, and its replay reproduced the two evidence-unavailable results; (vii) the companion skill's router check, the authoring prompt on `container`: the first run's first `AskUserQuestion` held a single option and was rejected by the agent's own input validation, and the run ended on a question asked in prose (graded `stalled`), with `authoring.md` read but no scenario written, as in 4.6.1. The release checklist's prompt now names the skill folder and the report path; the re-run read `authoring.md` (and `scenario-schema.md` and an assertion-catalog part) and wrote a session file and a scenario that pass `lint` with no error or warning. Live spend about **$2.07**. **The 4.6.1 pass, 2026-10-09, on `desktop-2.31226.0` (agent 2.1.293; host `claude` 2.1.295 for `protocol`), on the release-prep branch at `e5ca4bd9`**: (i) `vitest list --staticParse=false` listed all 24 live tests with no `SKIPPED` warning when the token was exported, and 6 with five warnings when none was resolvable; (ii) the live suite on `protocol`, `container` and `hostloop`: 23 passed, one pre-registered skip, and one red in the plugin hook-decision test: the model made a second, unprompted Bash call, the probe's hook denied both, and the harness counted both, so the probe's exact count of one Bash denial failed. The probe now denies only the calls its prompt asks for, and the re-run passed all twelve assertions with exactly one denial each from the Bash, Write and Stop hooks; (iii) the auto-memory check on all three tiers, `Tests  2 passed (2)` each, every init frame without `memory_paths`; (iv) Ctrl-C on `hostloop` on macOS, SIGINT to the harness's whole process group while the agent ran `sleep 120` in the sandbox: exit 130 in 1.28 s, and in 1.33 s with twelve concurrent `ps -A -o tty=` loops running, where the same stop took about 8 s before 4.6.1; both times nothing in the process tree captured before the signal survived, the sandbox container was gone and `status.json` ended `error`; (v) a `lane: remote` scenario asserting `artifact_text`, and one asserting `semantic_matches` with `evidence_files`, are refused by `run` at load (exit 2) before any spend, naming the key and a remedy, and `lint` reports them as errors (exit 1); (vi) the companion skill's router check, the authoring prompt on `container`: one run's first `AskUserQuestion` was rejected by the agent's own input validation and the run ended by asking for the skill's path, with no reference read and no scenario written; the re-run read `authoring.md` (and `scenario-schema.md` and an assertion-catalog part) and wrote a scenario that passes `lint` with no error or warning. Live spend about **$2.06**. **The 4.6.0 pass, 2026-10-08, on `desktop-2.26454.2` (agent 2.1.293; host `claude` 2.1.294 for `protocol`), on the release-prep branch at `daf3479d`**: (i) `vitest list --staticParse=false` listed all 24 live tests with no `SKIPPED` warning when the token was exported, and 6 with five warnings when none was resolvable; (ii) the live suite on `protocol`, `container` and `hostloop`: 24 passed, one pre-registered skip, including the plugin hook-decision test on `container` (a PreToolUse hook denying by JSON, one exiting 2, and a Stop hook blocking once, graded by `hook_decision` and both forms of `hook_event_blocked`); (iii) the auto-memory check on all three tiers, `Tests  2 passed (2)` each, every init frame without `memory_paths`; (iv) the companion skill's router check, three prompts on `container`, against the split assertion catalog: the authoring prompt read `authoring.md` and `scenario-schema.md` and wrote a session file and a scenario that pass `lint` with no error or warning, the debugging prompt read `debugging.md` and asked for the run's `result.json`, and the tool-timing prompt read `measurement.md` and answered with `trace … --view tool-durations`; (v) `gates_all_scripted` on `container`: a scripted multi-select gate passed, and the same scenario with its answer removed and `on_unanswered: first` failed on exactly that assertion, naming the gate as answered by `first`; (vi) `answer_channel: none` on `container`, with a probe skill that records a status file before asking: the run passed with a `parked_at_question` warning and no `stalled` fail, carried the headless label and `answerChannel: "none"`, graded the status file through an `artifact_json` glob, and its init frame listed no `AskUserQuestion`. Live spend about **$2.21**. The 4.4.1, 4.4.0 and 4.3.0 passes below ran on `desktop-2.19675.0` (agent 2.1.286). **The 4.5.0 pass, 2026-10-07, on `desktop-2.26454.0` (agent 2.1.289; host `claude` 2.1.292 for `protocol`), on the release-prep branch at `af26c3c7`** (the commits after it change docs and the companion skill only): (i) `vitest list --staticParse=false` listed all 23 live tests with no `SKIPPED` warning when the token was exported, and 6 with four warnings when none was resolvable; (ii) the live suite on `protocol`, `container` and `hostloop`: 23 passed, one pre-registered skip; (iii) the auto-memory check on all three tiers, `Tests  2 passed (2)` each, every init frame without `memory_paths`; (iv) the companion skill's router check, three prompts on `container`: the authoring prompt read `authoring.md` (and `assertion-catalog.md`, searching `scenario-schema.md`) but wrote a scenario with an inline `session:` block, which fails `lint`, so it failed; `authoring.md` now says `session:` is a path, and one re-run read `authoring.md` and `scenario-schema.md` and asked for the skill's path instead of writing a scenario, so the corrected sentence is checked by `lint`, not by a live answer. The debugging prompt read no reference and answered with correct clarifying questions; the tool-timing prompt loaded the skill, read no reference and asked which run to time, so `measurement.md` routing was not exercised in this pass; (v) the forced-compaction recipe on `container`: a task turn, a resumed `/compact` turn whose `result.json` records a manual `compact_boundary` (40,266 → 2,896 tokens; the check exits 1 on the task turn), and a resumed turn that still named the command it had described before the compaction. The recipe first said to check this with `verify-run`, which refuses a run dir with more than one turn; it now reads the `/compact` turn's own `result.json`. Live spend about **$2.53**. **The 4.4.1 pass, 2026-10-05, on the release-prep commit `3e6fe443`** (a companion-skill-only release; same baseline and staged agent; host `claude` 2.1.289 for `protocol`): (i) `vitest list --staticParse=false` listed all 23 live tests with no `SKIPPED` warning when the token was exported, and 6 with four warnings when none was resolvable; (ii) the auto-memory check on all three tiers, `Tests  2 passed (2)` each, every init frame without `memory_paths`; (iii) the companion skill's router check, three prompts on `container`: the authoring prompt read `authoring.md` (and `scenario-schema.md`, searching `assertion-catalog.md`) and wrote a scenario that passes `lint` with no error or warning, the debugging prompt read `debugging.md` and answered with correct clarifying questions, and the tool-timing prompt read `measurement.md` (and `debugging.md`, which documents `trace`'s views) and answered with `trace … --view tool-durations`. The live suite itself was not re-run: this release changes no code that runs. Live spend about **$1.08**. **The 4.4.0 pass, 2026-10-05, on the release-prep commit `e9c516df`** (same baseline and staged agent; host `claude` 2.1.289 for `protocol`): (i) `vitest list --staticParse=false` listed all 23 live tests with no `SKIPPED` warning when the token was exported, and 6 with four warnings when none was resolvable; (ii) the live suite on `protocol`, `container` and `hostloop`: 23 passed, one pre-registered skip; (iii) the auto-memory check on all three tiers, `Tests  2 passed (2)` each; (iv) the companion skill's router check, three prompts on `container`: the authoring prompt read `authoring.md` and wrote a scenario that passes `lint`, the tool-timing prompt read `measurement.md` and answered with `trace … --view tool-durations`, and the debugging prompt read no reference and answered with correct clarifying questions, so `debugging.md` routing was not exercised in this pass; (v) the re-grade scrub-set check: a live run on the default runs root created `scrubset.key` beside the runs directory (mode 0600) and recorded `scrubSet` (`v: 1`) with the configured scrub literal absent from `result.json`; a re-grade of that run with an edited rubric line and a smaller scrub set was refused as `rubric_unverifiable` (exit 2) naming only the edited line, with no judge call and the run directory byte-identical; and a re-grade of a run recorded before 4.4 with an edited rubric line was refused with the `legacy` reason and the one-line remedy, again with no judge call and the run directory byte-identical. Live spend about **$2.83** recorded. **The 4.3.0 pass, on the release commit `8a5e479f`:** (1) the live suite on `protocol`, `container` and `hostloop`: 23 passed, one pre-registered skip (the auto-memory prerequisites check, which runs only under `COWORK_LIVE_REQUIRE=1`); before it ran, `vitest list --staticParse=false` listed all 23 with no `SKIPPED` warning, and 6 with four warnings when no token was resolvable, so a silent skip would have shown; (2) the auto-memory check on all three tiers, `Tests  2 passed (2)` each, every init frame without `memory_paths`; (3) the graders' pinned effort: every judged assert that called the judge in the live hillclimb acceptance runs (45, Opus 4.8: 25 `semantic_matches`, 20 `semantic_pairwise`; host `claude` 2.1.288 and 2.1.289) records `judgeTransport.effort: "high"`; the five pairwise asserts without a `judgeTransport` are the reference variant's own rows, which call no judge. **Earlier in the same release, on the release candidate's tree:** (4) a `microvm` smoke on a freshly booted VM (agent 2.1.286); (5) `boundary-check`, 6/6 constraints enforced; (6) an `eval` A/A at `--reps 4` on `container`, every row `no detectable change`, its report rebuilt byte-identical; (7) the companion skill's router check, three prompts on `container`: routing was correct on all three; one answer was wrong because a reference showed `artifact_text`'s `contains` without its list shape (fixed in the references, with a test that keeps every list-typed field shown as a list), and the same prompt's verdict went red on a false `host_path_leak`, because a reference spelled out the host roots that check looks for and the agent echoed it (fixed in the skill's files and in the check, which no longer counts a literal that comes from the plugin's or a local skill's own staged files); re-run after the fixes, the prompt routed correctly, wrote a scenario that passes `lint` and loads, and carried no `host_path_leak`; (8) the `hostloop` detached-agent kill check: the first pass found the workspace sidecar container, its `docker run` client and its network surviving SIGINT, which was fixed, and the re-run passed on a normal run, SIGINT to the harness process and SIGINT to the whole process group (exit 130, nothing left behind); (9) `vm delete`'s usage refusal; (10) `chat` under a real terminal, on `protocol`: Ctrl-C mid-turn exits 130 in about 2 s with nothing left running and no `result.json`; `/exit` followed by Ctrl-C exits 130 after writing the turn's `result.json`; Ctrl-C at the idle prompt ends the session normally (exit 0, `result.json` written); SIGHUP mid-turn exits 129 with nothing left running; and closing the terminal window mid-turn (run by hand) leaves no harness or agent process and no `result.json`. Total live spend about **$11.20**, including the re-runs. **Scope-out, so this is not read as more than it is.** (a) A live pass verifies observed behaviour, not the whole spawn contract by construction, and these checks are model-dependent, so a single red is evidence of model variance until a re-run says otherwise. (b) **CI does not live-validate anything.** Its "scenario suite (… live inference)" job is skipped as a whole without an `ANTHROPIC_API_KEY` repository secret, and none is set, so it shows as *skipped*. Before 2026-09 it instead ran with every real step skipped and reported *success*, so an older green check there means nothing ran. This note, not CI, is the live evidence. Separately and not a live matter: all four committed cassettes were **re-recorded** on 2026-10-03 against `2.19675.0` with auto-memory off, each on its original model: `example-pdf-skill` and `test/fixtures/tool-call-dispatch/dispatch-shell.cassette.json` (`container`, agent 2.1.286), `hostloop-computer-links` (`hostloop`, agent 2.1.286) and `example-multiselect-gate` (`protocol`, which runs the host CLI, here 2.1.287), about $0.75 in all. Their init frames carry no `memory_paths`. The recorder's own scrub (now applied to every recording) removed, across the four, the account's model menu, a subscription account's `rate_limit_info` (from `example-pdf-skill`, `dispatch-shell` and `hostloop-computer-links`) and the agent's sub-agent hand-back frame (from `dispatch-shell`). `verify-cassettes` exits 0 on all four, with one accepted `unscanned` entry (`example-pdf-skill`'s uploaded artifact body, too large to commit). Re-stamp this paragraph, naming the baseline, whenever a live pass is actually re-run.
+> **Scope of that claim.** `2026-10-10 / desktop-2.31226.1` is the baseline carrying the latest live pass, run against agent **2.1.295** (the staged Linux ELF for `container`, and the staged native macOS build `9d6405864fdc` for `hostloop`, with no override of either); the `protocol` tier runs the HOST `claude`, **2.1.296** for this pass. No baselines have shipped since. **The 4.7.1 pass, 2026-10-10, on `desktop-2.31226.1` (agent 2.1.295; host `claude` 2.1.296 for `protocol`), on the release-prep branch at `3c13a4f8`** (a patch: a smoke on two tiers, not the live suite; the one code change after it widens which task types the usage-limit check reads, on a path neither run reached): (i) `container`, the sub-agent dispatch scenario (`test/fixtures/tool-call-dispatch/dispatch-shell.yaml`, `claude-sonnet-5`): a real sub-agent ran and completed, the stream carried `task_started` (`task_type: local_agent`), a `task_updated` whose patch was `completed` with no error, and a completed `task_notification`; the run passed, `delivered_clean`, with no `subagent_usage_limit` row; (ii) `protocol`, `examples/scenarios/protocol-smoke.yaml`: the first run had no credential in its environment, so it read the operator's real config dir and its verdict failed on `l0_host_config_contamination` alone (`result: success`, every assertion passed, no `subagent_usage_limit` row); the re-run with `COWORK_MANAGED_CONFIG=1` and the credential loaded read a sealed config dir and passed, `delivered_clean`, with no fail signal. A sub-agent that hits the usage limit was not triggered live (it needs a spent quota); that path is covered by tests built from the agent's frame shapes. Live spend about **$0.88**. **The 4.7.0 pass, 2026-10-10, on `desktop-2.31226.1` (agent 2.1.295; host `claude` 2.1.296 for `protocol`), on the release-prep branch at `2dba294f`** (the commits after it change docs only): (i) `vitest list --staticParse=false` listed all 24 live tests with no `SKIPPED` warning when the token was exported, and 6 with five warnings when none was resolvable; (ii) the live suite on `protocol`, `container` and `hostloop`: 24 passed, one pre-registered skip; (iii) `artifact_json` `schema:` on `container`: a whole-document schema passed, and a schema of `type: string` on the integer at `count` failed alone, naming the path and "must be string"; a schema using `format` was refused at load (exit 2) before any spend, and `lint` reported it as an error; (iv) `fixture export --session-paths` on `container`: without the flag, export refused the file holding the run's outputs path (`run_path`) and wrote nothing; with it, the path became `__COWORK_HARNESS_VM_SESSION_ROOT__/mnt/outputs`, and a next step staged from that fixture read its own new session's path, with no token left; (v) `record <dir/> --case`: the `--dry-run` budget check priced the named scenario alone and passed where the whole directory was refused, and a real `--case` record wrote one cassette for the named scenario only; it carries `hookFailureBlocks: {"events": []}` and stamps v12, as before; (vi) `onFailure: "block"` on `container`: two runs identical but for that field on a PreToolUse hook that exits 1 on the prompted Bash call. Both recorded the same failed hook frame (exit 1, `outcome: error`). With the field, the agent blocked the call, the run's inventory listed `PreToolUse`, and both forms of `no_hook_event_blocked` failed as evidence-unavailable; without it, the call ran, the inventory was empty, and both passed. Recorded with `--allow-failing`, the first run stamped v16, and its replay reproduced the two evidence-unavailable results; (vii) the companion skill's router check, the authoring prompt on `container`: the first run's first `AskUserQuestion` held a single option and was rejected by the agent's own input validation, and the run ended on a question asked in prose (graded `stalled`), with `authoring.md` read but no scenario written, as in 4.6.1. The release checklist's prompt now names the skill folder and the report path; the re-run read `authoring.md` (and `scenario-schema.md` and an assertion-catalog part) and wrote a session file and a scenario that pass `lint` with no error or warning. Live spend about **$2.07**. **The 4.6.1 pass, 2026-10-09, on `desktop-2.31226.0` (agent 2.1.293; host `claude` 2.1.295 for `protocol`), on the release-prep branch at `e5ca4bd9`**: (i) `vitest list --staticParse=false` listed all 24 live tests with no `SKIPPED` warning when the token was exported, and 6 with five warnings when none was resolvable; (ii) the live suite on `protocol`, `container` and `hostloop`: 23 passed, one pre-registered skip, and one red in the plugin hook-decision test: the model made a second, unprompted Bash call, the probe's hook denied both, and the harness counted both, so the probe's exact count of one Bash denial failed. The probe now denies only the calls its prompt asks for, and the re-run passed all twelve assertions with exactly one denial each from the Bash, Write and Stop hooks; (iii) the auto-memory check on all three tiers, `Tests  2 passed (2)` each, every init frame without `memory_paths`; (iv) Ctrl-C on `hostloop` on macOS, SIGINT to the harness's whole process group while the agent ran `sleep 120` in the sandbox: exit 130 in 1.28 s, and in 1.33 s with twelve concurrent `ps -A -o tty=` loops running, where the same stop took about 8 s before 4.6.1; both times nothing in the process tree captured before the signal survived, the sandbox container was gone and `status.json` ended `error`; (v) a `lane: remote` scenario asserting `artifact_text`, and one asserting `semantic_matches` with `evidence_files`, are refused by `run` at load (exit 2) before any spend, naming the key and a remedy, and `lint` reports them as errors (exit 1); (vi) the companion skill's router check, the authoring prompt on `container`: one run's first `AskUserQuestion` was rejected by the agent's own input validation and the run ended by asking for the skill's path, with no reference read and no scenario written; the re-run read `authoring.md` (and `scenario-schema.md` and an assertion-catalog part) and wrote a scenario that passes `lint` with no error or warning. Live spend about **$2.06**. **The 4.6.0 pass, 2026-10-08, on `desktop-2.26454.2` (agent 2.1.293; host `claude` 2.1.294 for `protocol`), on the release-prep branch at `daf3479d`**: (i) `vitest list --staticParse=false` listed all 24 live tests with no `SKIPPED` warning when the token was exported, and 6 with five warnings when none was resolvable; (ii) the live suite on `protocol`, `container` and `hostloop`: 24 passed, one pre-registered skip, including the plugin hook-decision test on `container` (a PreToolUse hook denying by JSON, one exiting 2, and a Stop hook blocking once, graded by `hook_decision` and both forms of `hook_event_blocked`); (iii) the auto-memory check on all three tiers, `Tests  2 passed (2)` each, every init frame without `memory_paths`; (iv) the companion skill's router check, three prompts on `container`, against the split assertion catalog: the authoring prompt read `authoring.md` and `scenario-schema.md` and wrote a session file and a scenario that pass `lint` with no error or warning, the debugging prompt read `debugging.md` and asked for the run's `result.json`, and the tool-timing prompt read `measurement.md` and answered with `trace … --view tool-durations`; (v) `gates_all_scripted` on `container`: a scripted multi-select gate passed, and the same scenario with its answer removed and `on_unanswered: first` failed on exactly that assertion, naming the gate as answered by `first`; (vi) `answer_channel: none` on `container`, with a probe skill that records a status file before asking: the run passed with a `parked_at_question` warning and no `stalled` fail, carried the headless label and `answerChannel: "none"`, graded the status file through an `artifact_json` glob, and its init frame listed no `AskUserQuestion`. Live spend about **$2.21**. The 4.4.1, 4.4.0 and 4.3.0 passes below ran on `desktop-2.19675.0` (agent 2.1.286). **The 4.5.0 pass, 2026-10-07, on `desktop-2.26454.0` (agent 2.1.289; host `claude` 2.1.292 for `protocol`), on the release-prep branch at `af26c3c7`** (the commits after it change docs and the companion skill only): (i) `vitest list --staticParse=false` listed all 23 live tests with no `SKIPPED` warning when the token was exported, and 6 with four warnings when none was resolvable; (ii) the live suite on `protocol`, `container` and `hostloop`: 23 passed, one pre-registered skip; (iii) the auto-memory check on all three tiers, `Tests  2 passed (2)` each, every init frame without `memory_paths`; (iv) the companion skill's router check, three prompts on `container`: the authoring prompt read `authoring.md` (and `assertion-catalog.md`, searching `scenario-schema.md`) but wrote a scenario with an inline `session:` block, which fails `lint`, so it failed; `authoring.md` now says `session:` is a path, and one re-run read `authoring.md` and `scenario-schema.md` and asked for the skill's path instead of writing a scenario, so the corrected sentence is checked by `lint`, not by a live answer. The debugging prompt read no reference and answered with correct clarifying questions; the tool-timing prompt loaded the skill, read no reference and asked which run to time, so `measurement.md` routing was not exercised in this pass; (v) the forced-compaction recipe on `container`: a task turn, a resumed `/compact` turn whose `result.json` records a manual `compact_boundary` (40,266 → 2,896 tokens; the check exits 1 on the task turn), and a resumed turn that still named the command it had described before the compaction. The recipe first said to check this with `verify-run`, which refuses a run dir with more than one turn; it now reads the `/compact` turn's own `result.json`. Live spend about **$2.53**. **The 4.4.1 pass, 2026-10-05, on the release-prep commit `3e6fe443`** (a companion-skill-only release; same baseline and staged agent; host `claude` 2.1.289 for `protocol`): (i) `vitest list --staticParse=false` listed all 23 live tests with no `SKIPPED` warning when the token was exported, and 6 with four warnings when none was resolvable; (ii) the auto-memory check on all three tiers, `Tests  2 passed (2)` each, every init frame without `memory_paths`; (iii) the companion skill's router check, three prompts on `container`: the authoring prompt read `authoring.md` (and `scenario-schema.md`, searching `assertion-catalog.md`) and wrote a scenario that passes `lint` with no error or warning, the debugging prompt read `debugging.md` and answered with correct clarifying questions, and the tool-timing prompt read `measurement.md` (and `debugging.md`, which documents `trace`'s views) and answered with `trace … --view tool-durations`. The live suite itself was not re-run: this release changes no code that runs. Live spend about **$1.08**. **The 4.4.0 pass, 2026-10-05, on the release-prep commit `e9c516df`** (same baseline and staged agent; host `claude` 2.1.289 for `protocol`): (i) `vitest list --staticParse=false` listed all 23 live tests with no `SKIPPED` warning when the token was exported, and 6 with four warnings when none was resolvable; (ii) the live suite on `protocol`, `container` and `hostloop`: 23 passed, one pre-registered skip; (iii) the auto-memory check on all three tiers, `Tests  2 passed (2)` each; (iv) the companion skill's router check, three prompts on `container`: the authoring prompt read `authoring.md` and wrote a scenario that passes `lint`, the tool-timing prompt read `measurement.md` and answered with `trace … --view tool-durations`, and the debugging prompt read no reference and answered with correct clarifying questions, so `debugging.md` routing was not exercised in this pass; (v) the re-grade scrub-set check: a live run on the default runs root created `scrubset.key` beside the runs directory (mode 0600) and recorded `scrubSet` (`v: 1`) with the configured scrub literal absent from `result.json`; a re-grade of that run with an edited rubric line and a smaller scrub set was refused as `rubric_unverifiable` (exit 2) naming only the edited line, with no judge call and the run directory byte-identical; and a re-grade of a run recorded before 4.4 with an edited rubric line was refused with the `legacy` reason and the one-line remedy, again with no judge call and the run directory byte-identical. Live spend about **$2.83** recorded. **The 4.3.0 pass, on the release commit `8a5e479f`:** (1) the live suite on `protocol`, `container` and `hostloop`: 23 passed, one pre-registered skip (the auto-memory prerequisites check, which runs only under `COWORK_LIVE_REQUIRE=1`); before it ran, `vitest list --staticParse=false` listed all 23 with no `SKIPPED` warning, and 6 with four warnings when no token was resolvable, so a silent skip would have shown; (2) the auto-memory check on all three tiers, `Tests  2 passed (2)` each, every init frame without `memory_paths`; (3) the graders' pinned effort: every judged assert that called the judge in the live hillclimb acceptance runs (45, Opus 4.8: 25 `semantic_matches`, 20 `semantic_pairwise`; host `claude` 2.1.288 and 2.1.289) records `judgeTransport.effort: "high"`; the five pairwise asserts without a `judgeTransport` are the reference variant's own rows, which call no judge. **Earlier in the same release, on the release candidate's tree:** (4) a `microvm` smoke on a freshly booted VM (agent 2.1.286); (5) `boundary-check`, 6/6 constraints enforced; (6) an `eval` A/A at `--reps 4` on `container`, every row `no detectable change`, its report rebuilt byte-identical; (7) the companion skill's router check, three prompts on `container`: routing was correct on all three; one answer was wrong because a reference showed `artifact_text`'s `contains` without its list shape (fixed in the references, with a test that keeps every list-typed field shown as a list), and the same prompt's verdict went red on a false `host_path_leak`, because a reference spelled out the host roots that check looks for and the agent echoed it (fixed in the skill's files and in the check, which no longer counts a literal that comes from the plugin's or a local skill's own staged files); re-run after the fixes, the prompt routed correctly, wrote a scenario that passes `lint` and loads, and carried no `host_path_leak`; (8) the `hostloop` detached-agent kill check: the first pass found the workspace sidecar container, its `docker run` client and its network surviving SIGINT, which was fixed, and the re-run passed on a normal run, SIGINT to the harness process and SIGINT to the whole process group (exit 130, nothing left behind); (9) `vm delete`'s usage refusal; (10) `chat` under a real terminal, on `protocol`: Ctrl-C mid-turn exits 130 in about 2 s with nothing left running and no `result.json`; `/exit` followed by Ctrl-C exits 130 after writing the turn's `result.json`; Ctrl-C at the idle prompt ends the session normally (exit 0, `result.json` written); SIGHUP mid-turn exits 129 with nothing left running; and closing the terminal window mid-turn (run by hand) leaves no harness or agent process and no `result.json`. Total live spend about **$11.20**, including the re-runs. **Scope-out, so this is not read as more than it is.** (a) A live pass verifies observed behaviour, not the whole spawn contract by construction, and these checks are model-dependent, so a single red is evidence of model variance until a re-run says otherwise. (b) **CI does not live-validate anything.** Its "scenario suite (… live inference)" job is skipped as a whole without an `ANTHROPIC_API_KEY` repository secret, and none is set, so it shows as *skipped*. Before 2026-09 it instead ran with every real step skipped and reported *success*, so an older green check there means nothing ran. This note, not CI, is the live evidence. Separately and not a live matter: all four committed cassettes were **re-recorded** on 2026-10-03 against `2.19675.0` with auto-memory off, each on its original model: `example-pdf-skill` and `test/fixtures/tool-call-dispatch/dispatch-shell.cassette.json` (`container`, agent 2.1.286), `hostloop-computer-links` (`hostloop`, agent 2.1.286) and `example-multiselect-gate` (`protocol`, which runs the host CLI, here 2.1.287), about $0.75 in all. Their init frames carry no `memory_paths`. The recorder's own scrub (now applied to every recording) removed, across the four, the account's model menu, a subscription account's `rate_limit_info` (from `example-pdf-skill`, `dispatch-shell` and `hostloop-computer-links`) and the agent's sub-agent hand-back frame (from `dispatch-shell`). `verify-cassettes` exits 0 on all four, with one accepted `unscanned` entry (`example-pdf-skill`'s uploaded artifact body, too large to commit). Re-stamp this paragraph, naming the baseline, whenever a live pass is actually re-run.
 
 > The staged agent ELF is unchanged (2.1.181) across the 1.14271.0→1.15200.0 asar bump, and 2.1.187 across the 1.15200.0→1.15962.0 bump. The live scenario suite (`protocol` + `container` tiers) was re-run against the 1.15200.0 baseline; the 1.15962.0 bump was verified via asar analysis (content byte-identical: host-loop generator, system prompt, identity string, gates, and egress domains all unchanged) plus a full local test suite pass. The 1.15962.1→1.17377.1 bump moved the staged agent to **2.1.197** and added `api.claude.ai` to the egress allowlist; re-verified via `sync` (no unknown deltas) plus a manual asar spot-check of the reconstructed prompt content (substantively unchanged — see the Parity entry in CHANGELOG.md) and a full live scenario-suite pass (`protocol` + `container` tiers).
 

package/README.md

@@ -36,7 +36,7 @@ npm ci && npm run build
 node dist/cli.js replay examples/replays/example-pdf-skill.cassette.json
 ```
 
-(Installing globally — `npm install -g "cowork-harness@^4.7.0"` — gives you the `cowork-harness` CLI for your own
+(Installing globally — `npm install -g "cowork-harness@^4.7.1"` — gives you the `cowork-harness` CLI for your own
 scenarios and cassettes; the bundled example above also replays from a global install — see the `$(npm root -g)` path below.)
 
 Full setup → [Quick start](./docs/cli.md#quick-start).
@@ -49,8 +49,8 @@ Three ways to use this project. Each row is the whole hook — follow the link f
 
 | I want to… | Start here | Needs |
 |---|---|---|
-| **Run scenarios myself** from a terminal | **[docs/cli.md](./docs/cli.md)**<br><br>`npm i -g "cowork-harness@^4.7.0"`<br>`cowork-harness replay examples/replays/example-pdf-skill.cassette.json` | Node ≥ 22. The replay demo above is token-free and needs nothing else; live tiers above `protocol` need Docker + a staged agent binary |
-| **Have Claude Code drive it** for me | **[docs/companion-skill.md](./docs/companion-skill.md)**<br><br>`/plugin marketplace add yaniv-golan/cowork-harness`<br>`/plugin install cowork-harness@cowork-harness` | Claude Code. The skill self-bootstraps the CLI via `npx "cowork-harness@^4.7.0"` |
+| **Run scenarios myself** from a terminal | **[docs/cli.md](./docs/cli.md)**<br><br>`npm i -g "cowork-harness@^4.7.1"`<br>`cowork-harness replay examples/replays/example-pdf-skill.cassette.json` | Node ≥ 22. The replay demo above is token-free and needs nothing else; live tiers above `protocol` need Docker + a staged agent binary |
+| **Have Claude Code drive it** for me | **[docs/companion-skill.md](./docs/companion-skill.md)**<br><br>`/plugin marketplace add yaniv-golan/cowork-harness`<br>`/plugin install cowork-harness@cowork-harness` | Claude Code. The skill self-bootstraps the CLI via `npx "cowork-harness@^4.7.1"` |
 | **Gate my skill in CI** | **[docs/ci.md](./docs/ci.md)**<br><br>`- uses: yaniv-golan/cowork-harness@v4`<br>`  with: { command: replay, path: cassettes/ }` | Nothing for the token-free gate; the live lane needs a self-hosted runner with Docker + an agent binary |
 
 Improving a skill round by round with Claude Code's `/claude-api hillclimb` loop? The harness is its runner, with

package/SPEC.md

@@ -839,8 +839,8 @@ abridged to the fields most consumers branch on. The complete field list is
   "fidelity": "protocol|container|microvm|hostloop|cowork  (replay: \"replay:<f>\")",
   "baseline": "string",                          // platform baseline appVersion
   "result": "success" | "error",                // did the agent turn end without error (NOT "task completed")
-  "resultErrorKind?": "transport|agent|usage_limit", // when result==="error": a tail-end transport drop, a genuine agent/skill failure, or usage_limit (quota exhausted — is_error + HTTP 429 + a terminal usage-limit message; not the skill's fault, retry after reset)
-  "errorSource?": "spawn|protocol|exit|agent|result|no_result|timeout|decider_timeout|answer_channel_violation", // finer diagnostic detail alongside resultErrorKind; no_result = stream ended with no terminal event; timeout = the harness's own wall-clock limit fired; decider_timeout = an external decider channel did not answer a gate within its backstop (an unanswered-gate partial); answer_channel_violation = a run under `answer_channel: none` received a question or permission request anyway (refused, never answered)
+  "resultErrorKind?": "transport|agent|usage_limit", // when result==="error": a tail-end transport drop, a genuine agent/skill failure, or usage_limit (quota exhausted — is_error + HTTP 429 + a terminal usage-limit message, or a sub-agent's task that failed on a terminal usage-limit error, whatever the main loop did next; not the skill's fault)
+  "errorSource?": "spawn|protocol|exit|agent|result|no_result|timeout|decider_timeout|answer_channel_violation", // finer diagnostic detail alongside resultErrorKind; agent = a non-fatal agent-side event came first — including, with resultErrorKind usage_limit, a sub-agent's terminal quota error on a run whose main loop ended success (the usage_limit verdict message and a `subagent_usage_limit` row in decisions name the sub-agent); no_result = stream ended with no terminal event; timeout = the harness's own wall-clock limit fired; decider_timeout = an external decider channel did not answer a gate within its backstop (an unanswered-gate partial); answer_channel_violation = a run under `answer_channel: none` received a question or permission request anyway (refused, never answered)
   "resultSubtype?": "string",                    // the SDK result message's subtype verbatim (e.g. error_max_turns), pass-through diagnostic
   "stderrLogPath?": "string",                    // absolute path to the agent's full stderr log; live only, absent on replay
   "stalledOnQuestion?": bool,                     // H2/H3: ended on a question or closing request for input, no productive tool work after its last gate → `stalled` verdict fail unless allow_stall
@@ -953,7 +953,7 @@ prints the same exclusion warning the run prints. `verify-run` follows the same
 not exist or is a file, or a scenario file that does not load, is `usage`; a directory holding no completed
 run stays `runtime`. `answer` splits the same way: a directory or gate that is not there is `usage`; a gate
 request that exists but cannot be read or parsed, or an answer that cannot be written, is `runtime`.
-**Per-command exceptions:** `critique` **never gates on findings** — it exits `0` for any finding of any classification, and even when the task run it graded ERRORED (that is a finding about the skill, not a broken instrument). It exits `2` only for a usage error or an **instrument failure**: the turn was killed, the reflection protocol broke, or the evaluator was never invoked *or threw* — i.e. no critique was produced. Do not gate CI on `critique`; that inverts its design. `eval` (and `eval report`) exits `0` when the comparison completed — whatever drops the rows show, unless `--fail-on` was given; `1` for a drop at the `--fail-on` level (`possible` or `confirmed`; no gating without the flag — under `possible` a drop includes an `insufficient_refusals` row, one the candidate's excess `semantic_matches` refusals for unavailable evidence took below the rep threshold, which would otherwise hide a drop as `insufficient`), every row `insufficient`, or a judge model that differed across reps; `2` for a usage error or any refusal before the first run (an alias model, an eval dir inside a git work tree, identical arms, the answer-key guard, a scenario input a run would refuse, an unreachable `--fail-on confirmed`, a `--max-budget-usd` refusal — `error.code: "budget_exceeded"`); `3` when an arm snapshot could not be copied or failed its staging preflight. Its `--output-format json` envelope is `{tool, version, command:"eval", ok, evalDir, arms, pins, sections, summary, cost, stoppedEarly, error}`, with `ok` ⇔ exit `0`. `eval --dry-run` runs no agent and creates no eval dir: it exits `0` with the plan (`{tool, version, command:"eval", ok:true, dryRun:true, plan, budget?, error:null}`), `2` for any refusal the real eval would make before its first run (the budget refusal included), `3` as above, or when its temp dir is inside a git work tree or git cannot tell whether it is (set TMPDIR) — its arm snapshots go to a temp dir that is removed afterwards. `hillclimb run` exits `0` when every attempted (case, rep) was scored, `1` when any attempt failed (an `errors.jsonl` row, or a scored row whose trace or copies could not be written), the pass stopped mid-run (rows already written are kept), a baseline pass could not freeze a pairwise case's reference, or `summary.json` could not be written after the pass, and `2` for any refusal before spending: a usage error, the harness gate, no usable agent credential at a selected case's tier (`doctor`'s check, as on `eval`; at `protocol` a login only in the real config dir too, since hillclimb runs protocol with a managed config dir), an alias model, a session that does not declare exactly one `plugins.local_plugins` entry (or an inline session, or cases tuning different plugins), `on_unanswered: prompt`, the `protocol` tier without a managed config dir, a flow-dir or `_state.json` problem, split or duplicate case ids, an unknown `--case`, the answer-key guard, a `harness_paths` entry inside the tuned plugin, an unknown, untracked or ambiguous `--skill` (the message lists the plugin's skills), a tracked skill (or none) that differs from a `meta.skill_tracked` recorded by the variant's existing rows, a requested model or effort for a case that differs from the `meta.model_requested` / `meta.effort` the variant's rows for that case recorded (scored or error rows) (a row recording neither is held to its served `model`), an `--effort` level the pinned model does not offer (or any effort on a model with no effort selector, dated ids included), `xhigh` or `max` with `extended_thinking: false`, `extended_thinking: false` on a model whose baseline entry sets `disallowThinkingDisabled`, a missing or incomplete variant snapshot, a scenario input a run would refuse, a `semantic_pairwise` reference that is missing, damaged or exposed through a mount (the gate a run applies), a host `claude` that cannot run the judge or LLM decider isolated (checked when a scenario uses `semantic_matches` or `semantic_pairwise`, or `on_unanswered: llm` with no decider channel, as on `eval`), or a decider with `--concurrency` above 1, or one scenario metric id declared differently in two scenarios, or declared differently from the signature the flow's existing rows carry for it in `meta.metric_sigs` (adding or removing a metric is allowed; removing one warns); `--dry-run` exits `0` unless such a refusal fires; the harness gate does not refuse a dry run, which reports its status instead. Under `--case`, the per-case refusals (the session, its model pins, `on_unanswered: prompt`, the `protocol` tier, a scenario input, a `semantic_pairwise` reference, the isolation check, and the mounts the answer-key guard checks) cover the selected cases only; the flow-level ones cover every case: every scenario file must parse, every case's baseline must load, the harness gate hashes every case's scenario and session file (an unselected case's inputs it cannot read — missing, a directory, unreadable — or that its unparseable session cannot name, are left out of it; the gate hashes each path's name, so the sha still moves), the answer-key guard keeps every case's scenario and session file unreadable, and the one-plugin rule covers every case whose session parses (an unselected one whose session is inline or does not parse is skipped, with a note). `hillclimb regrade --case` follows the same rule, with the same harness sha. `--timeout-s` bounds the whole attempt: the agent phase through the scenario's `timeout_ms` (lowered to it), and the judge phase through a deadline no judge may start after; an attempt whose agent phase reaches its bound, or whose judge phase reaches the deadline, is an `errors.jsonl` row (`timeout`), never a scored one. So is an attempt whose agent did not send the requested effort: its own session transcript shows a main-loop assistant message sent with another effort, with a value that is not an effort level, or with none (even when the agent then failed; a model with no effort selector may send none), or, on an otherwise valid run whose main loop answered, records no main-loop assistant message or cannot be found (read from the run's config root, or the session's pinned `plugins.config_dir` on hostloop and protocol, by the run's session id): `serving_substitution` with `meta.failure_rule: "effort_not_sent"`. Its envelope's `ok` ⇔ exit `0`; a refusal or a mid-run stop is the shared error envelope, its reason in `error.message` (category `usage` for a refusal, `runtime` for a stop), with the payload keys (`flow`, `variant`, `scheduled`, `scored`, `failed`, `exitCode`, and a dry run's `dryRun`) riding on it; `failed` counts failed (case, rep) slots, so a failed reference freeze or `summary.json` write exits `1` without adding to it. Its `--dry-run` `plan.cost` is the same object `eval --dry-run` emits at `plan.cost` (covered keys `jobs`, `meanUsd`, `p50Usd`, `p95Usd`, `worstObservedUsd`, `lowerBound`, `unpriced`, `pricedRuns`, `thinnest`; `p95Usd` and `worstObservedUsd` are pessimistic figures, never bounds, and `worstObservedUsd` is not the `--max-budget-usd` gate's figure), priced on `eval --dry-run`'s basis exactly — the scenario's runs on this machine at its effective tier and baseline, turn 1, every `hillclimb:` run excluded — so each covered key means the same on both commands. `jobs` is the slots the pass would still run after resuming. A scored row records how its judge ran in `meta.judge_transport` (`assertions[].judgeTransport`'s shape, `{isolation, cliVersion?, strictMcp?}`), or `meta.judge_transports` listing each distinct one when its asserts were judged differently; neither is present when no host judge recorded one, so a round judged under a different isolation can be told apart from a regression. `skill_invoked` is `1` or `0` for whether the run invoked the tracked skill; a blank cell means not measured (no tracked skill, or a record that could not tell), never "not invoked". The tracked skill is matched by the id the agent registers for it, `<plugin>:<name>`, the name being a skill directory's name, or a `SKILL.md`'s frontmatter `name` where the agent reads one (else its directory's name), with every character outside `[a-zA-Z0-9_-]` replaced by `-` (`skills/my.skill/` registers `<plugin>:my-skill`), among the skills the agent loads: `skills/*/` whenever `skills/` exists, each path in the plugin manifest's `skills` field (a string or an array of paths relative to the plugin root, each `.` or starting with `./`, as the agent's manifest schema requires, and each a skill directory or a directory of them), and a root `SKILL.md` only when the manifest has no `skills` field at all (`"skills": []` is a field, naming nothing) and there is no `skills/` dir; a `SKILL.md` that is not a regular file or is over 1 MiB is skipped, as the agent skips it. A plugin with one such skill is tracked; with several, `--skill <name>` (a skill directory's name or its registered name) picks one, and without it the column is omitted. Each scored row records the tracked id in `meta.skill_tracked`, absent when the column is omitted. A `--skill` selection enters the harness sha as `skill:<registered name>` and `--approve-harness` records it as `_state.json` `harness_skill`, so a changed, added or dropped `--skill` is a harness-gate refusal that names the change; without `--skill` the sha is unchanged. `--skill` is not remembered between passes: the runner command must pass the same one every time. A scenario's declared `metrics` are grade keys on every scored row of the flow, over the union of the cases' declarations: `<id>` only when measured — an unmeasured metric is omitted, never 0 — and `<id>_present` (1 measured; 0 on a case that does not declare it, on an agent-caused failure, or when unavailable, with the reason in `meta.metrics_unavailable`). `hillclimb state-template` declares each `<id>_present` with the other `_present` companions after `pass`, and each metric last as `{id, kind: "float", label, better}`, plus `scale` and `min` when the scenario declares them; every scored row records each flow metric's declaration signature in `meta.metric_sigs`. `hillclimb check`'s headroom takes a float headline's good end from `scale` (higher is better) or `min`, 0 when absent (lower is better), and without a `scale` says so instead of computing one; a row that lacks a declared metric (its `meta.metric_sigs` lacks the id) is a note, not an error — it predates the metric when a row after it carries it, and no scenario declares the metric any more when it comes after the last row that does (variants in order, then file order) — a float outside `[min, scale]` is a warning, and so are a case's rows carrying more than one `meta.assert_sig` (graded under different assertion sets) and rows whose `meta.assert_sig` is not their scenario's now — the scenario target `hillclimb check [<scenario.yaml | dir/>]` was given, else the scenario files the last `--approve-harness` hashed (the `.yaml` keys of `_state.json` `harness_files` whose stem is a case's id), else those `harness_paths` lists (a recorded file is a case's scenario when it has a `prompt:` and its stem is the case's id; a note when nothing is recorded, or names a recorded scenario not found from the current directory or that does not parse, or a case the target holds no scenario for, and skips that case; the remedy's `regrade` target is one that hashes exactly the files the flow was approved over — the scenarios' directory when it holds exactly them, else the case's own file or its directory, else `<scenarios>`). `hillclimb check` exits `0` clean, `1` on any error finding (no warning or note changes it), `2` usage (a target that does not load included); `hillclimb state-template` exits `0`, or `2` on usage or a refusal (a metric id declared differently in two scenarios included); `hillclimb regrade` re-evaluates every selected row from its kept run before any judge call — its deterministic asserts and `expect_denied` hosts with `verify-run`'s evaluation (a changed, added or removed one applied in a default re-grade), its metrics re-measured there (a row no judge re-grades included: a case with no judged assert, an agent-failed row — signatures and `<id>_present: 0`, never a value — and a fill row that needs no comparison; every rewritten row of a case that declares a metric, a re-judged one included, is marked `meta.regrade_remeasured: true`, and each variant's payload entry counts the rows re-measured with no judge call in `remeasured` and those re-evaluated in `reevaluated`); a row whose re-evaluation changes nothing stays byte for byte. It exits `0` when every selected row was rewritten or there was nothing to do, `1` when a row was listed instead (no scenario file for its case in the target, without `--case`; its kept run gone or refused, its kept work dir gone while its case declares a metric or a filesystem assert needs it, an assert the recorded `workspace_fixture` satisfies on its own; an assert whose literal the run scrubbed, on a row graded under another assertion list, since an edit inside a scrubbed literal cannot be applied; a re-judge of an assert scrubbed in the run that this process's scrub does not reproduce, without `--allow-scrub-change`; a part of a row's judge input not proven scrubbed with a set covering its run's, without `--allow-scrub-change`; judged evidence that cannot be recomposed; judged evidence that changed since its grade, without `--rejudge`; current evidence less redacted than the graded document, in both modes unless `--allow-scrub-change`; a scenario with no judged assert left to re-grade; in a fill an assert list, a rubric or a deterministic outcome that does not line up with the scenario as graded, a kept outcome judged against a reference that has changed since, an unreadable regrade file, or a fill that would move `pass`; its re-grade judge-invalid, misaligned or stopped; an open `judge_invalid` slot, which `hillclimb run <target> --flow <dir> --variant <v> --case <id> --reps <rep+1>` re-runs — it is never moved into `results.jsonl`; an assert unchanged since the run that re-evaluates differently from the kept run is never listed: the row keeps the run's own outcome and records `meta.regrade_kept_live`), `1` also for a failure after the first judge call (naming the variants already rewritten), and `2` on usage or a refusal before any judge call (a host `claude` that cannot run the judge isolated, the gate, a held lock, a scenario metric declared differently in two scenarios or from the flow's rows' `meta.metric_sigs`, as on `run`, an evidence refusal anywhere — nothing is written then); `hillclimb freeze-ref` exits `0` when no case was refused (an entry already complete is reported, not refused), `1` when a case was (no good row whose run is under the runs root and delivered its output, a damaged entry, a reference frozen for a different prompt, a missing compose key whose recorded run is gone, a missing compose key whose run's scrub set this process cannot prove it covers (re-run the variant), a store write that failed, or a run whose judged document cannot be composed, differs from the one its live judge read, or has no live fingerprint to check it against), and `2` on usage (a bad `--variant`, no flow dir, a variant with no `results.jsonl`, no selected case with `semantic_pairwise`, its variant's lock held by a live run). `lint` exits `127` when `python3` is missing (spawn error), and `1` — never `0` — when the scenario loader rejected a file but its findings could not be handed to the linter (an unwritable temp directory); `replay` exits
+**Per-command exceptions:** `critique` **never gates on findings** — it exits `0` for any finding of any classification, and even when the task run it graded ERRORED (that is a finding about the skill, not a broken instrument) — except a task run that hit the account's usage limit (its own result or a sub-agent's): nothing honest is left to grade, so no reflection is spawned and it exits `2`. It exits `2` only when no critique was produced: a usage error, a task turn that hit the account's usage limit, or an **instrument failure** (the turn was killed, the reflection protocol broke, or the evaluator was never invoked *or threw*). Do not gate CI on `critique`; that inverts its design. `eval` (and `eval report`) exits `0` when the comparison completed — whatever drops the rows show, unless `--fail-on` was given; `1` for a drop at the `--fail-on` level (`possible` or `confirmed`; no gating without the flag — under `possible` a drop includes an `insufficient_refusals` row, one the candidate's excess `semantic_matches` refusals for unavailable evidence took below the rep threshold, which would otherwise hide a drop as `insufficient`), every row `insufficient`, or a judge model that differed across reps; `2` for a usage error or any refusal before the first run (an alias model, an eval dir inside a git work tree, identical arms, the answer-key guard, a scenario input a run would refuse, an unreachable `--fail-on confirmed`, a `--max-budget-usd` refusal — `error.code: "budget_exceeded"`); `3` when an arm snapshot could not be copied or failed its staging preflight. Its `--output-format json` envelope is `{tool, version, command:"eval", ok, evalDir, arms, pins, sections, summary, cost, stoppedEarly, error}`, with `ok` ⇔ exit `0`. `eval --dry-run` runs no agent and creates no eval dir: it exits `0` with the plan (`{tool, version, command:"eval", ok:true, dryRun:true, plan, budget?, error:null}`), `2` for any refusal the real eval would make before its first run (the budget refusal included), `3` as above, or when its temp dir is inside a git work tree or git cannot tell whether it is (set TMPDIR) — its arm snapshots go to a temp dir that is removed afterwards. `hillclimb run` exits `0` when every attempted (case, rep) was scored, `1` when any attempt failed (an `errors.jsonl` row, or a scored row whose trace or copies could not be written), the pass stopped mid-run (rows already written are kept), a baseline pass could not freeze a pairwise case's reference, or `summary.json` could not be written after the pass, and `2` for any refusal before spending: a usage error, the harness gate, no usable agent credential at a selected case's tier (`doctor`'s check, as on `eval`; at `protocol` a login only in the real config dir too, since hillclimb runs protocol with a managed config dir), an alias model, a session that does not declare exactly one `plugins.local_plugins` entry (or an inline session, or cases tuning different plugins), `on_unanswered: prompt`, the `protocol` tier without a managed config dir, a flow-dir or `_state.json` problem, split or duplicate case ids, an unknown `--case`, the answer-key guard, a `harness_paths` entry inside the tuned plugin, an unknown, untracked or ambiguous `--skill` (the message lists the plugin's skills), a tracked skill (or none) that differs from a `meta.skill_tracked` recorded by the variant's existing rows, a requested model or effort for a case that differs from the `meta.model_requested` / `meta.effort` the variant's rows for that case recorded (scored or error rows) (a row recording neither is held to its served `model`), an `--effort` level the pinned model does not offer (or any effort on a model with no effort selector, dated ids included), `xhigh` or `max` with `extended_thinking: false`, `extended_thinking: false` on a model whose baseline entry sets `disallowThinkingDisabled`, a missing or incomplete variant snapshot, a scenario input a run would refuse, a `semantic_pairwise` reference that is missing, damaged or exposed through a mount (the gate a run applies), a host `claude` that cannot run the judge or LLM decider isolated (checked when a scenario uses `semantic_matches` or `semantic_pairwise`, or `on_unanswered: llm` with no decider channel, as on `eval`), or a decider with `--concurrency` above 1, or one scenario metric id declared differently in two scenarios, or declared differently from the signature the flow's existing rows carry for it in `meta.metric_sigs` (adding or removing a metric is allowed; removing one warns); `--dry-run` exits `0` unless such a refusal fires; the harness gate does not refuse a dry run, which reports its status instead. Under `--case`, the per-case refusals (the session, its model pins, `on_unanswered: prompt`, the `protocol` tier, a scenario input, a `semantic_pairwise` reference, the isolation check, and the mounts the answer-key guard checks) cover the selected cases only; the flow-level ones cover every case: every scenario file must parse, every case's baseline must load, the harness gate hashes every case's scenario and session file (an unselected case's inputs it cannot read — missing, a directory, unreadable — or that its unparseable session cannot name, are left out of it; the gate hashes each path's name, so the sha still moves), the answer-key guard keeps every case's scenario and session file unreadable, and the one-plugin rule covers every case whose session parses (an unselected one whose session is inline or does not parse is skipped, with a note). `hillclimb regrade --case` follows the same rule, with the same harness sha. `--timeout-s` bounds the whole attempt: the agent phase through the scenario's `timeout_ms` (lowered to it), and the judge phase through a deadline no judge may start after; an attempt whose agent phase reaches its bound, or whose judge phase reaches the deadline, is an `errors.jsonl` row (`timeout`), never a scored one. So is an attempt whose agent did not send the requested effort: its own session transcript shows a main-loop assistant message sent with another effort, with a value that is not an effort level, or with none (even when the agent then failed; a model with no effort selector may send none), or, on an otherwise valid run whose main loop answered, records no main-loop assistant message or cannot be found (read from the run's config root, or the session's pinned `plugins.config_dir` on hostloop and protocol, by the run's session id): `serving_substitution` with `meta.failure_rule: "effort_not_sent"`. Its envelope's `ok` ⇔ exit `0`; a refusal or a mid-run stop is the shared error envelope, its reason in `error.message` (category `usage` for a refusal, `runtime` for a stop), with the payload keys (`flow`, `variant`, `scheduled`, `scored`, `failed`, `exitCode`, and a dry run's `dryRun`) riding on it; `failed` counts failed (case, rep) slots, so a failed reference freeze or `summary.json` write exits `1` without adding to it. Its `--dry-run` `plan.cost` is the same object `eval --dry-run` emits at `plan.cost` (covered keys `jobs`, `meanUsd`, `p50Usd`, `p95Usd`, `worstObservedUsd`, `lowerBound`, `unpriced`, `pricedRuns`, `thinnest`; `p95Usd` and `worstObservedUsd` are pessimistic figures, never bounds, and `worstObservedUsd` is not the `--max-budget-usd` gate's figure), priced on `eval --dry-run`'s basis exactly — the scenario's runs on this machine at its effective tier and baseline, turn 1, every `hillclimb:` run excluded — so each covered key means the same on both commands. `jobs` is the slots the pass would still run after resuming. A scored row records how its judge ran in `meta.judge_transport` (`assertions[].judgeTransport`'s shape, `{isolation, cliVersion?, strictMcp?}`), or `meta.judge_transports` listing each distinct one when its asserts were judged differently; neither is present when no host judge recorded one, so a round judged under a different isolation can be told apart from a regression. `skill_invoked` is `1` or `0` for whether the run invoked the tracked skill; a blank cell means not measured (no tracked skill, or a record that could not tell), never "not invoked". The tracked skill is matched by the id the agent registers for it, `<plugin>:<name>`, the name being a skill directory's name, or a `SKILL.md`'s frontmatter `name` where the agent reads one (else its directory's name), with every character outside `[a-zA-Z0-9_-]` replaced by `-` (`skills/my.skill/` registers `<plugin>:my-skill`), among the skills the agent loads: `skills/*/` whenever `skills/` exists, each path in the plugin manifest's `skills` field (a string or an array of paths relative to the plugin root, each `.` or starting with `./`, as the agent's manifest schema requires, and each a skill directory or a directory of them), and a root `SKILL.md` only when the manifest has no `skills` field at all (`"skills": []` is a field, naming nothing) and there is no `skills/` dir; a `SKILL.md` that is not a regular file or is over 1 MiB is skipped, as the agent skips it. A plugin with one such skill is tracked; with several, `--skill <name>` (a skill directory's name or its registered name) picks one, and without it the column is omitted. Each scored row records the tracked id in `meta.skill_tracked`, absent when the column is omitted. A `--skill` selection enters the harness sha as `skill:<registered name>` and `--approve-harness` records it as `_state.json` `harness_skill`, so a changed, added or dropped `--skill` is a harness-gate refusal that names the change; without `--skill` the sha is unchanged. `--skill` is not remembered between passes: the runner command must pass the same one every time. A scenario's declared `metrics` are grade keys on every scored row of the flow, over the union of the cases' declarations: `<id>` only when measured — an unmeasured metric is omitted, never 0 — and `<id>_present` (1 measured; 0 on a case that does not declare it, on an agent-caused failure, or when unavailable, with the reason in `meta.metrics_unavailable`). `hillclimb state-template` declares each `<id>_present` with the other `_present` companions after `pass`, and each metric last as `{id, kind: "float", label, better}`, plus `scale` and `min` when the scenario declares them; every scored row records each flow metric's declaration signature in `meta.metric_sigs`. `hillclimb check`'s headroom takes a float headline's good end from `scale` (higher is better) or `min`, 0 when absent (lower is better), and without a `scale` says so instead of computing one; a row that lacks a declared metric (its `meta.metric_sigs` lacks the id) is a note, not an error — it predates the metric when a row after it carries it, and no scenario declares the metric any more when it comes after the last row that does (variants in order, then file order) — a float outside `[min, scale]` is a warning, and so are a case's rows carrying more than one `meta.assert_sig` (graded under different assertion sets) and rows whose `meta.assert_sig` is not their scenario's now — the scenario target `hillclimb check [<scenario.yaml | dir/>]` was given, else the scenario files the last `--approve-harness` hashed (the `.yaml` keys of `_state.json` `harness_files` whose stem is a case's id), else those `harness_paths` lists (a recorded file is a case's scenario when it has a `prompt:` and its stem is the case's id; a note when nothing is recorded, or names a recorded scenario not found from the current directory or that does not parse, or a case the target holds no scenario for, and skips that case; the remedy's `regrade` target is one that hashes exactly the files the flow was approved over — the scenarios' directory when it holds exactly them, else the case's own file or its directory, else `<scenarios>`). `hillclimb check` exits `0` clean, `1` on any error finding (no warning or note changes it), `2` usage (a target that does not load included); `hillclimb state-template` exits `0`, or `2` on usage or a refusal (a metric id declared differently in two scenarios included); `hillclimb regrade` re-evaluates every selected row from its kept run before any judge call — its deterministic asserts and `expect_denied` hosts with `verify-run`'s evaluation (a changed, added or removed one applied in a default re-grade), its metrics re-measured there (a row no judge re-grades included: a case with no judged assert, an agent-failed row — signatures and `<id>_present: 0`, never a value — and a fill row that needs no comparison; every rewritten row of a case that declares a metric, a re-judged one included, is marked `meta.regrade_remeasured: true`, and each variant's payload entry counts the rows re-measured with no judge call in `remeasured` and those re-evaluated in `reevaluated`); a row whose re-evaluation changes nothing stays byte for byte. It exits `0` when every selected row was rewritten or there was nothing to do, `1` when a row was listed instead (no scenario file for its case in the target, without `--case`; its kept run gone or refused, its kept work dir gone while its case declares a metric or a filesystem assert needs it, an assert the recorded `workspace_fixture` satisfies on its own; an assert whose literal the run scrubbed, on a row graded under another assertion list, since an edit inside a scrubbed literal cannot be applied; a re-judge of an assert scrubbed in the run that this process's scrub does not reproduce, without `--allow-scrub-change`; a part of a row's judge input not proven scrubbed with a set covering its run's, without `--allow-scrub-change`; judged evidence that cannot be recomposed; judged evidence that changed since its grade, without `--rejudge`; current evidence less redacted than the graded document, in both modes unless `--allow-scrub-change`; a scenario with no judged assert left to re-grade; in a fill an assert list, a rubric or a deterministic outcome that does not line up with the scenario as graded, a kept outcome judged against a reference that has changed since, an unreadable regrade file, or a fill that would move `pass`; its re-grade judge-invalid, misaligned or stopped; an open `judge_invalid` slot, which `hillclimb run <target> --flow <dir> --variant <v> --case <id> --reps <rep+1>` re-runs — it is never moved into `results.jsonl`; an assert unchanged since the run that re-evaluates differently from the kept run is never listed: the row keeps the run's own outcome and records `meta.regrade_kept_live`), `1` also for a failure after the first judge call (naming the variants already rewritten), and `2` on usage or a refusal before any judge call (a host `claude` that cannot run the judge isolated, the gate, a held lock, a scenario metric declared differently in two scenarios or from the flow's rows' `meta.metric_sigs`, as on `run`, an evidence refusal anywhere — nothing is written then); `hillclimb freeze-ref` exits `0` when no case was refused (an entry already complete is reported, not refused), `1` when a case was (no good row whose run is under the runs root and delivered its output, a damaged entry, a reference frozen for a different prompt, a missing compose key whose recorded run is gone, a missing compose key whose run's scrub set this process cannot prove it covers (re-run the variant), a store write that failed, or a run whose judged document cannot be composed, differs from the one its live judge read, or has no live fingerprint to check it against), and `2` on usage (a bad `--variant`, no flow dir, a variant with no `results.jsonl`, no selected case with `semantic_pairwise`, its variant's lock held by a live run). `lint` exits `127` when `python3` is missing (spawn error), and `1` — never `0` — when the scenario loader rejected a file but its findings could not be handed to the linter (an unwritable temp directory); `replay` exits
 `2` on a **whole-cassette operational failure** — anything `readCassette` rejects (unreadable, invalid
 shape, unsupported version, unrecognized assertion key) or any per-file throw, plus the batch loop's
 own source-resolution failures (`--assert-from`/`--reassert` drift, scenario-parse errors, `--write`