cowork-harness 4.7.0 → 4.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (83) hide show
  1. package/.claude/skills/cowork-harness/SKILL.md +9 -6
  2. package/.claude/skills/cowork-harness/references/assertion-catalog-agents-skills-budgets.md +1 -1
  3. package/.claude/skills/cowork-harness/references/assertion-catalog-gates-hooks-modifiers.md +1 -1
  4. package/.claude/skills/cowork-harness/references/assertion-catalog-outcome-files-tools.md +5 -5
  5. package/.claude/skills/cowork-harness/references/assertion-catalog.md +4 -4
  6. package/.claude/skills/cowork-harness/references/assertions-guide.md +1 -1
  7. package/.claude/skills/cowork-harness/references/authoring.md +14 -1
  8. package/.claude/skills/cowork-harness/references/ci-recipe.md +8 -6
  9. package/.claude/skills/cowork-harness/references/critique.md +63 -8
  10. package/.claude/skills/cowork-harness/references/debugging.md +10 -4
  11. package/.claude/skills/cowork-harness/references/eval.md +4 -2
  12. package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +1 -1
  13. package/.claude/skills/cowork-harness/references/gotchas.md +1 -1
  14. package/.claude/skills/cowork-harness/references/hillclimb-recipe.md +1 -1
  15. package/.claude/skills/cowork-harness/references/hillclimb.md +21 -1
  16. package/.claude/skills/cowork-harness/references/measurement.md +1 -1
  17. package/.claude/skills/cowork-harness/references/run-record-replay.md +6 -5
  18. package/.claude/skills/cowork-harness/references/scenario-schema.md +10 -4
  19. package/.claude/skills/cowork-harness/references/semantic-judging.md +1 -1
  20. package/.claude/skills/cowork-harness/references/task-recipes.md +7 -5
  21. package/.claude/skills/cowork-harness/scripts/assertion-keys.json +1 -0
  22. package/.claude/skills/cowork-harness/scripts/scenario.py +67 -3
  23. package/CHANGELOG.md +202 -0
  24. package/DESIGN.md +1 -1
  25. package/README.md +8 -6
  26. package/SPEC.md +17 -11
  27. package/dist/answer-channel.js +1 -1
  28. package/dist/cli.js +3 -1
  29. package/dist/critique/command.js +322 -25
  30. package/dist/critique/compare.js +392 -0
  31. package/dist/critique/corpus-digest.js +55 -0
  32. package/dist/critique/evaluator.js +22 -1
  33. package/dist/critique/evidence.js +3 -0
  34. package/dist/critique/git-target.js +176 -0
  35. package/dist/critique/package-evidence.js +85 -5
  36. package/dist/critique/reply-repair.js +61 -0
  37. package/dist/critique/scrub-artifacts.js +49 -1
  38. package/dist/critique/summary.js +134 -0
  39. package/dist/eval/classify.js +5 -3
  40. package/dist/eval/snapshot.js +22 -21
  41. package/dist/hostloop/remote-devices-handler.js +115 -0
  42. package/dist/prompt.js +58 -5
  43. package/dist/run/cassette.js +81 -13
  44. package/dist/run/chat-result.js +1 -0
  45. package/dist/run/execute.js +79 -10
  46. package/dist/run/input-host-paths.js +5 -1
  47. package/dist/run/lane-notice.js +9 -0
  48. package/dist/run/run.js +81 -2
  49. package/dist/run/tier-vacuous-tools.js +41 -3
  50. package/dist/run/verdict.js +12 -3
  51. package/dist/run/verify-context.js +7 -0
  52. package/dist/runtime/argv.js +1 -1
  53. package/dist/runtime/container.js +32 -8
  54. package/dist/runtime/device-container.js +400 -0
  55. package/dist/runtime/hostloop.js +8 -6
  56. package/dist/session.js +18 -3
  57. package/dist/types.js +16 -1
  58. package/dist/usage-limit.js +28 -0
  59. package/docs/cassette.md +11 -6
  60. package/docs/ci.md +3 -2
  61. package/docs/cli.md +10 -8
  62. package/docs/companion-skill.md +2 -2
  63. package/docs/critique.md +187 -37
  64. package/docs/eval.md +19 -1
  65. package/docs/fidelity-gaps.md +83 -8
  66. package/docs/hillclimb.md +20 -0
  67. package/docs/invariants.md +1 -1
  68. package/docs/run-status.md +2 -1
  69. package/docs/scenario.md +39 -12
  70. package/docs/session.md +3 -2
  71. package/examples/replays/README.md +1 -1
  72. package/llms.txt +1 -1
  73. package/package.json +1 -1
  74. package/python/test_scenario_lint.py +60 -1
  75. package/schema/cassette.v17.json +471 -0
  76. package/schema/critique-compare.json +78 -0
  77. package/schema/critique-report.json +104 -3
  78. package/schema/critique-summary.json +122 -0
  79. package/schema/run-result.json +94 -4
  80. package/schema/scenario.schema.json +32 -1
  81. package/schema/session.schema.json +0 -1
  82. package/scripts/check-versions.ts +2 -1
  83. package/scripts/gen-schema.ts +13 -0
@@ -3,8 +3,8 @@ name: cowork-harness
3
3
  description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, asserting artifacts, egress, or sub-agent dispatch, measuring how long each tool call took (toolDurations / trace), or debugging a failed run or verdict from its result.json or transcript. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. Also when iterating or hardening a skill across fixes, or grounding a skill's self-critique against its own run evidence — including a document-analysis skill (cap table, deck, financial model, transcript) that needs an uploaded file attached to be critiqued at all. Also for comparing two versions of a skill before merging an edit — did it make answers worse? (`eval`: paired A/B, pinned models) — or improving one round by round (`hillclimb`, the `/claude-api hillclimb` runner). NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold / critique / stats / eval / hillclimb commands and the session-vs-scenario split.
4
4
  metadata:
5
5
  author: cowork-harness
6
- version: 4.7.0
7
- tracks-harness: cowork-harness 4.7.0 (baseline desktop-2.31226.1)
6
+ version: 4.8.0
7
+ tracks-harness: cowork-harness 4.8.0 (baseline desktop-2.31226.1)
8
8
  ---
9
9
 
10
10
  # cowork-harness
@@ -26,7 +26,7 @@ allowlist). This skill exists mostly to keep you out of those traps — the *Inv
26
26
  full landmine catalog in [`references/gotchas.md`](references/gotchas.md) are the highest-value part.
27
27
  Read them.
28
28
 
29
- > **Version note:** the facts and `file:line` pointers here track `cowork-harness 4.7.0` (baseline
29
+ > **Version note:** the facts and `file:line` pointers here track `cowork-harness 4.8.0` (baseline
30
30
  > `desktop-2.31226.1`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
31
31
  > `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
32
32
 
@@ -43,7 +43,7 @@ Before the first command, confirm the CLI is reachable and **fail loud** (never
43
43
 
44
44
  - **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
45
45
  - **Replay-only? Skip `doctor`.** Replaying committed cassettes needs no Docker, no staged agent, and no token — and every tier's `doctor` validates the auth token (the live tiers also Docker + the staged agent), so a ✗ there is expected, not a blocker. Go straight to `cowork-harness replay <cassette>`.
46
- - **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 4.7.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@^4.7.0" <cmd>` (Node ≥ 22), or install once with `npm i -g "cowork-harness@^4.7.0"`. **Pin `@^4.7.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
46
+ - **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 4.8.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@^4.8.0" <cmd>` (Node ≥ 22), or install once with `npm i -g "cowork-harness@^4.8.0"`. **Pin `@^4.8.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
47
47
 
48
48
  This skill documents the CURRENT surface, not release history. If `cowork-harness --version` is
49
49
  OLDER than the floor, the per-release record of what you are missing is [CHANGELOG.md](https://github.com/yaniv-golan/cowork-harness/blob/main/CHANGELOG.md)
@@ -90,7 +90,9 @@ CI-grade scenario, and the post-hoc debug loop; the rest are narrower tools that
90
90
  `report.costUsd.totalUsd`. Reach for it when you want **findings**. **For "what does this skill
91
91
  **DO**" — routing, artifact location, narration — use `skill` instead**: no evaluator, a fraction of
92
92
  the cost, and it answers that question directly. Report and evidence-package shapes:
93
- `references/critique.md`.
93
+ `references/critique.md` — which also covers grading a commit (`git:<ref>:<path>`), the
94
+ "has the skill changed?" hashes, the public `--summary-out` file, and `critique --compare` for
95
+ reading several critiques side by side.
94
96
  - **Multi-turn / interactive reproduction** → `cowork-harness chat` (interactive; gates answered at the
95
97
  TTY, **not** an asserted test — see *Debugging with `chat`* in `references/debugging.md`).
96
98
  **"Interactive" splits two ways — don't take the wrong branch.** Want to answer gates yourself *and*
@@ -132,7 +134,8 @@ behind each, is [`references/gotchas.md`](references/gotchas.md).
132
134
  7. **The tier decides what exists.** `protocol` has no sandbox and no egress, tool names differ per tier
133
135
  (`container` serves `mcp__workspace__web_fetch`, not `WebFetch`), and every tier models Cowork's
134
136
  local lane only; new Pro and Max tasks do not use it from 2026-10-06. Behaviour-shaped results transfer;
135
- path, mount, delivery and egress results do not.
137
+ path, mount, delivery and egress results do not. `device_bridge: true` on `lane: remote` serves the cloud
138
+ task's `device_bash` (see `references/authoring.md`).
136
139
  8. **A WARN signal never blocks a green.** Read the verdict signals after every run
137
140
  (`prompt_asset_missing`, `undelivered_deliverables`, `model_fallback`, …).
138
141
 
@@ -1,6 +1,6 @@
1
1
  # Assertion catalog: sub-agents, skills and connectors, budgets, tasks, delivery
2
2
 
3
- Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). Part 2 of 3 of the per-key table. The conventions every key shares (an item with several
3
+ Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). Part 2 of 3 of the per-key table. The conventions every key shares (an item with several
4
4
  keys is an AND; glob vs regex name matching) and the verdict-signal table are in
5
5
  [`assertion-catalog.md`](./assertion-catalog.md); the other parts are [`assertion-catalog-outcome-files-tools.md`](./assertion-catalog-outcome-files-tools.md), [`assertion-catalog-gates-hooks-modifiers.md`](./assertion-catalog-gates-hooks-modifiers.md).
6
6
 
@@ -1,6 +1,6 @@
1
1
  # Assertion catalog: gates, hooks, path denial, verdict modifiers, egress, judged keys
2
2
 
3
- Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). Part 3 of 3 of the per-key table. The conventions every key shares (an item with several
3
+ Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). Part 3 of 3 of the per-key table. The conventions every key shares (an item with several
4
4
  keys is an AND; glob vs regex name matching) and the verdict-signal table are in
5
5
  [`assertion-catalog.md`](./assertion-catalog.md); the other parts are [`assertion-catalog-outcome-files-tools.md`](./assertion-catalog-outcome-files-tools.md), [`assertion-catalog-agents-skills-budgets.md`](./assertion-catalog-agents-skills-budgets.md).
6
6
 
@@ -1,6 +1,6 @@
1
1
  # Assertion catalog: outcome, transcript, files and artifacts, tools, tool results
2
2
 
3
- Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). Part 1 of 3 of the per-key table. The conventions every key shares (an item with several
3
+ Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). Part 1 of 3 of the per-key table. The conventions every key shares (an item with several
4
4
  keys is an AND; glob vs regex name matching) and the verdict-signal table are in
5
5
  [`assertion-catalog.md`](./assertion-catalog.md); the other parts are [`assertion-catalog-agents-skills-budgets.md`](./assertion-catalog-agents-skills-budgets.md), [`assertion-catalog-gates-hooks-modifiers.md`](./assertion-catalog-gates-hooks-modifiers.md).
6
6
 
@@ -11,12 +11,12 @@ keys is an AND; glob vs regex name matching) and the verdict-signal table are in
11
11
  | `transcript_not_contains: <str>` | it does not **Sees top-level `assistant_text` only — it excludes every `tool_use`/`tool_result`**, so text emitted inside a tool call (a gate question, an option label or description) can never match; use `question_context`/`question_asked` or `tool_result_contains` for those. |
12
12
  | `transcript_matches: <regex>` | the transcript matches (case-insensitive) — for stochastic prose **Sees top-level `assistant_text` only — it excludes every `tool_use`/`tool_result`**, so text emitted inside a tool call (a gate question, an option label or description) can never match; use `question_context`/`question_asked` or `tool_result_contains` for those. |
13
13
  | `transcript_not_matches: <regex>` | it does not match (e.g. no leaked stack trace) **Sees top-level `assistant_text` only — it excludes every `tool_use`/`tool_result`**, so text emitted inside a tool call (a gate question, an option label or description) can never match; use `question_context`/`question_asked` or `tool_result_contains` for those. |
14
- | `file_exists: <path>` | the path exists under the run's `work/` (anchored at `mnt/`, e.g. `outputs/x.md`). For a user-facing deliverable prefer `user_visible_artifact` — with a connected folder the file lands in `mnt/<folder>` (= `{{workspaceFolder}}`), not `mnt/outputs`, so `file_exists: outputs/x.md` misses it. Object form `{path, authored}`: `authored: true` also requires that THIS run created or rewrote the file (an untouched pre-run file — e.g. from a `workspace_fixture` — fails; no pre-run manifest ⇒ evidence-unavailable); `authored: false` states that inheriting it is fine. On a `workspace_fixture` path one of the two is REQUIRED (refused at load otherwise). On `lane: remote` plain `file_exists` is the proxy for a written path; with `authored: true` it is rejected at load. |
14
+ | `file_exists: <path>` | the path exists under the run's `work/` (anchored at `mnt/`, e.g. `outputs/x.md`). For a user-facing deliverable prefer `user_visible_artifact` — with a connected folder the file lands in `mnt/<folder>` (= `{{workspaceFolder}}`), not `mnt/outputs`, so `file_exists: outputs/x.md` misses it. Object form `{path, authored}`: `authored: true` also requires that THIS run created or rewrote the file (an untouched pre-run file — e.g. from a `workspace_fixture` — fails; no pre-run manifest ⇒ evidence-unavailable); `authored: false` states that inheriting it is fine. On a `workspace_fixture` path one of the two is REQUIRED (refused at load otherwise). On `lane: remote` plain `file_exists` is the proxy for a written path; with `authored: true` it is rejected at load. Under `device_bridge`, a path in a connected folder is refused: the folders live in the device container; assert the `device_bash` call instead. |
15
15
  | `user_visible_artifact: <path>` | exists **and** under a user-visible root (`outputs/` + each connected folder's mount name) — the right primitive for a workspace deliverable when a folder is connected. Takes the same `{path, authored}` object form as `file_exists` (and `artifact_text`/`artifact_json` take an `authored` field) |
16
16
  | `no_delete_in_outputs: true` | no delete op touched `mnt/outputs` — **only `true` is valid**; `false` is rejected. Checks on **every** baseline. Omitting it: on a baseline recording outputs `rw` (Desktop before 2.16120.0) a detected delete still fails via the `outputs_delete` verdict signal, which fires *because* the key was not authored (`allow_outputs_delete: true` accepts an intended one); on `rwd` (2.16120.0+, including `latest`, where Cowork allows deletes in outputs) nothing checks outputs deletes, so author this key to keep the check. Detection is a post-run bash-command scan plus a per-turn filesystem diff of `outputs/`, not mount enforcement, so a green means none was *detected*. Fails when the diff proves the delete, when a delete in command/call position has an `outputs/` path as its own operand, or when the diff could not verify; a hit resting only on the detector's inference (e.g. a Python variable named `rm`) with a clean diff passes, and the `outputs_delete_unconfirmed` warn is still raised in the run output — see that code for the classes of real delete that land there |
17
- | `no_delete_in_mounts: true` | no delete op touched `outputs` or any `rw` connected folder, except mounts waived by `allow_delete_in`. Covers `outputs` on **every** baseline, including those where Cowork allows an outputs delete (unless outputs is waived, authoring it arms the outputs check, so a filesystem-proven delete fails as the `outputs_delete` signal, which `allow_outputs_delete` waives). Production denies `unlink`/`rmdir` on a `rw` connected folder until approval, so `no_delete_in_outputs` covers only part of the rule. **only `true` is valid** |
17
+ | `no_delete_in_mounts: true` | no delete op touched `outputs` or any `rw` connected folder, except mounts waived by `allow_delete_in`. Covers `outputs` on **every** baseline, including those where Cowork allows an outputs delete (unless outputs is waived, authoring it arms the outputs check, so a filesystem-proven delete fails as the `outputs_delete` signal, which `allow_outputs_delete` waives). Production denies `unlink`/`rmdir` on a `rw` connected folder until approval, so `no_delete_in_outputs` covers only part of the rule. Under `device_bridge` it also reads `device_bash` commands and compares the device copy of each `rw` folder before and after the run. **only `true` is valid** |
18
18
  | `no_unexpected_files: [<glob>, …]` | every **newly created** file under a user-visible root matches ≥1 glob (workRoot-relative paths; `**` = whole path segment for any depth — use `outputs/handoff/**` for per-run subdirs); `[]` = no new files; **new-files-only** — overwriting a pre-existing file in place is invisible (use content-level producer stamping); live/verify-run without a pre-run manifest ⇒ evidence-unavailable hard-fail (live runs capture the baseline only when this key is asserted; recordings always capture); an **incomplete post-run filesystem walk** (an unreadable subtree — a permission/I-O error) also fails evidence-unavailable rather than reporting "no strays" over a partial tree — distinct from the missing-manifest case (a `--resume` run), which fails for a different reason; captured on every live sandbox tier including microvm (its outputs are snapshotted from the VM into the run dir); replay needs `cassette.preRunPaths` (≥0.24 recordings) — cassettes without it **exclude** the key with a loud warning. ⚠️ **Not a stand-in for "file X must not exist":** there is no negative-existence key, and inverting this one is a different claim — it is new-files-only (a pre-existing file is invisible to it) and needs a pre-run manifest. Assert the content-level consequence instead (`tool_not_called`, or the absent `artifact_json`) rather than widening an allowlist for every incidental lock/temp file. Rejected at load on `lane: remote`. |
19
- | `input_unmodified: <glob> \| [<glob>, …]` | a single glob or a list; every **pre-existing** file (incl. uploaded files under `uploads/**`) whose workRoot-relative path matches ≥1 glob keeps an unchanged content hash after the run — the in-place-mutation companion to `no_unexpected_files`'s new-files check (`[]` is rejected by the schema — list at least one glob); a glob that matches **no** pre-run path fails loud (a typo or renamed mount would otherwise verify zero files and pass vacuously); a matched file that was deleted counts as a content change (fails); live/verify-run without a pre-run hash manifest ⇒ evidence-unavailable hard-fail (a `--resume` run); captured on every live sandbox tier including microvm; replay needs `cassette.preRunHashes` — cassettes without it **exclude** the key with a loud warning; on replay it compares against the manifest's recorded `sha256`, never a re-hash of the materialized tree. On `lane: remote` it still reads the local stand-in for the inputs, which is right there: an edit to a staged copy inside the cloud container does not reach the device. |
19
+ | `input_unmodified: <glob> \| [<glob>, …]` | a single glob or a list; every **pre-existing** file (incl. uploaded files under `uploads/**`) whose workRoot-relative path matches ≥1 glob keeps an unchanged content hash after the run — the in-place-mutation companion to `no_unexpected_files`'s new-files check (`[]` is rejected by the schema — list at least one glob); a glob that matches **no** pre-run path fails loud (a typo or renamed mount would otherwise verify zero files and pass vacuously); a matched file that was deleted counts as a content change (fails); live/verify-run without a pre-run hash manifest ⇒ evidence-unavailable hard-fail (a `--resume` run); captured on every live sandbox tier including microvm; replay needs `cassette.preRunHashes` — cassettes without it **exclude** the key with a loud warning; on replay it compares against the manifest's recorded `sha256`, never a re-hash of the materialized tree. On `lane: remote` it still reads the local stand-in for the inputs, which is right there: an edit to a staged copy inside the cloud container does not reach the device. Under `device_bridge`, a glob on a connected folder's path is refused (the folders live in the device container). |
20
20
  | `self_heal_ran: <bool>` | a bash command the model wrote did (not) name a plugin under `/sessions/<id>/mnt/.local-plugins/` or `/sessions/<id>/mnt/.remote-plugins/` — the plugin-root self-heal path. It reads the command as the model wrote it, so a host plugin path that `hostloop`'s bash tool rewrote to the VM mount does not count |
21
21
  | `file_absent: <path>` | the named path does **not** exist under the work root after the run — the direct negative-existence key. Do NOT invert `no_unexpected_files` for this: that is an allowlist over NEW files only, and it needs a pre-run manifest. **LIVE/verify-run only** (a cassette records no walk health, so absence is unprovable on replay); rejected at load on `lane: remote`; evidence-unavailable on `preRunOrigin: remote-unavailable` |
22
22
  | `artifact_text: {artifact, contains?: [..], not_contains?: [..], matches?, not_matches?}` | assert over a delivered artifact's TEXT body — `artifact_json`'s companion for non-JSON files, and how you prove an internal name did not leak into a file the user receives. Literal path (no glob), so one entry per delivered surface. Manifest-class; body-less / symlinked / over-cap targets fail evidence-unavailable, and a non-UTF-8 body fails the NEGATIVE matchers rather than passing against bytes it never read. Rejected at load on `lane: remote` (that lane's container filesystem is not locally observable) |
@@ -26,7 +26,7 @@ keys is an AND; glob vs regex name matching) and the verdict-signal table are in
26
26
  | `tool_called: {tool: <glob> \| [..], input?, input_any?, result?, scope?, subagent_type?, count?}` | **object form** — asserts what a call CARRIED, where it RAN and what its paired result SAID, which no name glob and no `transcript_*` key can see (`transcript_*` reads top-level prose only, so it passes when the agent merely *says* it ran a command). `tool`: a glob or a list of globs (any-of; list both shells, `[Bash, mcp__workspace__bash]`, for a claim that must hold at hostloop too). `input: {<field>: <regex>}`: every named top-level input field must match (case-insensitive, unanchored; a missing field is no match; a non-string value is matched as JSON). `input_any: <regex>`: some top-level field matches. `result: {matches?, not_matches?, is_error?}`: predicates on the call's paired `tool_result` (by `toolUseId`, 10,240 characters of text — 32,768 for a top-level `Skill` result). `scope`: `main` (default — the main agent, including a `Skill`'s or `Agent(fork)`'s children, the set the string form reads), `subagent` (the parent is a sub-agent dispatch this run recorded, at any depth), or `any` (also calls whose parent is not a recorded dispatch, e.g. a `Skill` invoked inside a sub-agent). `subagent_type` (with `scope: subagent`): regex over the IMMEDIATE parent dispatch's type or description. `count: {min?, max?}` (default `min: 1`). **Fails closed:** an unpaired call never satisfies `result`; a truncated result or input that cannot settle a predicate, or a result.json without `toolCalls`, reports *evidence unavailable*, never a pass or a plain "not called". A red lists the calls it considered and names matches in another scope ("1 matching call in scope subagent — set `scope: any`"). `{tool: X}` alone is exactly `tool_called: X`. A cassette using this form stamps **v13**. |
27
27
  | `tool_not_called: {tool: <glob> \| [..], input?, input_any?, result?, scope?, subagent_type?}` | **object form** — no call in scope satisfies every predicate (no `count`). Same fields and fail-closed rules as above: a candidate whose result is unpaired or truncated, or whose input was truncated, is *evidence unavailable*, never a pass. Refused at load when EVERY listed tool is one the tier does not serve. ⚠️ **Redaction hazard — this is the dangerous direction.** A committed cassette is redacted, which rewrites the recorded inputs: an `input` regex naming a literal the policy rewrites (a home path, an email) cannot see its target on replay. On replay, any candidate call whose field (or paired result, for `result.matches`) carries a redaction token is *evidence unavailable*, whatever the regex — the evaluator cannot know what the token replaced — so a negative check never passes over rewritten bytes. `record` warns — naming the redacted call and the ways out (narrow `scope`/`tool`, the string form, or live-only) — and refuses the write (the exact guard), and `lint` flags a regex naming a redactable literal (`tool-input-regex-redactable`) — a heuristic that reads `.cowork-redact.json` from the current and scenario directories only, not the cassette's. Match a part of the input redaction leaves alone (the verb and flags, a workspace-relative path), or keep the check on a live gate. Note also that the default `scope: main` does not see a sub-agent's call — use `scope: any` for "nothing anywhere ran X". |
28
28
  | **legacy tool names** | The agent binary canonicalizes legacy spellings (`Task`→`Agent`, `KillShell`/`KillBash`→`TaskStop`, …) while the spawn tool list still declares the LEGACY one — so the init inventory shows `Task` and every call is emitted as `Agent`. `tool_called`/`tool_not_called`/`subagent_tool_used`/`subagent_tool_absent` match EITHER spelling, globs included. Before this, `tool_called: "Task"` could never pass and `tool_not_called: "Task"` always did. |
29
- | **tier vacuity — now REFUSED** | `tool_not_called` **and `subagent_tool_absent`** naming a tool the tier does not serve is rejected at scenario load (`Bash`/`WebFetch`/`NotebookEdit` at `hostloop`; `mcp__workspace__bash` at `container`/`microvm`), because it could never be violated. `scenario.py lint` WARNs on the same set before any run. The table is CLOSED to tools the harness itself removes or registers — `--tools` gates only the built-in set while every tier separately passes `--mcp-config`, so a session-MCP tool name is offered without appearing in any tool list and is never refused. Globs are never refused. `subagent_tool_absent` is covered because it reads the tools sub-agents actually USED, not a per-dispatch declared list. Under-approximates on purpose: `REPL` at hostloop is vacuous too and is not caught, which is the right side to err on for a hard refusal. |
29
+ | **tier vacuity — now REFUSED** | `tool_not_called` **and `subagent_tool_absent`** naming a tool the tier does not serve is rejected at scenario load (`Bash`/`WebFetch`/`NotebookEdit` at `hostloop`; `mcp__workspace__bash` at `container`/`microvm`; on those three tiers, `mcp__remote-devices__device_bash`/`get_device_info` without `device_bridge: true`, and every other `mcp__remote-devices__*` tool the harness never serves, bridge or not), because it could never be violated. `scenario.py lint` WARNs on the same set before any run. The table is CLOSED to tools the harness itself removes or registers — `--tools` gates only the built-in set while every tier separately passes `--mcp-config`, so a session-MCP tool name is offered without appearing in any tool list and is never refused. Globs are never refused. `subagent_tool_absent` is covered because it reads the tools sub-agents actually USED, not a per-dispatch declared list. Under-approximates on purpose: `REPL` at hostloop is vacuous too and is not caught, which is the right side to err on for a hard refusal. |
30
30
  | `reference_read: <regex>` | a skill `references/`/`scripts/` file whose path matches this **regex** was ACCESSED — main agent or sub-agents, via `Read`, `Grep`/`Glob` (`path` input), or a `Bash`/`mcp__workspace__bash` command naming the path. Regex is **unanchored + case-insensitive** (the shared helper every regex key uses). **Under-approximates by design:** the path must be rooted in the mounted plugin, so a `cd` into the skill dir then a bare `cat references/x.md`, a heredoc body, and a `$VAR`-built path are invisible. Fails **evidence unavailable** when the run recorded no observable tool stream. Replay-capable (cassettes freeze whole tool inputs) |
31
31
  | `no_observed_reference_access: <regex>` | no OBSERVED access matched the regex — the progressive-disclosure check: a reference the skill's routing never reaches. Named `observed` because detection under-approximates (see above), so it is **not proof the file went unread** — an agent that `cd`s and `cat`s it passes. Fails **evidence unavailable** rather than passing vacuously when no observable tool stream was recorded |
32
32
  | `tool_result_contains: <str>` | a tool result includes the literal string (content / replay-checkable — substring match) |
@@ -1,6 +1,6 @@
1
1
  # Assertion catalog
2
2
 
3
- Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). Every `assert:` key with its semantics, and the
3
+ Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). Every `assert:` key with its semantics, and the
4
4
  verdict-signal table. Which keys survive `replay` is in [`scenario-schema.md`](./scenario-schema.md#replay-class);
5
5
  the scenario and session YAML fields are there too.
6
6
 
@@ -47,7 +47,7 @@ codes (`VerdictSignal["code"]` in `src/run/verdict.ts`):
47
47
  |---|---|---|
48
48
  | `assertion` | fail | An authored `assert:` item failed |
49
49
  | `result_error` | fail | The run's SDK result was `"error"` |
50
- | `usage_limit` | fail | Usage/quota limit hit (not a skill failure) — retry after the limit resets. Emitted when `RunResult.resultErrorKind === "usage_limit"` |
50
+ | `usage_limit` | fail | Usage/quota limit hit (not a skill failure) — retry after the limit resets. Emitted when `RunResult.resultErrorKind === "usage_limit"` — including when a sub-agent hit the limit and the main loop ended `success`, where the message names the sub-agent and says to check its model and the account's plan instead of promising a reset (`decisions[]` row `subagent_usage_limit`) |
51
51
  | `transport_error` | fail | The connection dropped mid/after-run |
52
52
  | `permissive_auto_allow` | fail | A cowork-parity auto-allow real Cowork would block (opt out: `allow_permissive_auto_allow`) |
53
53
  | `outputs_delete` | fail | An unauthorized delete touched `mnt/outputs` on a baseline that records outputs `rw` (not raised on `rwd`, Desktop 2.16120.0+, unless `no_delete_in_outputs` or `no_delete_in_mounts` (outputs not waived) is authored — with `no_delete_in_outputs` that assertion fails instead; with `no_delete_in_mounts` this signal itself fires, and its message names that key), confirmed: the per-turn filesystem diff proves it, a delete in command/call position has an `outputs/` path as its own operand, or the diff could not verify the turn. Authoring `no_delete_in_outputs` moves it into that assertion; `allow_outputs_delete` waives it |
@@ -57,7 +57,7 @@ codes (`VerdictSignal["code"]` in `src/run/verdict.ts`):
57
57
  | `host_path_leak` | fail | A host path leaked into model-visible text (opt out: author `transcript_no_host_path`). A path copied verbatim from the scenario's own uploads, connected folders, prompt, or declared plugins' or local skills' files is not a leak |
58
58
  | `l0_host_config_contamination` | fail | `protocol` ran against the operator's REAL config dir, so host-installed plugins/skills/MCP servers may have answered instead of the thing under test — the run did not necessarily measure it (opt out: `allow_l0_host_config_contamination`). Name predates the meaning: delivery at L0 works, contamination is what this reports |
59
59
  | `missing_capability` | fail | A `requires_capabilities` need was unmet, or the skill used a capability the image omits (opt out: `allow_missing_capability`, or `skill --allow-missing-capability` on an open-ended run) |
60
- | `infra_error` | fail | A supervising process died mid-run (VM/egress sidecar) — the run's evidence is contaminated, not author-suppressible |
60
+ | `infra_error` | fail | A supervising process died mid-run (the VM/egress sidecar, or the `device_bridge` container) — the run's evidence is contaminated, not author-suppressible |
61
61
  | `stalled` | fail | The run ended asking for input with no tool work after the last gate. "Asking" = the final turn's closing sentence ends in `?`; or, once an `AskUserQuestion` gate has fired, it ends in `?` after trailing bold/quotes/`)`/emoji, is a `?` followed only by a `For example: …`/parenthetical aside, or is a request that says the input comes back to the agent (`Please share/provide/upload… so I can…`/`…here`, `Let me know which… you'd like me to use`, `Once you share… I'll…`, `I need X to proceed`, `Once I have the file, I'll…` after a sentence asking for it). Polite closers and hand-offs (`Let me know if…`, `Feel free…`, `with your`/`to your`, `before sending`, `If you…`) and a closing code block or blockquote never count. Hand-offs to a named third party (`with the team`, `to the founders`, `with the CFO`) never count either; `here` is a cue only after share/paste/upload/drop/reply/type/send or as the last word. English-only. Opt out with `allow_stall` (on `run`, `replay` and `eval`); without it, under `eval` a stalled rep is `errored_agent` and fails every row |
62
62
  | `non_deterministic` | warn | The run was LLM/external/human-decided — not reproducible |
63
63
  | `model_fallback` | warn | The agent fell back off the requested model mid-run (SDK `model_fallback` event); a `model_not_found`/`model_blocked` trigger repeats every run until the pin changes |
@@ -68,7 +68,7 @@ codes (`VerdictSignal["code"]` in `src/run/verdict.ts`):
68
68
  | `delivery_unobservable` | warn | `lane: remote` only — the run produced file(s), but whether any reached the user CANNOT be verified: nothing is delivered by location on that lane and the harness models no remote delivery tool (production uses the agent-native `SendUserFile`). The honest counterpart to `undelivered_deliverables`, which would otherwise fire on every remote run that writes anything — a signal that always fires carries no information. Mutually exclusive with it; quiet when the run produced nothing to deliver. A harness coverage gap, not a skill defect. Opt out: `allow_undelivered_deliverables` |
69
69
  | `partly_scripted_gate` | warn | A question batch (one `AskUserQuestion` with several sub-questions) that the scenario's `answers:` matched only PART of. Answers are delivered atomically, so the whole batch went to the `on_unanswered` fallback and the matched answers were NOT delivered — the fallback may contradict them. The message names the matched and unmatched sub-questions and who answered instead; `result.partlyScriptedGates` carries the full lists. Fires only on a partial match within one batch (a single-question gate no rule matched is the ordinary unanswered case). Re-derived on `replay` (from the cassette's frozen answers) and `verify-run` (from the current scenario). Fix: script every sub-question of the batch |
70
70
  | `parked_at_question` | warn | The session declares `answer_channel: none` and the run ended on a question nobody can answer: its last message ends in `?`, even after tools ran (a finished answer ending on an offer warns too; a request without `?` does not fire). Takes the place of the `stalled` fail and never changes the verdict: with no answer channel, stopping at the question is the contract. Completion is judged from the file assertions such a scenario must carry (refused at load without one). No opt-out needed |
71
- | `exec_infra_error` | warn | Host-loop: one or more container `exec` calls failed for infrastructure reasons, so those tool calls returned an error to the agent. Warns rather than fails because the run's other evidence is intact — unlike `infra_error`, where a dead supervisor contaminates everything. Caveat: if *every* exec failed, the agent ran nothing and this still only warns — check `result.infraErrors` |
71
+ | `exec_infra_error` | warn | Host-loop, or a `device_bash` call under `device_bridge`: one or more container `exec` calls failed for infrastructure reasons, so those tool calls returned an error to the agent. Warns rather than fails because the run's other evidence is intact — unlike `infra_error`, where a dead supervisor contaminates everything. Caveat: if *every* exec failed, the agent ran nothing and this still only warns — check `result.infraErrors` |
72
72
 
73
73
  A **fail**-severity signal does not change `result.result` (still `"success"`), but it DOES fail the
74
74
  overall run verdict and exit code — `assert result: success` alone won't catch it; check
@@ -1,6 +1,6 @@
1
1
  # Assertions guide
2
2
 
3
- Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). Read it when choosing assertion keys: the two orthogonal axes and the goal → key map. The full catalog starts at `assertion-catalog.md`, which indexes the per-family files that hold every key's row.
3
+ Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). Read it when choosing assertion keys: the two orthogonal axes and the goal → key map. The full catalog starts at `assertion-catalog.md`, which indexes the per-family files that hold every key's row.
4
4
 
5
5
  ### Assertions: two orthogonal axes
6
6
 
@@ -1,6 +1,6 @@
1
1
  # Authoring a scenario
2
2
 
3
- Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). Read it when composing a `scenarios/*.yaml`: session vs scenario, discovery, the fidelity tier, the answer path, `web_fetch`, and scaffold + lint.
3
+ Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). Read it when composing a `scenarios/*.yaml`: session vs scenario, discovery, the fidelity tier, the answer path, `web_fetch`, and scaffold + lint.
4
4
 
5
5
  ## Part I — AUTHOR a scenario
6
6
 
@@ -74,6 +74,18 @@ reach the device), and
74
74
  plain `file_exists` is the proxy for a written path. Delivery itself is the `delivery_unobservable` WARN (see
75
75
  `run-record-replay.md`).
76
76
 
77
+ **`device_bridge: true`** (with `lane: remote` and `fidelity: container`) serves the remote lane's shell on the user's
78
+ computer, `mcp__remote-devices__device_bash`, from a second container holding COPIES of the connected folders at
79
+ `$HOME/mnt/<folder-name>`; the agent's own container gets no folder. Assert on that call, never on the folder's
80
+ files: `tool_called: {tool: mcp__remote-devices__device_bash, result: {matches: <text>}}`, plus `transcript_matches`.
81
+ `file_exists` / `input_unmodified` on a connected folder's path are refused under the bridge (they would read the
82
+ agent's container). `no_delete_in_mounts` covers the folders (the device copy allows a delete production refuses).
83
+ `tool_not_called` on `device_bash`/`get_device_info` without the bridge, or on a device tool the harness never serves
84
+ (with it or without), is refused as vacuous. Egress follows the session allowlist;
85
+ `egress.unrestricted: true` matches production's pass-through. With no `permission_mode` in the session the agent
86
+ runs in `acceptEdits` (a production cloud task's mode under Manual); a mode you set wins. The run's `deviceBridge` lists every known
87
+ difference from production.
88
+
77
89
  ### Choose an answer path (gates: AskUserQuestion + tool-permission)
78
90
 
79
91
  Default to **deterministic**: scripted `answers:` + `on_unanswered: fail`. Anything that brings a
@@ -258,6 +270,7 @@ cowork-harness lint scenarios/*.yaml
258
270
  | `host-path-assert-cowork` | WARN | `transcript_no_host_path` on `cowork` — it fails by design if the tier resolves to hostloop |
259
271
  | `host-path-assert-tier` | ERROR | `transcript_no_host_path` on `hostloop` / `protocol`, where it fails by design |
260
272
  | `lane-remote-incompatible-key` | ERROR | a key that can never pass on `lane: remote`, which the runtime refuses at load: `present_files_called` / `no_scratchpad_leak` / `user_visible_artifact` (delivery), `artifact_text` / `artifact_json` (file bodies), `file_absent` / `no_unexpected_files` (absence), `computer_links_resolve(_if_present)`, `no_lost_write_back`, `file_exists` with `authored: true`, and `semantic_matches` / `semantic_pairwise` with `evidence_files`. Tier rules are suppressed there |
273
+ | `device-bridge-misplaced` | ERROR | `device_bridge` set anywhere but `true` with `lane: remote` and `fidelity: container`; the loader refuses it |
261
274
  | `linter-extra-findings-invalid` | ERROR | the loader findings `cowork-harness lint` hands the linter could not be read |
262
275
  | `linter-unclassified-key` | ERROR | a valid assertion key this linter cannot classify (the linter is out of date) |
263
276
  | `manifest-needs-snapshot` | INFO | manifest-backed keys, which evaluate on replay only when the cassette carries an `artifacts` manifest (not reported on `lane: remote` for `user_visible_artifact` / `artifact_json` / `artifact_text`, which cannot pass there) |
@@ -1,6 +1,6 @@
1
1
  # CI recipe — replay vs live lanes
2
2
 
3
- Self-contained reference. Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`).
3
+ Self-contained reference. Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`).
4
4
 
5
5
  **Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
6
6
  job-summary reporter (verdict table, staleness findings, cost/turns when available):
@@ -17,7 +17,7 @@ job-summary reporter (verdict table, staleness findings, cost/turns when availab
17
17
  CLI major reaches your workflow the moment it is promoted even though your `uses:` ref never changed — so a
18
18
  copy-pasted recipe that omits the input takes a major bump with no say in it. `^4` holds the major, needs no
19
19
  patch number to remember, and only wants a human decision at the next major. Pin an exact version
20
- (e.g. `version: "4.7.0"`) instead when you want byte-reproducible CI.
20
+ (e.g. `version: "4.8.0"`) instead when you want byte-reproducible CI.
21
21
 
22
22
  Reach for the manual multi-step form below only when you need per-step control the Action's inputs don't
23
23
  cover (a custom flag combination, a different runner matrix per step, or `lint`/`verify-cassettes` gated
@@ -82,7 +82,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
82
82
  GitHub-hosted runners, no token/Docker/agent:
83
83
 
84
84
  ```yaml
85
- - run: npm i -g "cowork-harness@^4.7.0"
85
+ - run: npm i -g "cowork-harness@^4.8.0"
86
86
  - run: cowork-harness lint scenarios/*.yaml --strict --min-severity WARN
87
87
  # no silent false-greens. WITHOUT --strict this
88
88
  # step cannot fail on a WARN-class rule (e.g.
@@ -380,7 +380,9 @@ A typical skill repo runs four stages, fastest/cheapest first:
380
380
  ```
381
381
 
382
382
  The `jq -e` IS the gate — `--corpus-only` exits 0 on a measurement even over the ceiling. Same
383
- packager, same git filter as the paid run; the number is a floor (a run-time read can only add).
383
+ packager, same git filter as the paid run; the number is a floor (a run-time read can only add). The same
384
+ payload's `skillTreeHash` (every delivered file of the skill) tells a job whether the skill changed since
385
+ its last critique, at no cost.
384
386
  3. **Scenarios (replay)** — `cowork-harness replay cassettes/` on every PR (the committed `*.cassette.json`).
385
387
  Token-free; content + structure + gate delivery.
386
388
  4. **Parity / live (nightly, self-hosted)** — `cowork-harness run scenarios/` with a token + Docker +
@@ -409,7 +411,7 @@ jobs:
409
411
  with: { node-version: '24' }
410
412
  - uses: actions/setup-python@v5
411
413
  with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
412
- - run: npm i -g "cowork-harness@^4.7.0"
414
+ - run: npm i -g "cowork-harness@^4.8.0"
413
415
  - run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
414
416
  - run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
415
417
  - run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
@@ -438,7 +440,7 @@ jobs:
438
440
  echo "live=true" >> "$GITHUB_OUTPUT"
439
441
  fi
440
442
  - if: steps.guard.outputs.live == 'true'
441
- run: npm i -g "cowork-harness@^4.7.0"
443
+ run: npm i -g "cowork-harness@^4.8.0"
442
444
  - if: steps.guard.outputs.live == 'true'
443
445
  run: cowork-harness run scenarios/ --output-format json
444
446
  env:
@@ -1,10 +1,10 @@
1
1
  # Critique — the facts a plugin install can't otherwise reach
2
2
 
3
- Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). This is **not** a trim of the full
3
+ Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). This is **not** a trim of the full
4
4
  [`docs/critique.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/critique.md) (repo-only —
5
- flags, cost, reproduction discipline, known limitations all live there). This file covers exactly what a
5
+ flags, cost, reading several critiques, known limitations all live there). This file covers exactly what a
6
6
  plugin install cannot otherwise discover: the run-dir artifact a harvester actually reads, the report's
7
- real field names, and what the evaluator was and was not shown. See **Recipe 5** in `task-recipes.md` for
7
+ real field names, and what the evaluator was and was not shown. See **Recipe 6** in `task-recipes.md` for
8
8
  the harvest → reproduce → fix loop this feeds.
9
9
 
10
10
  ## The run-dir artifact a harvester reads
@@ -49,12 +49,12 @@ carry that; read all three before touching anything.
49
49
  | `infraFailure` | the reason |
50
50
  | `infraFailurePhase` | `task turn` (the graded run) or `reflection turn` (critique's own protocol turn) |
51
51
  | `infraFailureKind` | why it failed — a harness `ErrCategory` (error envelope, exit 2/3) **or** a `resultErrorKind` (`usage_limit`/`transport`/`agent`) from a turn that RAN and errored (exit 1, top-level `error: null`). **Absent** = killed, or no envelope |
52
- | `gradedErrorReason` | on a `taskResult: "error"` run (still gradeable, exit 0): why the GRADED turn errored, so a quota exhaustion is not read as a skill defect |
52
+ | `gradedErrorReason` | on a `taskResult: "error"` run (still gradeable, exit 0): why the GRADED turn errored, so a dropped connection is not read as a skill defect. A task turn that hit the usage limit (its own result, or a sub-agent's) is not graded: `infraFailurePhase: "task turn"`, `infraFailureKind: "usage_limit"`, exit 2 |
53
53
 
54
54
  **Do NOT read "has a kind" as "the instrument is fine".** The CLI's top-level catch turns every
55
55
  unexpected throw into category **`internal`** — Docker down, container start failure, missing staged
56
56
  agent, harness bug — and `runtime` carries a refused run dir. Only **`unanswered`, `usage`, `boundary`**
57
- and, from the result-row taxonomy, **`usage_limit`** (quota exhausted — retry after reset) and
57
+ and, from the result-row taxonomy, **`usage_limit`** (the account's quota — the main loop's or a sub-agent's; on the task turn critique exits 2 with `RUN FAILED (task turn, usage_limit)` and spawns no reflection) and
58
58
  **`transport`** (a tail-end drop) are the caller's problem. `agent` is not: for critique's own protocol
59
59
  turn that IS the instrument breaking. The header encodes exactly that split and fails closed (an
60
60
  unrecognized kind renders as infrastructure):
@@ -97,15 +97,23 @@ messages, never from the flag. So `graded model(s): unknown` means no assistant
97
97
  (crash, kill, or a gate before the first reply); passing `--model` does not change that line. Past runs
98
98
  can be checked without re-running: the same ids are in each kept run dir's `turns/1/result.json`.
99
99
 
100
+ ## A repaired evaluator reply — `evaluatorRepair`
101
+
102
+ When an evaluator reply was complete except for its final closing `}`, critique appends that one brace and
103
+ parses as usual; the report records `evaluatorRepair: [{pass, appended}]` (summary: `evaluatorRepaired: true`).
104
+ Nothing else is ever repaired — a reply that would also need a `]` (a cut-off list) or was cut mid-item or
105
+ mid-string still fails (exit 2): re-run the critique.
106
+
100
107
  ## The report's item shape — no `title`, no `summary`
101
108
 
102
109
  Each `items[]` entry's prose fields are **`idea`** and **`recommendedAction`** — there is no `title` field
103
110
  and no `summary` field. A consumer that parses for either gets silently blank output instead of an error.
104
111
  Full required set: `source` (`evaluator`|`self-report`), `idea`, `classification`, `evidence` (the cited
105
112
  excerpt, verbatim-checked against the evidence package), `recommendedAction`. The optional
106
- `findingFingerprint` hashes `idea`+`classification`+`recommendedAction` — high-precision, low-recall: a
107
- match proves the same finding recurred; a mismatch does NOT prove it didn't (the same finding reworded
108
- fingerprints differently).
113
+ `findingFingerprint` hashes `idea`+`classification`+`recommendedAction` — exact-match on model wording, so a
114
+ LOWER bound: a match shows the same wording came back; across repeats of the same probe it often never recurs,
115
+ and no match shows nothing. Read repeat reports side by side, aligned by `classification`; a shared `evidence`
116
+ excerpt is a cue to compare two items, not a match (unrelated findings often cite the same passage).
109
117
 
110
118
  ## What the evaluator was actually shown — `evidenceBudget`
111
119
 
@@ -171,6 +179,53 @@ from `dist/` source:
171
179
  | `trimRecord` | which section the transcript trim shaved, and by how much |
172
180
  | `packageTruncated` | `true` if ANY section was cut — check this, not `corpusCuts`, for "was anything trimmed": a transcript-only cut leaves `corpusCuts` empty and would otherwise read as "nothing cut" |
173
181
 
182
+ ## Has the skill changed? — the three hashes
183
+
184
+ Every report and every `--corpus-only` payload carries three `sha256:` hashes, keyed by path inside the mount (so
185
+ two clones of one commit hash alike):
186
+
187
+ | Hash | Covers | Use it for |
188
+ |---|---|---|
189
+ | `skillTreeHash` | every file staging delivered for the skill (`scripts/` included) + resolved agents + linked plugin-root references | "did ANYTHING change since the last critique?" — equal in `--corpus-only` and a graded run |
190
+ | `corpusHash` | the evaluator's static corpus (`SKILL.md`, `references/**`, agents, linked root refs) | a FLOOR: unchanged ≠ unchanged behaviour, since `scripts/` is not in it |
191
+ | `packagedCorpusHash` | what the packager put in the evaluator's corpus, incl. root files the agent READ and the ceiling's cuts | evidence of what was graded — moves with the agent's reading, never a "changed" key |
192
+
193
+ `corpusManifest` lists the files (diff two to see which one moved a hash); `hashBasis` is `git-tracked`
194
+ (working-tree bytes of tracked files), `worktree-all` or `git-commit`; untracked files under the skill are listed
195
+ in `skillTreeUntracked` (first 50; `skillTreeUntrackedCount` = total), never hashed; `.git` is never hashed. `corpusDrift` means a file changed during the run — critique a commit
196
+ instead.
197
+
198
+ ## Critique a commit — `git:<ref>:<path>`
199
+
200
+ `critique git:<ref>:<path> …` grades a snapshot of the commit, so a moved HEAD or an edit mid-run changes nothing;
201
+ `source.commit` records the resolved id. A path at `<plugin>/skills/<name>` grades that skill of the plugin (same
202
+ as `<plugin> --skill <name>`). Refused before spend: a git filter such as LFS (`.gitattributes`,
203
+ `.git/info/attributes`, `core.attributesFile`), a symlink whose real path leaves the snapshot, a submodule. Works
204
+ with `--corpus-only` ($0). Kept snapshots: `~/.cowork-harness/critique-snapshots/` (or
205
+ `COWORK_HARNESS_CRITIQUE_SNAPSHOTS_DIR`); nothing prunes them.
206
+
207
+ ## A public ledger line — `--summary-out`
208
+
209
+ `--summary-out <file>` writes a summary safe for a PUBLIC repo: ids, hashes, enums, and per finding only
210
+ `findingFingerprint` / `classification` / `source` / `adjudicable` — no finding text, prompt, host path or git
211
+ ref. Values are shape-checked (a failure is `null` + named in `withheld`); a summary a configured secret would
212
+ alter is NOT written (warning, exit unchanged). Cost and the probe hash are opt-in
213
+ (`--summary-include-cost`, `--summary-include-prompt-hash`). With `--summary-out`, a `--label` must match
214
+ `[A-Za-z0-9._:+-]{1,64}` (e.g. `v1.2.0-rc1`, not `fix/x`). Schema: [`schema/critique-summary.json`](https://github.com/yaniv-golan/cowork-harness/blob/main/schema/critique-summary.json).
215
+
216
+ ## Several critiques side by side — `critique --compare`
217
+
218
+ `critique --compare <report.json | summary.json …>` ($0, no model) groups by `--label` (one group, or two for
219
+ before/after) and lists every finding per report, aligned by classification. It gives **no verdict**: an exact
220
+ `findingFingerprint` match is `sameWording` k/N — a LOWER bound (it often never recurs across repeats); a
221
+ `sharedExcerpt` is the same cited passage, NOT proven the same finding. Equal `corpusHash` and one shared
222
+ `skillTreeHash` in both groups = `noiseFloorControl` (a scripts/ edit never is). Refused: mixed skills / schemes /
223
+ harness majors, mixed `corpusHash` within a group, the same critique twice, a summary that withheld a key
224
+ field, a pre-4.8 report. Marked, and refused
225
+ by `--strict`: mixed evaluator models / hash bases / skill trees / probes, a pass-1-only member. Marked only:
226
+ `mixedPackagedCorpus`, `probeUnverified` (no prompt hash — the summary default).
227
+ Its output carries finding text — not public-safe. Details: [`docs/critique.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/critique.md) (repo-only).
228
+
174
229
  ## An untracked skill file is not graded
175
230
 
176
231
  Staging delivers **git-tracked files only** (same rule as a marketplace plugin install) — an untracked
@@ -1,6 +1,6 @@
1
1
  # Debugging a run
2
2
 
3
- Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). Read it when a run misbehaved or a green looks wrong: triage, the observability output, and `chat`.
3
+ Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). Read it when a run misbehaved or a green looks wrong: triage, the observability output, and `chat`.
4
4
 
5
5
  ## Part III — Debug
6
6
 
@@ -107,7 +107,9 @@ decide which assertions from *Assertions: two orthogonal axes* in `assertions-gu
107
107
  `semantic_*` with `evidence_files`);
108
108
  `semantic_*` without it is judged on the transcript only. Reach for it
109
109
  to check a skill's delivery survives the cloud lane, where new Pro and Max tasks run from 2026-10-06. Orthogonal to `fidelity` — a `lane: remote`
110
- scenario still runs locally.
110
+ scenario still runs locally. Add `device_bridge: true` (with `fidelity: container`) to serve `device_bash`, the
111
+ cloud task's shell on the user's computer; a red there reads from the `device_bash` tool calls and the run's
112
+ `deviceBridge` record, and a `device-container` / `device-exec` infra error is that second container.
111
113
  - **`cowork-harness stats [--metric <m>]`** — aggregate across the run index: `cost`, `duration`,
112
114
  `tokens`, `cache-tokens`, `model-cost`, `turns`, `pass-rate`. Filters: `--since`/`--baseline`/`--branch`,
113
115
  plus `--skill-hash <prefix>`/`--label <tag>` to narrow to ONE skill generation and
@@ -148,8 +150,12 @@ decide which assertions from *Assertions: two orthogonal axes* in `assertions-gu
148
150
  **`stderrLogPath`** (the captured agent stderr) — read those and `trace <run-dir>` *before* re-running;
149
151
  a re-record rarely tells you more than the captured stderr already does. Also check
150
152
  **`resultErrorKind`** (`"transport" | "agent" | "usage_limit"`) before spending another paid run: a
151
- `"usage_limit"` failure is a quota exhaustion, not a skill bug — retry after the limit resets rather
152
- than debugging; `"transport"`/`"agent"` means something actually broke, worth localizing before
153
+ `"usage_limit"` failure is the account's quota, not a skill bug — don't debug the skill. It is also set
154
+ when a **sub-agent** hit the limit and the main loop ended `success` (then `errorSource: "agent"`): the
155
+ `usage_limit` verdict message names that sub-agent (dispatch id, type, limit text), and `decisions[]`
156
+ carries a `name: "subagent_usage_limit"` row (`detail: {toolUseId?, subagentType?, taskId?, error,
157
+ prior?}`). Retry once the quota allows; for a sub-agent's limit also check the model it is pinned to and
158
+ the account's plan — a model-scoped or credits limit does not reset on a timer; `"transport"`/`"agent"` means something actually broke, worth localizing before
153
159
  re-running.
154
160
  - **Attributing cost to sub-agent work.** `subagents[]` gives the dispatch tree — each sub-agent's
155
161
  `dispatchModel`/`resolvedModel`, `toolsUsed`, `prompt`/`output`, and `attributedSkillId` — but **not** its own token/cost;
@@ -1,6 +1,6 @@
1
1
  # `eval` — paired before/after comparison of a skill edit (EXPERIMENTAL)
2
2
 
3
- Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). The full guide is
3
+ Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). The full guide is
4
4
  [docs/eval.md](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/eval.md); this is the part you
5
5
  need while running it.
6
6
 
@@ -48,7 +48,9 @@ cowork-harness eval report <eval-dir> # rebuild the report from the eval dir
48
48
  | `insufficient` | too few valid reps in an arm (4 of 5 needed by default) |
49
49
 
50
50
  A drop is a signal to investigate, not proof: open the run dirs the report links for that row. In order,
51
- first match wins: an infrastructure failure is excluded and reported — including a rep where no model
51
+ first match wins: an infrastructure failure is excluded and reported — including a usage limit the run lane
52
+ named (the main loop's or a sub-agent's, rule `kind_usage_limit`, even on a rep that otherwise timed out or hit
53
+ `error_max_turns`), and a rep where no model
52
54
  answered (the agent's `Not logged in` / `Authentication required` reply, rule `auth`; a usage or
53
55
  spend limit as its final message, even on a nonzero exit after spend, rule `usage_limit`; or only
54
56
  `<synthetic>` models at $0, rule `no_model_answered`). An agent-caused failure (timeout, max turns,
@@ -1,6 +1,6 @@
1
1
  # Fidelity tiers & answer paths
2
2
 
3
- Self-contained reference. Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`).
3
+ Self-contained reference. Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`).
4
4
 
5
5
  > **This page vs. the repo docs.** This is the **offline snapshot** that ships inside the installed
6
6
  > plugin — it is self-contained on purpose. The repo carries four other fidelity views, each answering a
@@ -1,6 +1,6 @@
1
1
  # Gotchas
2
2
 
3
- Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). The full "✓ passed ≠ correct" landmine catalog.
3
+ Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). The full "✓ passed ≠ correct" landmine catalog.
4
4
 
5
5
  ## Gotchas — the "✓ passed ≠ correct" landmines
6
6
 
@@ -1,6 +1,6 @@
1
1
  # Recipe 7 — Climb a skill with `/claude-api hillclimb` and the harness as its runner
2
2
 
3
- Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). It needs a `cowork-harness` whose
3
+ Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). It needs a `cowork-harness` whose
4
4
  `hillclimb --help` lists `--skill` (help goes to stderr). This page is the loop's procedure, step by step, in the order of the
5
5
  `/claude-api hillclimb` guide. Every mechanic (flags, refusals, the gate, `regrade`, `freeze-ref`, exit codes,
6
6
  row keys) is in [`hillclimb.md`](hillclimb.md); the setup and the full list of differences from the guide's own
@@ -1,6 +1,6 @@
1
1
  # `hillclimb` — the runner for a `/claude-api hillclimb` loop
2
2
 
3
- Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). It needs a `cowork-harness` whose
3
+ Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). It needs a `cowork-harness` whose
4
4
  `hillclimb --help` lists `--skill` (help goes to stderr). The command reference is
5
5
  [docs/cli.md](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/cli.md); this is the part a loop needs
6
6
  while it runs. It covers `run`, `check`, `state-template`, `freeze-ref` and `regrade`.
@@ -382,6 +382,26 @@ Flags: `--flow DIR`, `--variant all|baseline|v<N>` (default `all`: every variant
382
382
  row, never by `--allow-doc-drift`); one whose assert has an edit inside a scrubbed literal (only a re-run applies
383
383
  it); and an open `judge_invalid` slot.
384
384
 
385
+ ## Cutting the cost per rep
386
+
387
+ Every rep re-runs the whole scenario, so a case that tests the last step of a long pipeline pays for every step
388
+ before it on every rep. Start such a case from a workspace fixture instead: a directory whose files are copied into
389
+ the session's `outputs/` before turn 1, so the prompt asks for the late step alone
390
+ ([Starting from a saved workspace](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/scenario.md#starting-from-a-saved-workspace-workspace_fixture)).
391
+
392
+ - **Make one from a kept run.** `cowork-harness fixture export <run-dir> --out <dir>`, then point the case's
393
+ `workspace_fixture:` at `<dir>` and commit both ([fixture export](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/cli.md#exporting-a-runs-outputs-as-a-fixture-fixture-export)).
394
+ Add `--session-paths` when the skill reads back paths it recorded (an outputs-dir probe, a sub-agent's output
395
+ path). `--exclude <path>` leaves out a file the step re-creates, such as a probe the skill re-runs.
396
+ - **Limits.** A fixture with session-path tokens is refused on the `protocol` tier, and `--session-paths` refuses a
397
+ protocol run. Binary files are never rewritten; one that holds a token is refused. Any other host path in a text
398
+ file is still refused (`--allow-host-paths` accepts an ordinary one, never a path into a run dir).
399
+ - **What the case then measures.** Only the steps after the fixture. An untouched fixture file is pre-run, not
400
+ authored, so a judge grades only what this step wrote, and the conversation starts fresh. The earlier steps are no
401
+ longer exercised: keep a full-pipeline case for them.
402
+ - **The fixture is in the harness gate.** Its files are hashed with the scenario, so an edited or re-exported fixture
403
+ changes the sha and needs `--approve-harness` again.
404
+
385
405
  ## Exit codes
386
406
 
387
407
  - `run`: `0` every attempted (case, rep) was scored; `1` an attempt failed (an `errors.jsonl` row, or a scored
@@ -1,6 +1,6 @@
1
1
  # Measurement
2
2
 
3
- Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). Read it before comparing runs: `--repeat`, `--ablate-skill`, and the hygiene that keeps a batch valid.
3
+ Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). Read it before comparing runs: `--repeat`, `--ablate-skill`, and the hygiene that keeps a batch valid.
4
4
 
5
5
  ### Measure — before/after, with/without (`--repeat`, `--ablate-skill`)
6
6
 
@@ -1,6 +1,6 @@
1
1
  # Run, record and lock
2
2
 
3
- Tracks `cowork-harness 4.7.0` (baseline `desktop-2.31226.1`). Read it when running a scenario, recording or placing a cassette, reading verdict signals, checking a background run, or choosing CI lanes.
3
+ Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). Read it when running a scenario, recording or placing a cassette, reading verdict signals, checking a background run, or choosing CI lanes.
4
4
 
5
5
  ## Part II — RUN, RECORD & LOCK
6
6
 
@@ -132,8 +132,9 @@ current skill") rather than vouch against stale labels, but the trace/inspect pa
132
132
  deliberately. (Same fail-closed family: corrupt gate evidence — unparseable `events.jsonl` lines, or fewer
133
133
  gates than `trace.json` recorded questions — a structurally invalid `result.json`, a `command:"replay"`
134
134
  result (a replay is a re-check of a recorded cassette, not run evidence — verify the original live run dir),
135
- and a `mode:"chat"` result (chat carries no assertions or verdict by contract) also refuse rather than
136
- certify.) (A token-free probe of "which gates fire" isn't possible — gates are model-decided per run.)
135
+ a `mode:"chat"` result (chat carries no assertions or verdict by contract), and a `device_bridge` run dir (its
136
+ folders lived in the device container; re-run or replay the cassette, in `regrade` and `refs` too) also refuse
137
+ rather than certify.) (A token-free probe of "which gates fire" isn't possible — gates are model-decided per run.)
137
138
 
138
139
  Run artifacts are written to `~/.cowork-harness/runs/…` by default — **outside any working tree**, so a run
139
140
  launched from a repo root never drops sensitive skill inputs/outputs into it. Pass `--run-dir <path>` (or set
@@ -203,7 +204,7 @@ cassette — has its own recipe:
203
204
  `--label <tag>`); an un-split window spanning more than one generation now warns.
204
205
  **Multi-skill plugin caveat (post-1.7.0 CLIs with `--skill`):**
205
206
  skillHash keys the whole MOUNTED plugin, so on a multi-skill plugin the hash alone cross-pairs
206
- critiques of DIFFERENT skills — pair by the report's `(gradedSkillHash, gradedSkill)` pair. **On a pre-1.5.0 CLI the `skill` lane emits no `fingerprint.skillHash` at all**, so a
207
+ critiques of DIFFERENT skills — pair critiques by the report's per-skill `skillTreeHash` / `corpusHash` (4.8.0+), or by `(gradedSkillHash, gradedSkill)`. **On a pre-1.5.0 CLI the `skill` lane emits no `fingerprint.skillHash` at all**, so a
207
208
  pairing step there silently groups on an absent key instead of erroring — check the field is present, or
208
209
  require ≥ 1.5.0. See [`docs/debugging.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/debugging.md)
209
210
  (repo-only) for the full loop.
@@ -299,7 +300,7 @@ Recognize these before "fixing" a non-bug:
299
300
  skill's own files never exempt a path under its host source); `scan.inputHostPathTokens` counts the
300
301
  host-path tokens the inputs carried, `scan.hostPathsFromInputs` the ones exempted, and a non-zero exemption
301
302
  prints a `::notice::` so a clean scan that relied on it is never silent.
302
- - **`exec_infra_error`** (`WARN`, host-loop) — one or more container `exec` calls failed for
303
+ - **`exec_infra_error`** (`WARN`, host-loop, or `device_bash` under `device_bridge`) — one or more container `exec` calls failed for
303
304
  infrastructure reasons (daemon/container-level), so those tool calls returned an error to the agent
304
305
  rather than the command's own output. Warn-severity because the run's other evidence is intact — unlike
305
306
  the fail-severity `infra_error`, where a **supervising process** died and contaminated everything. Note