cowork-harness 1.7.0 → 1.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (55) hide show
  1. package/.claude/skills/cowork-harness/SKILL.md +54 -12
  2. package/.claude/skills/cowork-harness/references/ci-recipe.md +5 -5
  3. package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +17 -1
  4. package/.claude/skills/cowork-harness/references/scenario-schema.md +2 -2
  5. package/.claude/skills/cowork-harness/references/task-recipes.md +2 -2
  6. package/CHANGELOG.md +249 -0
  7. package/CONTRIBUTING.md +7 -6
  8. package/DESIGN.md +1 -1
  9. package/README.md +19 -13
  10. package/RELEASING.md +7 -5
  11. package/SPEC.md +7 -0
  12. package/dist/agent/session.js +11 -4
  13. package/dist/cli.js +24 -5
  14. package/dist/critique/command.js +426 -48
  15. package/dist/critique/evaluator.js +111 -43
  16. package/dist/critique/evidence.js +15 -2
  17. package/dist/critique/limitations.js +43 -27
  18. package/dist/critique/package-evidence.js +130 -12
  19. package/dist/decide/llm-transport.js +3 -1
  20. package/dist/hostloop/canusetool-gate.js +4 -4
  21. package/dist/hostloop/safety.js +6 -5
  22. package/dist/run/analyze-artifact-runtime.js +3 -4
  23. package/dist/run/analyze-artifact.js +1 -1
  24. package/dist/run/artifacts.js +12 -0
  25. package/dist/run/chat-result.js +8 -5
  26. package/dist/run/diff.js +12 -2
  27. package/dist/run/doctor.js +65 -0
  28. package/dist/run/execute.js +53 -20
  29. package/dist/run/skill-flag-surface.js +14 -3
  30. package/dist/run/subagent-reasoning.js +58 -3
  31. package/dist/run/trace-view.js +38 -0
  32. package/dist/runtime/hostloop-prompt.js +23 -4
  33. package/dist/runtime/hostloop.js +9 -3
  34. package/dist/runtime/lima.js +13 -1
  35. package/dist/sync/cowork-sync.js +5 -5
  36. package/dist/types.js +1 -1
  37. package/docker/Dockerfile.agent +9 -0
  38. package/docs/README.md +8 -3
  39. package/docs/boundary.md +1 -1
  40. package/docs/chat.md +1 -1
  41. package/docs/critique.md +163 -24
  42. package/docs/debugging.md +36 -27
  43. package/docs/gotchas.md +2 -0
  44. package/docs/scenario.md +2 -1
  45. package/docs/subagents.md +1 -1
  46. package/examples/README.md +1 -1
  47. package/examples/replays/README.md +1 -1
  48. package/llms.txt +2 -1
  49. package/package.json +1 -1
  50. package/python/README.md +7 -2
  51. package/schema/critique-report.json +221 -0
  52. package/schema/run-result.json +27 -0
  53. package/scripts/bump-version.ts +5 -3
  54. package/scripts/check-versions.ts +45 -11
  55. package/scripts/release-preflight.ts +0 -2
@@ -3,8 +3,8 @@ name: cowork-harness
3
3
  description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. Also when iterating or hardening a skill across fixes, or grounding a skill's self-critique against its own run evidence — including a document-analysis skill (cap table, deck, financial model, transcript) that needs an uploaded file attached to be critiqued at all. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
4
4
  metadata:
5
5
  author: cowork-harness
6
- version: 1.7.0
7
- tracks-harness: cowork-harness 1.7.0 (baseline desktop-1.24012.1)
6
+ version: 1.9.0
7
+ tracks-harness: cowork-harness 1.9.0 (baseline desktop-1.24012.1)
8
8
  ---
9
9
 
10
10
  # cowork-harness
@@ -22,7 +22,7 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
22
22
  allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
23
23
  the highest-value part. Read it.
24
24
 
25
- > **Version note:** the facts and `file:line` pointers here track `cowork-harness 1.7.0` (baseline
25
+ > **Version note:** the facts and `file:line` pointers here track `cowork-harness 1.9.0` (baseline
26
26
  > `desktop-1.24012.1`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
27
27
  > `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
28
28
 
@@ -39,9 +39,42 @@ Before the first command, confirm the CLI is reachable and **fail loud** (never
39
39
 
40
40
  - **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
41
41
  - **Replay-only? Skip `doctor`.** Replaying committed cassettes needs no Docker, no staged agent, and no token — and every tier's `doctor` validates the auth token (the live tiers also Docker + the staged agent), so a ✗ there is expected, not a blocker. Go straight to `cowork-harness replay <cassette>`.
42
- - **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 1.7.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=1.7.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=1.7.0"`. **Pin `@>=1.7.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
43
-
44
- What the ≥ 1.7.0 floor gates, by release:
42
+ - **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 1.9.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=1.9.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=1.9.0"`. **Pin `@>=1.9.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
43
+
44
+ What the ≥ 1.9.0 floor gates, by release:
45
+
46
+ - **1.9.0 (critique report schema + shareable, screenshot-safe output):**
47
+ `schema/critique-report.json` (descriptive, test-pinned — parse the report against it, not prose;
48
+ NOT §12-frozen while critique is EXPERIMENTAL), `gradedSkill` in the report (text header + JSON;
49
+ multi-skill pairing key — pair by `(gradedSkillHash, gradedSkill)`, since the hash alone keys the whole
50
+ plugin and cross-pairs its skills). Critique's text report and stderr diagnostics now **collapse host
51
+ paths to `~`** (a shared report or screenshot no longer leaks the username + filesystem layout; the JSON
52
+ report and persisted-artifact paths stay raw for machine consumers), a **missing or non-directory skill
53
+ folder is refused pre-spawn** (fast exit 2, no stray run dir left behind), and the report header + stderr
54
+ prefixes now read **`critique:`** (were `skill-critique:`).
55
+
56
+ - **1.8.0 (critique hardening + hostloop unpin):** `critique --skill <name>` (multi-skill-plugin
57
+ grading — a plugin root with no `--skill` is refused pre-spend; the evidence package gains the
58
+ invoked skill's `agents/<name>.md` + bounded `references/*.md` CONTENT), run-dir artifacts
59
+ (`critique-report.json` always, `critique-evidence-package.txt` when the evaluator ran,
60
+ `critique-salvage.json` on exit 2) + `--out <path>`, per-critique `costUsd` across all four
61
+ workloads, `findingFingerprint` per item (cross-INPUT clustering; HIGH-precision
62
+ LOW-recall — a match proves reproduction, a mismatch does NOT prove non-reproduction, the same
63
+ finding reworded fingerprints differently; skillHash stays the cross-FIX
64
+ key), gate-answer `--answer` echo in the report, per-item-tolerant evaluator parse
65
+ (`droppedEvaluatorItems`), `skillMdTruncated` (a readable-but-oversized SKILL.md is flagged
66
+ "graded a cut copy" — distinct from missing/unreadable), `subagents[].webSearches` +
67
+ `trace --view subagent-research` (live/record lane), and the hostloop uploads-bullet fix (Read of
68
+ an uploaded file at the advertised path works — no copy-into-outputs workaround needed). Also:
69
+ **`critique --fidelity hostloop`** (the container-tier pin is lifted — resume-continuity proven
70
+ live on the native binary; `microvm`/`protocol`/`cowork` stay refused, each with a tagged reason),
71
+ **`skill --allow-host-writes`** (hostloop writable-folder consent; `critique` forwards it to both
72
+ turns), `verdictProvenance` stamped on every critique report (advisory self-run, not an
73
+ attestation), evidence caps 64KB SKILL.md / 32KB transcript / 144KB package (a flagship-sized
74
+ SKILL.md grades untruncated; ~2–2.5× evaluator cost on large skills), pinned sessions stamp their
75
+ fidelity and a cross-tier `--resume` fails loud pre-spawn, and `doctor`'s advisory
76
+ `image-freshness` check (pulled agent image behind the published GHCR one → warn + re-pull
77
+ remedy).
45
78
 
46
79
  - **1.7.0 (per-turn run-directory layout, single shape):** every run dir — `run`/`skill`/`chat`,
47
80
  single-turn or multi-turn (any `--session-id` + `--resume`, and **every `critique`**: task turn +
@@ -61,8 +94,9 @@ Before the first command, confirm the CLI is reachable and **fail loud** (never
61
94
  - **1.7.0 (limitation provenance):** every `critique` limitation in `critique --help` and
62
95
  docs/critique.md is tagged with WHY it exists — `[structural]` (permanent), `[unverified]` (unproven,
63
96
  **not** known-impossible), `[deliberate]`, `[not-built]`. **Read the tag before telling a user to
64
- design around a limitation.** In particular `critique`'s container-tier pin is `[unverified]`, not
65
- permanent: do not advise building a second hostloop test lane on the assumption it can never lift.
97
+ design around a limitation.** Worked example of why the tag matters: `critique`'s container-tier pin
98
+ was tagged `[unverified]`, not permanent — and 1.8.0 lifted it (hostloop allowed, see above) exactly
99
+ as the tag predicted a proof could.
66
100
 
67
101
  - **1.6.0 (`critique`'s skill-flag parity):** `critique` accepts most `skill` flags under the same names,
68
102
  which is what makes a document-analysis skill (cap table, deck, financial model, transcript)
@@ -414,7 +448,9 @@ cassette — has its own recipe:
414
448
  input + result are captured, not just errored ones. When iterating, tag generations with `--label` and
415
449
  pair a critique only with a `result.json` whose `fingerprint.skillHash` **matches** the skill that
416
450
  produced it (`inspect`/the run-index row surface a short `skillHash` prefix; `verify-run` warns when a
417
- kept run predates the current skill). **On a pre-1.5.0 CLI the `skill` lane emits no `fingerprint.skillHash` at all**, so a
451
+ kept run predates the current skill). **Multi-skill plugin caveat (post-1.7.0 CLIs with `--skill`):**
452
+ skillHash keys the whole MOUNTED plugin, so on a multi-skill plugin the hash alone cross-pairs
453
+ critiques of DIFFERENT skills — pair by the report's `(gradedSkillHash, gradedSkill)` pair. **On a pre-1.5.0 CLI the `skill` lane emits no `fingerprint.skillHash` at all**, so a
418
454
  pairing step there silently groups on an absent key instead of erroring — check the field is present, or
419
455
  require ≥ 1.5.0. See `docs/debugging.md`
420
456
  (repo-only) for the full loop.
@@ -679,9 +715,15 @@ repeats the assertion/replay-relevant ones alongside the schema (a scoped subset
679
715
  7. **`protocol` is rejected (not silently passed) if the scenario asserts egress** — boundary
680
716
  assertions need a sandboxed tier (`container`+). Good: this one fails loud by design.
681
717
 
682
- 8. **Read-only mounts are enforced; delete-deny is not.** `mode:r` mounts get a real `:ro` bind
683
- (a write fails in-guest). But `rw` vs `rwd` (write-but-no-delete on `outputs/` / connected folders) is
684
- *not* mount-enforced — `rm` succeeds and is only caught post-hoc by `no_delete_in_outputs`.
718
+ 8. **Read-only mounts are enforced; delete-deny is a HARNESS gap — production DOES enforce it.**
719
+ `mode:r` mounts get a real `:ro` bind (a write fails in-guest). But `rw` vs `rwd`
720
+ (write-but-no-delete on `outputs/` / connected folders) is *not* mount-enforced **in the harness** —
721
+ `rm` succeeds and is only caught post-hoc by `no_delete_in_outputs`. **Real Cowork enforces it live:**
722
+ `rm` under `outputs/` fails `Operation not permitted`, and a skill must request approval via
723
+ `allow_cowork_file_delete` to delete. Two consequences: a skill should not stage disposable scratch
724
+ under `outputs/` (in production, cleanup there costs an approval prompt), and a skill's
725
+ "catch-EPERM-then-request-approval" branch cannot be exercised at any harness tier (the `rm` just
726
+ succeeds here). Do not read this gotcha as "delete-deny may not be real in production" — it is real.
685
727
 
686
728
  9. **Keep `.env` out of any mounted folder** — it is copied into the sandbox and the token could
687
729
  leak. Put it at a working-dir or install root (token resolution: env > `--dotenv` > `./.env` >
@@ -1,6 +1,6 @@
1
1
  # CI recipe — replay vs live lanes
2
2
 
3
- Self-contained reference. Tracks `cowork-harness 1.7.0` (baseline `desktop-1.20186.1`).
3
+ Self-contained reference. Tracks `cowork-harness 1.9.0` (baseline `desktop-1.24012.1`).
4
4
 
5
5
  **Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
6
6
  job-summary reporter (verdict table, staleness findings, cost/turns when available):
@@ -13,7 +13,7 @@ job-summary reporter (verdict table, staleness findings, cost/turns when availab
13
13
  ```
14
14
 
15
15
  The Action's `version` input defaults to `latest` — intentional so a copy-pasted recipe tracks the current
16
- release; pin an exact version (e.g. `version: "1.7.0"`) for reproducible CI.
16
+ release; pin an exact version (e.g. `version: "1.9.0"`) for reproducible CI.
17
17
 
18
18
  Reach for the manual multi-step form below only when you need per-step control the Action's inputs don't
19
19
  cover (a custom flag combination, a different runner matrix per step, or `lint`/`verify-cassettes` gated
@@ -57,7 +57,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
57
57
  GitHub-hosted runners, no token/Docker/agent:
58
58
 
59
59
  ```yaml
60
- - run: npm i -g "cowork-harness@>=1.7.0"
60
+ - run: npm i -g "cowork-harness@>=1.9.0"
61
61
  - run: cowork-harness lint scenarios/*.yaml # no silent false-greens
62
62
  - run: cowork-harness verify-cassettes cassettes/ # privacy + staleness
63
63
  - run: cowork-harness replay cassettes/ # token-free content/structure
@@ -197,7 +197,7 @@ jobs:
197
197
  with: { node-version: '20' }
198
198
  - uses: actions/setup-python@v5
199
199
  with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
200
- - run: npm i -g "cowork-harness@>=1.7.0"
200
+ - run: npm i -g "cowork-harness@>=1.9.0"
201
201
  - run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
202
202
  - run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
203
203
  - run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
@@ -226,7 +226,7 @@ jobs:
226
226
  echo "live=true" >> "$GITHUB_OUTPUT"
227
227
  fi
228
228
  - if: steps.guard.outputs.live == 'true'
229
- run: npm i -g "cowork-harness@>=1.7.0"
229
+ run: npm i -g "cowork-harness@>=1.9.0"
230
230
  - if: steps.guard.outputs.live == 'true'
231
231
  run: cowork-harness run scenarios/ --output-format json
232
232
  env:
@@ -1,6 +1,6 @@
1
1
  # Fidelity tiers & answer paths
2
2
 
3
- Self-contained reference. Tracks `cowork-harness 1.7.0` (baseline `desktop-1.20186.1`).
3
+ Self-contained reference. Tracks `cowork-harness 1.9.0` (baseline `desktop-1.24012.1`).
4
4
 
5
5
  ## Fidelity tiers (`fidelity:` in the scenario)
6
6
 
@@ -24,6 +24,22 @@ Self-contained reference. Tracks `cowork-harness 1.7.0` (baseline `desktop-1.201
24
24
  `skill` (any tier) and `chat` (`protocol`/`container`/`hostloop`; only `microvm`/`cowork` unsupported); `run` rejects an extra `--fidelity`
25
25
  positional ("Fidelity is set by the scenario's `fidelity:` field, not a flag").
26
26
 
27
+ ### hostloop paths: what Read sees vs what bash sees (uploads included)
28
+
29
+ At `hostloop` the native file tools (Read/Write/Edit/Glob/Grep) run **on the host** and are gated by a
30
+ path-containment hook (production's own model), while `mcp__workspace__bash` runs **in the VM sidecar**
31
+ and sees `/sessions/<id>/...` paths. Consequences an agent (or a debugging author) must know:
32
+
33
+ - A `/sessions/...` path handed to `Read` fails ("is a VM path") — that split is **production-faithful**,
34
+ not a harness bug. Translate via the prompt's path-mapping table.
35
+ - **Uploads ARE Read-able at hostloop**: the staged uploads dir on the host is in the containment
36
+ allowlist (read-only — writes/edits there are blocked, matching production). If a Read of an uploads
37
+ path fails "outside this session's connected folders", the path *form* is wrong (e.g. a relative
38
+ `mnt/uploads/...`, which resolves against the outputs cwd) — the file itself is reachable at the
39
+ staged host uploads dir; it does not need to be copied into `outputs/` first.
40
+ - In real Cowork an upload is a **hardlink** in the session's uploads dir; the harness **copies** it
41
+ instead. Read-reachability is the same; only the advertised parent dir can differ.
42
+
27
43
  ### `microvm` prerequisites & lifecycle
28
44
 
29
45
  Requires macOS on arm64 and Lima (`brew install lima`; binary expected at
@@ -1,7 +1,7 @@
1
1
  # Scenario & session schema, assertion catalog, web_fetch, full gotchas
2
2
 
3
- Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 1.7.0`
4
- (baseline `desktop-1.20186.1`). If your checkout is newer, prefer the live `docs/scenario.md`,
3
+ Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 1.9.0`
4
+ (baseline `desktop-1.24012.1`). If your checkout is newer, prefer the live `docs/scenario.md`,
5
5
  `docs/session.md`, and `SPEC.md`.
6
6
 
7
7
  **Minimal scenario** — `prompt` is the only required field:
@@ -1,8 +1,8 @@
1
1
  # Task recipes — end-to-end paths for the jobs consumers actually do
2
2
 
3
3
  Each recipe composes facts that live scattered across SKILL.md and the other references into one
4
- decision path. Every one answers a question a real fleet owner had to work out the hard way. Facts track the harness version in SKILL.md's
5
- front-matter (currently 1.5.0). Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
4
+ decision path. Every one answers a question a real fleet owner had to work out the hard way.
5
+ Tracks `cowork-harness 1.9.0` (baseline `desktop-1.24012.1`), same as SKILL.md's front-matter. Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
6
6
  Recipe 3's `init-redact` shipped in 0.24.0 and are part of the current feature set — no version gate
7
7
  needed if your CLI meets SKILL.md's version floor.
8
8
 
package/CHANGELOG.md CHANGED
@@ -6,6 +6,255 @@ All notable changes to this project are documented here. The format is based on
6
6
 
7
7
  ## [Unreleased]
8
8
 
9
+ ## [1.9.0] — 2026-07-24
10
+
11
+ ### Added
12
+
13
+ - **`schema/critique-report.json`** — a descriptive, test-pinned schema for `critique`'s JSON report /
14
+ `critique-report.json` artifact, so automation consumers (budget pacers gating on `costUsd.complete`,
15
+ harvesters) parse field names/shapes from a schema instead of prose. Deliberately **not** a SPEC
16
+ §12-frozen surface (unlike `doctor.json`) — critique is EXPERIMENTAL and additive field changes are
17
+ expected; the schema says so in its own description, and a two-way sync test pins it against the
18
+ actual report builder on every branch (findings / infraFailure / evaluatorError).
19
+ - **`gradedSkill` in the critique report** (text header + JSON): the resolved `skills/<name>` the
20
+ packager graded under `--skill`/auto-selection. Load-bearing for multi-skill plugins:
21
+ `gradedSkillHash` keys the whole mounted plugin, so pairing by hash alone cross-pairs critiques of
22
+ *different* skills — pair by `(gradedSkillHash, gradedSkill)`. Docs updated accordingly.
23
+
24
+ ### Docs
25
+
26
+ - Truncation→DROPPED mechanics: a finding whose `evidence` quotes SKILL.md text past the packaging cap
27
+ fails citation-resolution and lands in DROPPED (the check runs against the *cut* copy) — documented
28
+ next to `skillMdTruncated` so a back-half DROPPED skew on an oversized skill has its cause named.
29
+ - `findingFingerprint` direction-of-inference: high-precision, LOW-RECALL — a match proves
30
+ reproduction; a mismatch does NOT prove non-reproduction (the same finding reworded fingerprints
31
+ differently). The Reproduction section says so before anyone concludes "didn't reproduce".
32
+ - Evaluator cost share: the two evaluator passes were ~3/4 of a measured e2e total — the
33
+ calibrate-then-`--evaluator-model` strategy is now in the cost section and `critique --help`, with
34
+ the armor-verification-is-default-evaluator-only caveat.
35
+ - SPEC §12 now names the critique report explicitly under **NOT covered**: `schema/critique-report.json`
36
+ is descriptive (parse against it, not prose) but not the compatibility contract while critique is
37
+ EXPERIMENTAL — its surface-baseline presence is for change visibility, and it is the promotion
38
+ candidate on the `doctor.json` template once critique stabilizes. README, llms.txt, and the shipped
39
+ skill point at the schema and the `(gradedSkillHash, gradedSkill)` pairing rule.
40
+
41
+ ### Fixed
42
+
43
+ - **critique's "Attached inputs" evidence no longer reports `(none)` when `mounts.json` is corrupt.**
44
+ `listAttachedInputs` derived connected-folder names from `loadVmPathContext`, which returns `null` for
45
+ BOTH an absent `mounts.json` (legitimately no mounts) and a present-but-unparseable one (the folder map
46
+ is UNKNOWN). Uploads already distinguished these (the ENOENT-vs-read-fault split), but folders — which
47
+ have no fixed-layout fallback — silently collapsed to `[]`, so a corrupt `mounts.json` rendered `(none)`,
48
+ telling the evaluator "the agent correctly saw no connected folder" when the truth was unknown. It now
49
+ surfaces a corrupt `mounts.json` as an explicit UNKNOWN note, completing the same
50
+ confabulation-vs-correct guard the uploads path already applies.
51
+ - **`diff` no longer treats two tool inputs that differ only past the ~2000-char cap as the same call.**
52
+ `canonicalizeInput` truncated a tool input's canonical JSON to a 2000-char cap and used that truncated
53
+ string as the tool-sequence equality key, so two `Write`/`Bash` calls sharing a long identical prefix
54
+ but differing only in the dropped tail compared as `op: "same"` in `diffToolSequence` — a false "no
55
+ change" that could flip the advisory `diff` exit code to 0. A truncated key now folds in a
56
+ `#<len>·<sha16>` hash of the full canonical string, so the key depends on the entire content while the
57
+ visible prefix stays readable in the hunk; both diff sides canonicalize identically, so the comparison
58
+ stays consistent.
59
+ - **`COWORK_VM_GATEWAY` is now validated as a canonical IPv4 literal.** The L2-microVM gateway override was
60
+ interpolated verbatim into a root-run guest `iptables -A OUTPUT -d <gateway>` command (via `sh -c`), so a
61
+ malformed or hostile value could inject shell syntax into privileged provisioning — or, more mundanely,
62
+ leave the firewall in an unknown state. `vmGatewayIp()` now rejects anything that isn't a canonical IPv4
63
+ literal (digits-and-dots only, octets 0–255, no leading zeros), failing loud instead of reaching the
64
+ shell. Operator-set env var, so this is defense-in-depth; no valid gateway value is affected.
65
+ - **The `agent.stderr.log` sink is now flushed before the teardown secret-scrub reads it.** The stderr sink
66
+ was piped fire-and-forget and never awaited, so bytes still buffered when `scrubRawRunLogs` read the file
67
+ could land raw *afterwards* — a persisted-secret leak in a narrow teardown window. `LiveAgentSession` now
68
+ pipes it with `{ end: false }` and ends+awaits it in the same session-teardown drain that already flushes
69
+ `events.jsonl` / `control-out.jsonl`, so the session generator resolves only after the sink is fully
70
+ flushed — the scrub always sees the complete log.
71
+ - **`critique` no longer prints raw host paths in its report or diagnostics.** The text report's `run dir:`
72
+ line, the `inspect <dir>` hints, the write-failure diagnostics, and the echoed skill-folder path all
73
+ printed absolute `$HOME`-rooted paths, so a shared report or screenshot leaked the username + filesystem
74
+ layout (it landed in a video frame). critique was the one rendering path in the CLI that never called
75
+ `tildeify`, while `skill`/`run` scrub unconditionally. Every human-facing path is now collapsed to `~`;
76
+ the JSON report and persisted-artifact paths stay raw (machine data a consumer feeds back to a tool). The
77
+ `--demo` rejection is unchanged — it was never the fix — but its reason now notes the report already
78
+ collapses paths.
79
+ - **`critique` fails fast on a missing or non-directory skill folder.** A typo'd/absent positional folder
80
+ previously minted a session and spawned the task turn before infra-failing (exit 2), leaving a stray run
81
+ dir behind. `resolveCritiquedSkillDir` now `existsSync`/`isDirectory`-checks the folder up front — before
82
+ any session is minted or spawned — so a bad path exits 2 immediately with nothing left on disk. A
83
+ present-but-`SKILL.md`-less folder still defers to the packager's degraded flow, unchanged.
84
+ - **`critique`'s report header and stderr diagnostics now say `critique:` (were `skill-critique:`).** A
85
+ leftover label from the `scripts/skill-critique.ts` instrument; the invoked command is `critique`.
86
+ Cosmetic, no schema change.
87
+
88
+ ## [1.8.0] — 2026-07-23
89
+
90
+ ### Added
91
+
92
+ - **`check:versions` closes the stale-pin gap a docs audit found.** Three new enforcement surfaces:
93
+ every companion-skill `references/*.md` `(baseline desktop-X.Y.Z)` pin must match SKILL.md's
94
+ `tracks-harness` baseline; `task-recipes.md` now carries the same guarded
95
+ `` Tracks `cowork-harness X.Y.Z` `` stamp as the other references (and is a `bump-version` target);
96
+ DESIGN.md's one present-tense "currently **X**, per `baselines/desktop-Y.json`" sentence must name the
97
+ max committed baseline + its `agentVersion` (its dated verification-pass notes stay exempt). A new
98
+ docs-index sync test keeps README's Documentation table and docs/README.md's Guides table from
99
+ drifting apart (that drift is how `critique.md` went missing from the README table).
100
+
101
+ - **`doctor` warns when a *pulled* agent image is behind the published one.** For the
102
+ `container`/`hostloop`/`cowork` tiers, a new advisory `image-freshness` check compares the local agent
103
+ image's registry digest against the current published `ghcr.io/yaniv-golan/cowork-agent-base:2` (or
104
+ `-full`) and warns — with a re-pull + retag remedy — when they diverge. Best-effort and never blocking:
105
+ a locally-built image (no registry digest), an offline host, `docker buildx` being unavailable, or a
106
+ custom `COWORK_AGENT_IMAGE` is a quiet skip, never a false "stale"; only a pulled image incurs a network
107
+ probe. The `doctor --output-format json` envelope gains an `image-freshness` entry (the open
108
+ `checks[].id` set — SPEC §12 — already permits this).
109
+ - `critique` now stamps `verdictProvenance` on every report (JSON key + text "verdict scope" line): the
110
+ verdict is an advisory self-run, not an independent attestation.
111
+ - **`critique --skill <name>` — plugin-aware grading.** A multi-skill plugin root (`skills/<name>/SKILL.md`,
112
+ no root SKILL.md) previously graded a missing file, downgrading every coverage finding to "not
113
+ adjudicable". `--skill` selects the invoked skill's folder for the packager (selection only — the
114
+ positional folder is still what both turns mount, and `fingerprint.skillHash` is unchanged: per-plugin,
115
+ not per-skill); a multi-skill root with no `--skill` is refused **before any model spend**, naming the
116
+ available skills; a single-skill plugin auto-selects with a notice. The evidence package now also
117
+ carries the invoked skill's `agents/<name>.md` body and bounded `references/*.md` **content** (not just
118
+ filenames), and the report carries an advisory `skillInvocationObserved` (false = the graded run's own
119
+ `skillActivity` never mentions the selected skill).
120
+ - **`critique` persists run-dir artifacts.** Every critique writes `critique-report.json` (the
121
+ machine-readable report); when the evaluator ran, `critique-evidence-package.txt` (the exact armored
122
+ corpus it graded against — re-grade a disputed finding offline); on an instrument failure (exit 2),
123
+ `critique-salvage.json` (the self-report + each evaluator pass's RAW reply captured **pre-parse**).
124
+ New `--out <path>` also writes the selected-format report to a file. The two `not-built` limitations
125
+ these close (`evidence-not-persisted`, `report-stdout-only`) are removed from the registry.
126
+ - **Per-critique cost across all four workloads.** The report's `costUsd` sums the task turn, reflection
127
+ turn, and BOTH evaluator passes (whose usage the transport previously discarded), marked `INCOMPLETE`
128
+ whenever any workload is unpriced. The header also carries the pinned `fidelity` plus the graded turn's
129
+ recorded `gradedEffectiveFidelity`/`gradedBaseline`, and the graded run's resolved gate answers are
130
+ echoed as copy-pasteable `--answer` lines (JSON: `gateAnswers`).
131
+ - **`critique` evaluator parse is per-item tolerant.** One malformed item in an `{"items":[...]}` reply
132
+ previously discarded the whole document ("no valid JSON found", a broken discovery run). Valid items now
133
+ survive; malformed ones are dropped AND counted (`droppedEvaluatorItems`, surfaced in both report
134
+ formats). The integrity canary is recognized by its `idea` alone (a mutated echo still proves
135
+ instruction-following) and stripped before dedup, so a full-document + canary-only-restatement reply no
136
+ longer trips the ambiguity throw. Fail-loud preserved: garbage, or all-malformed-with-no-canary, still
137
+ throws — now naming which field check failed.
138
+ - **Sub-agent research is observable end-to-end.** A sub-agent's WebSearch never enters the main
139
+ `toolCounts`/`webSearches[]`; its query + result are now captured from the child session transcript as
140
+ `subagents[].webSearches` (bounded, live/record lane only — absent on replay, and absence is never
141
+ evidence of no research), surfaced by the new **`trace --view subagent-research`**, and packaged into
142
+ critique's evidence as a "Sub-agent research" section so "researched" claims are groundable.
143
+ - Each critique item now carries a **`findingFingerprint`** (sha16 over the normalized
144
+ idea + classification + recommendedAction, deliberately excluding the input-specific evidence excerpt) —
145
+ clusters the SAME finding across DIFFERENT inputs, complementing `skillHash` (same skill across fixes).
146
+ docs/critique.md gains a "Reproduction" section documenting the ≥2-run discipline.
147
+ - SKILL.md truncation is now reported distinctly: a readable-but-oversized SKILL.md marks the report
148
+ `skillMdTruncated` ("the evaluator graded a cut copy") instead of being indistinguishable from a fully
149
+ packaged one; only missing/unreadable still forces the mechanical "already-covered" downgrade — the
150
+ `--help`/docs limitation wording now says so.
151
+
152
+ ### Changed
153
+
154
+ - **Published agent images (`cowork-agent-base` / `cowork-agent-full`) now carry OCI metadata labels**
155
+ (`org.opencontainers.image.{title,description,source,documentation,licenses}`), so their GHCR package
156
+ pages render a description, repo link, license, and the "contains no Anthropic binary" provenance note
157
+ instead of appearing bare. The `publish-image` workflow sets the `full` variant's title/description via
158
+ `--label` so its page isn't mislabeled by the base image's baked-in defaults.
159
+ - `critique` evidence caps raised: SKILL.md 16KB→64KB, transcript 16KB→32KB, overall package 48KB→144KB
160
+ (the overall cap sits above the worst-case per-section sum, agents/references/research sections included), so
161
+ a flagship-sized (~51KB) SKILL.md no longer grades permanently truncated. Increases per-critique evaluator
162
+ token cost on large skills (~2–2.5×).
163
+ - `sync`'s two code-tripwire warnings (the `getMcpSkillSources` caller-count and MCP-skills-capability
164
+ checks) now carry a self-contained instruction — re-verify whether MCP servers can contribute skills and
165
+ whether the harness must model MCP-contributed skill sources — instead of pointing at a reference the
166
+ published repo does not carry.
167
+ - **Pinned sessions stamp their fidelity tier on `session.json`, and a cross-tier `--resume` fails loud**
168
+ (pre-spawn, with a "re-run at `--fidelity <stamped>`" remedy). The agent's native conversation store is
169
+ tier-LOCAL — container persists it under the work tree, hostloop under the host config dir — so resuming
170
+ a session at a different tier would hand the binary a `--resume` for a conversation its store has never
171
+ seen. Legacy stampless manifests (pre-dating the stamp) are let through with a warning; every manifest
172
+ written from now on carries the stamp. `readSessionManifest` gains a required `expectedFidelity`
173
+ argument.
174
+
175
+ ### Docs
176
+
177
+ - README documents that the agent images are **published to GHCR** and can be pulled + retagged instead of
178
+ built from scratch (`:2` floats to the latest release; `:2-<version>` pins an immutable per-release
179
+ build), and that the harness resolves the *local* tag — so a stale local image shadows the published one;
180
+ re-pull after upgrading. The `doctor` command row notes the new freshness warning.
181
+
182
+ ### Fixed
183
+
184
+ - **hostloop uploads bullet advertised a non-readable path.** The dynamic "## Shell access" section
185
+ rendered the uploads mapping's file-tool side as `dirname(upload.hostPath)` — the user's original
186
+ source dir, which the path-containment gate does not allow — while the base prompt pointed at the
187
+ correct staged dir. An agent following the bullet got "outside this session's connected folders" and
188
+ worked around it via copy-into-outputs + `rm`, tripping a spurious `outputs-delete` fail. The bullet
189
+ now advertises the staged uploads dir — the SAME hoisted value the path gate allows, so the prompt and
190
+ the gate cannot disagree. (Live-verified: hostloop `--upload` + Read-tool read succeeds at the
191
+ advertised path.)
192
+ - The task-turn timeout kill message now names `--timeout` and the 10-minute default; a "missing"
193
+ SKILL.md report note now points at `--skill` / the invoked skill's folder (the multi-skill-plugin-root
194
+ cause) instead of only stating the symptom.
195
+ - Shipped-doc corrections that induced field misdiagnoses: the cowork-harness skill's gotcha #8 now states
196
+ that **production enforces** outputs delete-deny (EPERM + approval) and the harness gap is
197
+ detection-only; docs/critique.md documents that a WebSearch produces **no search-host entries in the
198
+ container egress.log** (an `api.anthropic.com`-only log is consistent with research working —
199
+ live-verified with a first-party capture) and that sub-agent searches don't increment the main
200
+ `toolCounts.WebSearch`; the shipped fidelity reference explains the hostloop Read-vs-bash path split
201
+ including uploads readability.
202
+ - **`critique` now runs at `--fidelity hostloop` as well as `container` — the container-tier pin is
203
+ lifted.** The pin existed because the reflection turn *resumes* the task turn and resume-continuity was
204
+ only proven for the container Linux ELF; a live two-turn proof at hostloop's **native** agent binary
205
+ (`test/live-contract.test.ts`, "resume-continuity proof at hostloop") cleared it, and a live
206
+ `critique --fidelity hostloop` e2e validates the full protocol there. `microvm`/`protocol`/`cowork` stay
207
+ refused, each with its own stated reason: the single `container-tier-only` limitation is replaced by
208
+ three tagged ones — `microvm-tier-refused` `[unverified]` (a resume-continuity proof at the microVM guest
209
+ would lift it), `protocol-tier-refused` `[not-built]` (protocol never plumbs a session id/`--resume`),
210
+ and `cowork-tier-refused` `[deliberate]` (the synced loop gate would make the graded tier
211
+ baseline-dependent, adding noise to skillHash-paired generation comparisons).
212
+ - **`skill` accepts `--allow-host-writes`** (and `critique` forwards it to BOTH turns) — the
213
+ hostloop writable-connected-folder consent that previously only `chat` (its own flag) and `run` (the
214
+ `allow_host_writes: true` scenario field) could grant; a plain `skill --fidelity hostloop --folder X`
215
+ was refused with no way to consent at all. Folder-less runs (skill dir + uploads) still need no consent —
216
+ uploads and skill/plugin mounts are read-only.
217
+ - **`critique`'s spaced flag parser no longer silently grabs the next flag as a value.** `critique
218
+ <folder> --prompt --output-format json` (a forgotten `--prompt` value) swallowed `--output-format` as the
219
+ prompt AND dropped the real flag, then ran a four-workload critique on the wrong input. The spaced form
220
+ now fails loud and points at the equals escape hatch (`--prompt=<value>`) for a value that intentionally
221
+ starts with `-`.
222
+ - **`critique`'s subprocess byte cap is now one combined stdout+stderr budget.** `boundedSpawn` kept two
223
+ independent per-stream counters, so a looping or hostile child splitting output across both streams could
224
+ buffer ~2× the documented cap before either tripped. It now charges both streams against a single budget
225
+ and slices the terminal chunk to the remaining room, so captured output never exceeds the cap.
226
+ - **`critique`'s verbatim citation-grounding is now case-sensitive.** The mechanical check that drops any
227
+ finding whose cited evidence is not a verbatim excerpt normalized whitespace AND folded case — so a
228
+ case-altered (paraphrased) citation resolved as "grounded," weakening the principal defense against
229
+ evaluator hallucination. It now matches case-exact; whitespace reflow stays tolerated (models reflow
230
+ spacing when quoting — they don't change case).
231
+ - **critique's "Attached inputs" evidence no longer reports `(none)` when the uploads dir is unreadable.**
232
+ `listAttachedInputs` caught every `readdirSync` failure identically, so an unreadable uploads dir
233
+ (`EACCES`/`ENOTDIR`/…) collapsed to the same `(none)` as a legitimately-absent one — telling the
234
+ evaluator "the agent correctly saw no file" when attachment presence was actually UNKNOWN. It now
235
+ distinguishes `ENOENT` (absent) from a genuine read fault and surfaces the uncertainty loudly.
236
+ - **critique flags a partly-corrupt archived turn-1 transcript instead of grading it as clean.**
237
+ `readTurn1Transcript` skipped malformed JSONL rows (resilient) but returned `degraded: false` on the
238
+ first valid transcript record regardless — so archive corruption vanished from evidence health and the
239
+ evaluator graded a silently-incomplete transcript as solid ground truth. It now scans the whole
240
+ (turn-1-only) archive and sets `turn1SliceDegraded` when a malformed row was skipped or the archive
241
+ doesn't hold exactly one transcript record; the transcript is still delivered, just flagged.
242
+ - **critique flags a crashed task turn instead of grading evidence from it.** `taskTurnInfraFailure` only
243
+ treated a *killed* (timed-out / byte-capped) task turn as an instrument failure — a task that exited
244
+ nonzero without ever printing a parseable result envelope (a crash after the early `[status]` line)
245
+ slipped through, and its `[status]`-recovered run dir was reflected on and graded as a legitimate task.
246
+ It now also flags a nonzero exit with no parseable envelope. Deliberately narrow: a completed run that
247
+ reported a failing verdict (`ok:false` / `result:"error"` with a valid envelope) stays gradeable.
248
+ - **A nested unreadable output subtree no longer persists a partial file list as complete.** The
249
+ `workspaceFiles` / `artifacts` list was recorded as UNAVAILABLE (`undefined`, the evidence-unavailable
250
+ convention) only when the workspace ROOT was unobservable; a nested `EACCES`/`EIO` subtree left the walk
251
+ partial (`walkComplete: false`) but still persisted the incompletely-enumerated list as if complete. An
252
+ authored file inside the unreadable subtree then vanished with no signal, so an absence-sensitive
253
+ consumer (`delivered_clean` / `ended_with_question`, the replay `diff`, `scaffold`, `file_exists`) could
254
+ read it as absent — a silent false-clean. All three RunResult producers (run, partial-salvage, chat) now
255
+ route through one shared `trustedWorkspaceFiles` gate that collapses a missing root OR any incomplete
256
+ walk to `undefined`, and the run lanes emit a `::warning::` naming the unreadable subtree.
257
+
9
258
  ## [1.7.0] — 2026-07-22
10
259
 
11
260
  ### Added
package/CONTRIBUTING.md CHANGED
@@ -12,7 +12,7 @@ npm run format:check # prettier over src/ + test/ .ts — the most common first
12
12
  npm run typecheck # tsc --noEmit
13
13
  npm test # vitest
14
14
  npm run build # -> dist/
15
- npm run ci # typecheck + build + test — the core TS gate (CI's `unit` job runs more; see below)
15
+ npm run ci # typecheck + build + test — the core TS gate (CI's `build` + `test` jobs run more; see below)
16
16
  ```
17
17
 
18
18
  Before pushing: `npm run format:check` and `npm run ci` (see RELEASING.md's release checklist for the full pre-tag list).
@@ -25,10 +25,11 @@ Extra prerequisites for specific stages:
25
25
  - **Docker (arm64)** — required for `boundary-check` and the **L1 `container`** + `hostloop` fidelity tiers (the container sandbox + agent image).
26
26
  - **Lima (`limactl`, macOS arm64)** — required only for the **L2 `microvm`** tier and the `vm` commands; the guest runs on Apple Virtualization.framework (`vmType: vz`). `microvm` does **not** use Docker. (`cowork-harness doctor --tier microvm` checks for Lima, not Docker.)
27
27
 
28
- CI Stage 1 (the `unit` job in `.github/workflows/ci.yml`) does **not** invoke `npm run ci`. It runs the
29
- gate steps individually — e.g. `check:versions`, `format:check`, `typecheck`, `npm test`, `build`, a CLI
30
- smoke (`node dist/cli.js list`), three token-free `replay` gate fixtures, `verify-cassettes`, `lint`, and
31
- source-guard checks (see `ci.yml` for the authoritative list). Only `release.yml` calls `npm run ci`.
28
+ CI Stage 1 (the `build` job in `.github/workflows/ci.yml`) does **not** invoke `npm run ci`. It runs the
29
+ gate steps individually — e.g. `check:versions`, `format:check`, `typecheck`, `build`, a CLI smoke
30
+ (`node dist/cli.js list`), three token-free `replay` gate fixtures, `verify-cassettes`, `lint`, and
31
+ source-guard checks — while the unit suite runs separately as the 4-shard `test` job (see `ci.yml` for
32
+ the authoritative list). Only `release.yml` calls `npm run ci`.
32
33
 
33
34
  > **Cutting a release?** See [RELEASING.md](./RELEASING.md) for the branch → PR → tag → publish flow.
34
35
 
@@ -85,7 +86,7 @@ When a Desktop release moves something `sync` doesn't read, it reports an `unkno
85
86
  ## Commit & PR
86
87
 
87
88
  - Conventional, imperative commit subjects (`add …`, `fix …`, `parity: sync to <ver>`).
88
- - Open PRs against `main`. CI runs a six-stage pipeline (`unit`, `action-self-test`, `python`, `boundary`, `scenarios`, `parity-drift` — see `ci.yml`) on every PR including forks (no secrets needed) except `scenarios`; live scenarios only run on same-repo PRs/pushes with `ANTHROPIC_API_KEY` set.
89
+ - Open PRs against `main`. CI runs a seven-stage pipeline (`build`, `test`, `action-self-test`, `python`, `boundary`, `scenarios`, `parity-drift` — see `ci.yml`) on every PR including forks (no secrets needed) except `scenarios`; live scenarios only run on same-repo PRs/pushes with `ANTHROPIC_API_KEY` set.
89
90
  - Describe *what changed and why*; link issues.
90
91
 
91
92
  ## Reporting issues
package/DESIGN.md CHANGED
@@ -38,7 +38,7 @@ flowchart TB
38
38
  A Cowork session is the Desktop app driving an agent that runs **inside an Apple Virtualization.framework microVM**:
39
39
 
40
40
  - VM bundle: `~/Library/Application Support/Claude/vm_bundles/claudevm.bundle/` (`rootfs.img`, `sessiondata.img`, `efivars.fd`, `machineIdentifier`, `gvisorMacAddress`, `vmIP`); a warm pool at `vm_bundles/warm/<sha>/`.
41
- - In-VM agent: `~/Library/Application Support/Claude/claude-code-vm/<ver>/claude` (currently **2.1.205**, per `baselines/desktop-1.20186.1.json`, an **ELF aarch64** binary), spawned by the host **in cowork mode via the `CLAUDE_CODE_IS_COWORK=1` env var** — *not* a `--cowork` flag (that flag is plugin-scope and the staged agent rejects it; see the control-protocol note below). Each baseline records this ELF's `sha256` (`agentBinary.sha256`/`shaProvenance`), and the resolver integrity-checks the binary it's about to run against that hash by default (opt out `COWORK_HARNESS_VERIFY_AGENT_SHA=0`), so "the same pinned agent" is enforced, not just asserted. Old versions are re-downloadable + verifiable from the official release channel (see `docs/maintenance.md`).
41
+ - In-VM agent: `~/Library/Application Support/Claude/claude-code-vm/<ver>/claude` (currently **2.1.217**, per `baselines/desktop-1.24012.1.json`, an **ELF aarch64** binary), spawned by the host **in cowork mode via the `CLAUDE_CODE_IS_COWORK=1` env var** — *not* a `--cowork` flag (that flag is plugin-scope and the staged agent rejects it; see the control-protocol note below). Each baseline records this ELF's `sha256` (`agentBinary.sha256`/`shaProvenance`), and the resolver integrity-checks the binary it's about to run against that hash by default (opt out `COWORK_HARNESS_VERIFY_AGENT_SHA=0`), so "the same pinned agent" is enforced, not just asserted. Old versions are re-downloadable + verifiable from the official release channel (see `docs/maintenance.md`).
42
42
  - Network: `vm_network_mode: "gvisor"`, egress through a userspace netstack with a **compiled domain allowlist**; off-list partners rejected (`partner rejected: entry not on compiled allowlist`).
43
43
  - Control plane: Electron renderer→main typed IPC on channels named `$eipc_message$_<per-build-UUID>_$_claude.web_$_<Class>_$_<method>`, every handler validating `event.senderFrame.url` against a trusted-origin allowlist. The session manager is `LocalAgentModeSessions` (80 methods: `start`, `sendMessage`, `setDraftSessionFolders`, `onToolPermissionRequest`, `respondToToolPermission`, `getTranscript`, `onEvent`, …), bridged to the renderer as `window.cowork`.
44
44