cowork-harness 1.7.0 → 1.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/cowork-harness/SKILL.md +39 -11
- package/.claude/skills/cowork-harness/references/ci-recipe.md +5 -5
- package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +17 -1
- package/.claude/skills/cowork-harness/references/scenario-schema.md +2 -2
- package/.claude/skills/cowork-harness/references/task-recipes.md +2 -2
- package/CHANGELOG.md +170 -0
- package/CONTRIBUTING.md +7 -6
- package/DESIGN.md +1 -1
- package/README.md +17 -11
- package/RELEASING.md +7 -5
- package/dist/cli.js +24 -5
- package/dist/critique/command.js +394 -40
- package/dist/critique/evaluator.js +111 -43
- package/dist/critique/evidence.js +15 -2
- package/dist/critique/limitations.js +43 -27
- package/dist/critique/package-evidence.js +120 -12
- package/dist/decide/llm-transport.js +3 -1
- package/dist/hostloop/canusetool-gate.js +4 -4
- package/dist/hostloop/safety.js +6 -5
- package/dist/run/analyze-artifact-runtime.js +3 -4
- package/dist/run/analyze-artifact.js +1 -1
- package/dist/run/artifacts.js +12 -0
- package/dist/run/chat-result.js +8 -5
- package/dist/run/doctor.js +65 -0
- package/dist/run/execute.js +48 -16
- package/dist/run/skill-flag-surface.js +13 -2
- package/dist/run/subagent-reasoning.js +58 -3
- package/dist/run/trace-view.js +38 -0
- package/dist/runtime/hostloop-prompt.js +23 -4
- package/dist/runtime/hostloop.js +9 -3
- package/dist/sync/cowork-sync.js +5 -5
- package/dist/types.js +1 -1
- package/docker/Dockerfile.agent +9 -0
- package/docs/README.md +8 -3
- package/docs/boundary.md +1 -1
- package/docs/chat.md +1 -1
- package/docs/critique.md +134 -23
- package/docs/debugging.md +36 -27
- package/docs/gotchas.md +2 -0
- package/docs/subagents.md +1 -1
- package/examples/README.md +1 -1
- package/examples/replays/README.md +1 -1
- package/llms.txt +1 -1
- package/package.json +1 -1
- package/python/README.md +7 -2
- package/schema/run-result.json +27 -0
- package/scripts/bump-version.ts +5 -3
- package/scripts/check-versions.ts +45 -11
- package/scripts/release-preflight.ts +0 -2
|
@@ -3,8 +3,8 @@ name: cowork-harness
|
|
|
3
3
|
description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. Also when iterating or hardening a skill across fixes, or grounding a skill's self-critique against its own run evidence — including a document-analysis skill (cap table, deck, financial model, transcript) that needs an uploaded file attached to be critiqued at all. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
|
|
4
4
|
metadata:
|
|
5
5
|
author: cowork-harness
|
|
6
|
-
version: 1.
|
|
7
|
-
tracks-harness: cowork-harness 1.
|
|
6
|
+
version: 1.8.0
|
|
7
|
+
tracks-harness: cowork-harness 1.8.0 (baseline desktop-1.24012.1)
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
# cowork-harness
|
|
@@ -22,7 +22,7 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
|
|
|
22
22
|
allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
|
|
23
23
|
the highest-value part. Read it.
|
|
24
24
|
|
|
25
|
-
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 1.
|
|
25
|
+
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 1.8.0` (baseline
|
|
26
26
|
> `desktop-1.24012.1`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
|
|
27
27
|
> `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
|
|
28
28
|
|
|
@@ -39,9 +39,30 @@ Before the first command, confirm the CLI is reachable and **fail loud** (never
|
|
|
39
39
|
|
|
40
40
|
- **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
|
|
41
41
|
- **Replay-only? Skip `doctor`.** Replaying committed cassettes needs no Docker, no staged agent, and no token — and every tier's `doctor` validates the auth token (the live tiers also Docker + the staged agent), so a ✗ there is expected, not a blocker. Go straight to `cowork-harness replay <cassette>`.
|
|
42
|
-
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 1.
|
|
43
|
-
|
|
44
|
-
What the ≥ 1.
|
|
42
|
+
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 1.8.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=1.8.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=1.8.0"`. **Pin `@>=1.8.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
|
|
43
|
+
|
|
44
|
+
What the ≥ 1.8.0 floor gates, by release:
|
|
45
|
+
|
|
46
|
+
- **1.8.0 (critique hardening + hostloop unpin):** `critique --skill <name>` (multi-skill-plugin
|
|
47
|
+
grading — a plugin root with no `--skill` is refused pre-spend; the evidence package gains the
|
|
48
|
+
invoked skill's `agents/<name>.md` + bounded `references/*.md` CONTENT), run-dir artifacts
|
|
49
|
+
(`critique-report.json` always, `critique-evidence-package.txt` when the evaluator ran,
|
|
50
|
+
`critique-salvage.json` on exit 2) + `--out <path>`, per-critique `costUsd` across all four
|
|
51
|
+
workloads, `findingFingerprint` per item (cross-INPUT clustering; skillHash stays the cross-FIX
|
|
52
|
+
key), gate-answer `--answer` echo in the report, per-item-tolerant evaluator parse
|
|
53
|
+
(`droppedEvaluatorItems`), `skillMdTruncated` (a readable-but-oversized SKILL.md is flagged
|
|
54
|
+
"graded a cut copy" — distinct from missing/unreadable), `subagents[].webSearches` +
|
|
55
|
+
`trace --view subagent-research` (live/record lane), and the hostloop uploads-bullet fix (Read of
|
|
56
|
+
an uploaded file at the advertised path works — no copy-into-outputs workaround needed). Also:
|
|
57
|
+
**`critique --fidelity hostloop`** (the container-tier pin is lifted — resume-continuity proven
|
|
58
|
+
live on the native binary; `microvm`/`protocol`/`cowork` stay refused, each with a tagged reason),
|
|
59
|
+
**`skill --allow-host-writes`** (hostloop writable-folder consent; `critique` forwards it to both
|
|
60
|
+
turns), `verdictProvenance` stamped on every critique report (advisory self-run, not an
|
|
61
|
+
attestation), evidence caps 64KB SKILL.md / 32KB transcript / 144KB package (a flagship-sized
|
|
62
|
+
SKILL.md grades untruncated; ~2–2.5× evaluator cost on large skills), pinned sessions stamp their
|
|
63
|
+
fidelity and a cross-tier `--resume` fails loud pre-spawn, and `doctor`'s advisory
|
|
64
|
+
`image-freshness` check (pulled agent image behind the published GHCR one → warn + re-pull
|
|
65
|
+
remedy).
|
|
45
66
|
|
|
46
67
|
- **1.7.0 (per-turn run-directory layout, single shape):** every run dir — `run`/`skill`/`chat`,
|
|
47
68
|
single-turn or multi-turn (any `--session-id` + `--resume`, and **every `critique`**: task turn +
|
|
@@ -61,8 +82,9 @@ Before the first command, confirm the CLI is reachable and **fail loud** (never
|
|
|
61
82
|
- **1.7.0 (limitation provenance):** every `critique` limitation in `critique --help` and
|
|
62
83
|
docs/critique.md is tagged with WHY it exists — `[structural]` (permanent), `[unverified]` (unproven,
|
|
63
84
|
**not** known-impossible), `[deliberate]`, `[not-built]`. **Read the tag before telling a user to
|
|
64
|
-
design around a limitation.**
|
|
65
|
-
|
|
85
|
+
design around a limitation.** Worked example of why the tag matters: `critique`'s container-tier pin
|
|
86
|
+
was tagged `[unverified]`, not permanent — and 1.8.0 lifted it (hostloop allowed, see above) exactly
|
|
87
|
+
as the tag predicted a proof could.
|
|
66
88
|
|
|
67
89
|
- **1.6.0 (`critique`'s skill-flag parity):** `critique` accepts most `skill` flags under the same names,
|
|
68
90
|
which is what makes a document-analysis skill (cap table, deck, financial model, transcript)
|
|
@@ -679,9 +701,15 @@ repeats the assertion/replay-relevant ones alongside the schema (a scoped subset
|
|
|
679
701
|
7. **`protocol` is rejected (not silently passed) if the scenario asserts egress** — boundary
|
|
680
702
|
assertions need a sandboxed tier (`container`+). Good: this one fails loud by design.
|
|
681
703
|
|
|
682
|
-
8. **Read-only mounts are enforced; delete-deny is
|
|
683
|
-
(a write fails in-guest). But `rw` vs `rwd`
|
|
684
|
-
|
|
704
|
+
8. **Read-only mounts are enforced; delete-deny is a HARNESS gap — production DOES enforce it.**
|
|
705
|
+
`mode:r` mounts get a real `:ro` bind (a write fails in-guest). But `rw` vs `rwd`
|
|
706
|
+
(write-but-no-delete on `outputs/` / connected folders) is *not* mount-enforced **in the harness** —
|
|
707
|
+
`rm` succeeds and is only caught post-hoc by `no_delete_in_outputs`. **Real Cowork enforces it live:**
|
|
708
|
+
`rm` under `outputs/` fails `Operation not permitted`, and a skill must request approval via
|
|
709
|
+
`allow_cowork_file_delete` to delete. Two consequences: a skill should not stage disposable scratch
|
|
710
|
+
under `outputs/` (in production, cleanup there costs an approval prompt), and a skill's
|
|
711
|
+
"catch-EPERM-then-request-approval" branch cannot be exercised at any harness tier (the `rm` just
|
|
712
|
+
succeeds here). Do not read this gotcha as "delete-deny may not be real in production" — it is real.
|
|
685
713
|
|
|
686
714
|
9. **Keep `.env` out of any mounted folder** — it is copied into the sandbox and the token could
|
|
687
715
|
leak. Put it at a working-dir or install root (token resolution: env > `--dotenv` > `./.env` >
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# CI recipe — replay vs live lanes
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 1.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 1.8.0` (baseline `desktop-1.24012.1`).
|
|
4
4
|
|
|
5
5
|
**Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
|
|
6
6
|
job-summary reporter (verdict table, staleness findings, cost/turns when available):
|
|
@@ -13,7 +13,7 @@ job-summary reporter (verdict table, staleness findings, cost/turns when availab
|
|
|
13
13
|
```
|
|
14
14
|
|
|
15
15
|
The Action's `version` input defaults to `latest` — intentional so a copy-pasted recipe tracks the current
|
|
16
|
-
release; pin an exact version (e.g. `version: "1.
|
|
16
|
+
release; pin an exact version (e.g. `version: "1.8.0"`) for reproducible CI.
|
|
17
17
|
|
|
18
18
|
Reach for the manual multi-step form below only when you need per-step control the Action's inputs don't
|
|
19
19
|
cover (a custom flag combination, a different runner matrix per step, or `lint`/`verify-cassettes` gated
|
|
@@ -57,7 +57,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
|
|
|
57
57
|
GitHub-hosted runners, no token/Docker/agent:
|
|
58
58
|
|
|
59
59
|
```yaml
|
|
60
|
-
- run: npm i -g "cowork-harness@>=1.
|
|
60
|
+
- run: npm i -g "cowork-harness@>=1.8.0"
|
|
61
61
|
- run: cowork-harness lint scenarios/*.yaml # no silent false-greens
|
|
62
62
|
- run: cowork-harness verify-cassettes cassettes/ # privacy + staleness
|
|
63
63
|
- run: cowork-harness replay cassettes/ # token-free content/structure
|
|
@@ -197,7 +197,7 @@ jobs:
|
|
|
197
197
|
with: { node-version: '20' }
|
|
198
198
|
- uses: actions/setup-python@v5
|
|
199
199
|
with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
|
|
200
|
-
- run: npm i -g "cowork-harness@>=1.
|
|
200
|
+
- run: npm i -g "cowork-harness@>=1.8.0"
|
|
201
201
|
- run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
|
|
202
202
|
- run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
|
|
203
203
|
- run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
|
|
@@ -226,7 +226,7 @@ jobs:
|
|
|
226
226
|
echo "live=true" >> "$GITHUB_OUTPUT"
|
|
227
227
|
fi
|
|
228
228
|
- if: steps.guard.outputs.live == 'true'
|
|
229
|
-
run: npm i -g "cowork-harness@>=1.
|
|
229
|
+
run: npm i -g "cowork-harness@>=1.8.0"
|
|
230
230
|
- if: steps.guard.outputs.live == 'true'
|
|
231
231
|
run: cowork-harness run scenarios/ --output-format json
|
|
232
232
|
env:
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Fidelity tiers & answer paths
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 1.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 1.8.0` (baseline `desktop-1.24012.1`).
|
|
4
4
|
|
|
5
5
|
## Fidelity tiers (`fidelity:` in the scenario)
|
|
6
6
|
|
|
@@ -24,6 +24,22 @@ Self-contained reference. Tracks `cowork-harness 1.7.0` (baseline `desktop-1.201
|
|
|
24
24
|
`skill` (any tier) and `chat` (`protocol`/`container`/`hostloop`; only `microvm`/`cowork` unsupported); `run` rejects an extra `--fidelity`
|
|
25
25
|
positional ("Fidelity is set by the scenario's `fidelity:` field, not a flag").
|
|
26
26
|
|
|
27
|
+
### hostloop paths: what Read sees vs what bash sees (uploads included)
|
|
28
|
+
|
|
29
|
+
At `hostloop` the native file tools (Read/Write/Edit/Glob/Grep) run **on the host** and are gated by a
|
|
30
|
+
path-containment hook (production's own model), while `mcp__workspace__bash` runs **in the VM sidecar**
|
|
31
|
+
and sees `/sessions/<id>/...` paths. Consequences an agent (or a debugging author) must know:
|
|
32
|
+
|
|
33
|
+
- A `/sessions/...` path handed to `Read` fails ("is a VM path") — that split is **production-faithful**,
|
|
34
|
+
not a harness bug. Translate via the prompt's path-mapping table.
|
|
35
|
+
- **Uploads ARE Read-able at hostloop**: the staged uploads dir on the host is in the containment
|
|
36
|
+
allowlist (read-only — writes/edits there are blocked, matching production). If a Read of an uploads
|
|
37
|
+
path fails "outside this session's connected folders", the path *form* is wrong (e.g. a relative
|
|
38
|
+
`mnt/uploads/...`, which resolves against the outputs cwd) — the file itself is reachable at the
|
|
39
|
+
staged host uploads dir; it does not need to be copied into `outputs/` first.
|
|
40
|
+
- In real Cowork an upload is a **hardlink** in the session's uploads dir; the harness **copies** it
|
|
41
|
+
instead. Read-reachability is the same; only the advertised parent dir can differ.
|
|
42
|
+
|
|
27
43
|
### `microvm` prerequisites & lifecycle
|
|
28
44
|
|
|
29
45
|
Requires macOS on arm64 and Lima (`brew install lima`; binary expected at
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# Scenario & session schema, assertion catalog, web_fetch, full gotchas
|
|
2
2
|
|
|
3
|
-
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 1.
|
|
4
|
-
(baseline `desktop-1.
|
|
3
|
+
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 1.8.0`
|
|
4
|
+
(baseline `desktop-1.24012.1`). If your checkout is newer, prefer the live `docs/scenario.md`,
|
|
5
5
|
`docs/session.md`, and `SPEC.md`.
|
|
6
6
|
|
|
7
7
|
**Minimal scenario** — `prompt` is the only required field:
|
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
# Task recipes — end-to-end paths for the jobs consumers actually do
|
|
2
2
|
|
|
3
3
|
Each recipe composes facts that live scattered across SKILL.md and the other references into one
|
|
4
|
-
decision path. Every one answers a question a real fleet owner had to work out the hard way.
|
|
5
|
-
|
|
4
|
+
decision path. Every one answers a question a real fleet owner had to work out the hard way.
|
|
5
|
+
Tracks `cowork-harness 1.8.0` (baseline `desktop-1.24012.1`), same as SKILL.md's front-matter. Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
|
|
6
6
|
Recipe 3's `init-redact` shipped in 0.24.0 and are part of the current feature set — no version gate
|
|
7
7
|
needed if your CLI meets SKILL.md's version floor.
|
|
8
8
|
|
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,176 @@ All notable changes to this project are documented here. The format is based on
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [1.8.0] — 2026-07-23
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- **`check:versions` closes the stale-pin gap a docs audit found.** Three new enforcement surfaces:
|
|
14
|
+
every companion-skill `references/*.md` `(baseline desktop-X.Y.Z)` pin must match SKILL.md's
|
|
15
|
+
`tracks-harness` baseline; `task-recipes.md` now carries the same guarded
|
|
16
|
+
`` Tracks `cowork-harness X.Y.Z` `` stamp as the other references (and is a `bump-version` target);
|
|
17
|
+
DESIGN.md's one present-tense "currently **X**, per `baselines/desktop-Y.json`" sentence must name the
|
|
18
|
+
max committed baseline + its `agentVersion` (its dated verification-pass notes stay exempt). A new
|
|
19
|
+
docs-index sync test keeps README's Documentation table and docs/README.md's Guides table from
|
|
20
|
+
drifting apart (that drift is how `critique.md` went missing from the README table).
|
|
21
|
+
|
|
22
|
+
- **`doctor` warns when a *pulled* agent image is behind the published one.** For the
|
|
23
|
+
`container`/`hostloop`/`cowork` tiers, a new advisory `image-freshness` check compares the local agent
|
|
24
|
+
image's registry digest against the current published `ghcr.io/yaniv-golan/cowork-agent-base:2` (or
|
|
25
|
+
`-full`) and warns — with a re-pull + retag remedy — when they diverge. Best-effort and never blocking:
|
|
26
|
+
a locally-built image (no registry digest), an offline host, `docker buildx` being unavailable, or a
|
|
27
|
+
custom `COWORK_AGENT_IMAGE` is a quiet skip, never a false "stale"; only a pulled image incurs a network
|
|
28
|
+
probe. The `doctor --output-format json` envelope gains an `image-freshness` entry (the open
|
|
29
|
+
`checks[].id` set — SPEC §12 — already permits this).
|
|
30
|
+
- `critique` now stamps `verdictProvenance` on every report (JSON key + text "verdict scope" line): the
|
|
31
|
+
verdict is an advisory self-run, not an independent attestation.
|
|
32
|
+
- **`critique --skill <name>` — plugin-aware grading.** A multi-skill plugin root (`skills/<name>/SKILL.md`,
|
|
33
|
+
no root SKILL.md) previously graded a missing file, downgrading every coverage finding to "not
|
|
34
|
+
adjudicable". `--skill` selects the invoked skill's folder for the packager (selection only — the
|
|
35
|
+
positional folder is still what both turns mount, and `fingerprint.skillHash` is unchanged: per-plugin,
|
|
36
|
+
not per-skill); a multi-skill root with no `--skill` is refused **before any model spend**, naming the
|
|
37
|
+
available skills; a single-skill plugin auto-selects with a notice. The evidence package now also
|
|
38
|
+
carries the invoked skill's `agents/<name>.md` body and bounded `references/*.md` **content** (not just
|
|
39
|
+
filenames), and the report carries an advisory `skillInvocationObserved` (false = the graded run's own
|
|
40
|
+
`skillActivity` never mentions the selected skill).
|
|
41
|
+
- **`critique` persists run-dir artifacts.** Every critique writes `critique-report.json` (the
|
|
42
|
+
machine-readable report); when the evaluator ran, `critique-evidence-package.txt` (the exact armored
|
|
43
|
+
corpus it graded against — re-grade a disputed finding offline); on an instrument failure (exit 2),
|
|
44
|
+
`critique-salvage.json` (the self-report + each evaluator pass's RAW reply captured **pre-parse**).
|
|
45
|
+
New `--out <path>` also writes the selected-format report to a file. The two `not-built` limitations
|
|
46
|
+
these close (`evidence-not-persisted`, `report-stdout-only`) are removed from the registry.
|
|
47
|
+
- **Per-critique cost across all four workloads.** The report's `costUsd` sums the task turn, reflection
|
|
48
|
+
turn, and BOTH evaluator passes (whose usage the transport previously discarded), marked `INCOMPLETE`
|
|
49
|
+
whenever any workload is unpriced. The header also carries the pinned `fidelity` plus the graded turn's
|
|
50
|
+
recorded `gradedEffectiveFidelity`/`gradedBaseline`, and the graded run's resolved gate answers are
|
|
51
|
+
echoed as copy-pasteable `--answer` lines (JSON: `gateAnswers`).
|
|
52
|
+
- **`critique` evaluator parse is per-item tolerant.** One malformed item in an `{"items":[...]}` reply
|
|
53
|
+
previously discarded the whole document ("no valid JSON found", a broken discovery run). Valid items now
|
|
54
|
+
survive; malformed ones are dropped AND counted (`droppedEvaluatorItems`, surfaced in both report
|
|
55
|
+
formats). The integrity canary is recognized by its `idea` alone (a mutated echo still proves
|
|
56
|
+
instruction-following) and stripped before dedup, so a full-document + canary-only-restatement reply no
|
|
57
|
+
longer trips the ambiguity throw. Fail-loud preserved: garbage, or all-malformed-with-no-canary, still
|
|
58
|
+
throws — now naming which field check failed.
|
|
59
|
+
- **Sub-agent research is observable end-to-end.** A sub-agent's WebSearch never enters the main
|
|
60
|
+
`toolCounts`/`webSearches[]`; its query + result are now captured from the child session transcript as
|
|
61
|
+
`subagents[].webSearches` (bounded, live/record lane only — absent on replay, and absence is never
|
|
62
|
+
evidence of no research), surfaced by the new **`trace --view subagent-research`**, and packaged into
|
|
63
|
+
critique's evidence as a "Sub-agent research" section so "researched" claims are groundable.
|
|
64
|
+
- Each critique item now carries a **`findingFingerprint`** (sha16 over the normalized
|
|
65
|
+
idea + classification + recommendedAction, deliberately excluding the input-specific evidence excerpt) —
|
|
66
|
+
clusters the SAME finding across DIFFERENT inputs, complementing `skillHash` (same skill across fixes).
|
|
67
|
+
docs/critique.md gains a "Reproduction" section documenting the ≥2-run discipline.
|
|
68
|
+
- SKILL.md truncation is now reported distinctly: a readable-but-oversized SKILL.md marks the report
|
|
69
|
+
`skillMdTruncated` ("the evaluator graded a cut copy") instead of being indistinguishable from a fully
|
|
70
|
+
packaged one; only missing/unreadable still forces the mechanical "already-covered" downgrade — the
|
|
71
|
+
`--help`/docs limitation wording now says so.
|
|
72
|
+
|
|
73
|
+
### Changed
|
|
74
|
+
|
|
75
|
+
- **Published agent images (`cowork-agent-base` / `cowork-agent-full`) now carry OCI metadata labels**
|
|
76
|
+
(`org.opencontainers.image.{title,description,source,documentation,licenses}`), so their GHCR package
|
|
77
|
+
pages render a description, repo link, license, and the "contains no Anthropic binary" provenance note
|
|
78
|
+
instead of appearing bare. The `publish-image` workflow sets the `full` variant's title/description via
|
|
79
|
+
`--label` so its page isn't mislabeled by the base image's baked-in defaults.
|
|
80
|
+
- `critique` evidence caps raised: SKILL.md 16KB→64KB, transcript 16KB→32KB, overall package 48KB→144KB
|
|
81
|
+
(the overall cap sits above the worst-case per-section sum, agents/references/research sections included), so
|
|
82
|
+
a flagship-sized (~51KB) SKILL.md no longer grades permanently truncated. Increases per-critique evaluator
|
|
83
|
+
token cost on large skills (~2–2.5×).
|
|
84
|
+
- `sync`'s two code-tripwire warnings (the `getMcpSkillSources` caller-count and MCP-skills-capability
|
|
85
|
+
checks) now carry a self-contained instruction — re-verify whether MCP servers can contribute skills and
|
|
86
|
+
whether the harness must model MCP-contributed skill sources — instead of pointing at a reference the
|
|
87
|
+
published repo does not carry.
|
|
88
|
+
- **Pinned sessions stamp their fidelity tier on `session.json`, and a cross-tier `--resume` fails loud**
|
|
89
|
+
(pre-spawn, with a "re-run at `--fidelity <stamped>`" remedy). The agent's native conversation store is
|
|
90
|
+
tier-LOCAL — container persists it under the work tree, hostloop under the host config dir — so resuming
|
|
91
|
+
a session at a different tier would hand the binary a `--resume` for a conversation its store has never
|
|
92
|
+
seen. Legacy stampless manifests (pre-dating the stamp) are let through with a warning; every manifest
|
|
93
|
+
written from now on carries the stamp. `readSessionManifest` gains a required `expectedFidelity`
|
|
94
|
+
argument.
|
|
95
|
+
|
|
96
|
+
### Docs
|
|
97
|
+
|
|
98
|
+
- README documents that the agent images are **published to GHCR** and can be pulled + retagged instead of
|
|
99
|
+
built from scratch (`:2` floats to the latest release; `:2-<version>` pins an immutable per-release
|
|
100
|
+
build), and that the harness resolves the *local* tag — so a stale local image shadows the published one;
|
|
101
|
+
re-pull after upgrading. The `doctor` command row notes the new freshness warning.
|
|
102
|
+
|
|
103
|
+
### Fixed
|
|
104
|
+
|
|
105
|
+
- **hostloop uploads bullet advertised a non-readable path.** The dynamic "## Shell access" section
|
|
106
|
+
rendered the uploads mapping's file-tool side as `dirname(upload.hostPath)` — the user's original
|
|
107
|
+
source dir, which the path-containment gate does not allow — while the base prompt pointed at the
|
|
108
|
+
correct staged dir. An agent following the bullet got "outside this session's connected folders" and
|
|
109
|
+
worked around it via copy-into-outputs + `rm`, tripping a spurious `outputs-delete` fail. The bullet
|
|
110
|
+
now advertises the staged uploads dir — the SAME hoisted value the path gate allows, so the prompt and
|
|
111
|
+
the gate cannot disagree. (Live-verified: hostloop `--upload` + Read-tool read succeeds at the
|
|
112
|
+
advertised path.)
|
|
113
|
+
- The task-turn timeout kill message now names `--timeout` and the 10-minute default; a "missing"
|
|
114
|
+
SKILL.md report note now points at `--skill` / the invoked skill's folder (the multi-skill-plugin-root
|
|
115
|
+
cause) instead of only stating the symptom.
|
|
116
|
+
- Shipped-doc corrections that induced field misdiagnoses: the cowork-harness skill's gotcha #8 now states
|
|
117
|
+
that **production enforces** outputs delete-deny (EPERM + approval) and the harness gap is
|
|
118
|
+
detection-only; docs/critique.md documents that a WebSearch produces **no search-host entries in the
|
|
119
|
+
container egress.log** (an `api.anthropic.com`-only log is consistent with research working —
|
|
120
|
+
live-verified with a first-party capture) and that sub-agent searches don't increment the main
|
|
121
|
+
`toolCounts.WebSearch`; the shipped fidelity reference explains the hostloop Read-vs-bash path split
|
|
122
|
+
including uploads readability.
|
|
123
|
+
- **`critique` now runs at `--fidelity hostloop` as well as `container` — the container-tier pin is
|
|
124
|
+
lifted.** The pin existed because the reflection turn *resumes* the task turn and resume-continuity was
|
|
125
|
+
only proven for the container Linux ELF; a live two-turn proof at hostloop's **native** agent binary
|
|
126
|
+
(`test/live-contract.test.ts`, "resume-continuity proof at hostloop") cleared it, and a live
|
|
127
|
+
`critique --fidelity hostloop` e2e validates the full protocol there. `microvm`/`protocol`/`cowork` stay
|
|
128
|
+
refused, each with its own stated reason: the single `container-tier-only` limitation is replaced by
|
|
129
|
+
three tagged ones — `microvm-tier-refused` `[unverified]` (a resume-continuity proof at the microVM guest
|
|
130
|
+
would lift it), `protocol-tier-refused` `[not-built]` (protocol never plumbs a session id/`--resume`),
|
|
131
|
+
and `cowork-tier-refused` `[deliberate]` (the synced loop gate would make the graded tier
|
|
132
|
+
baseline-dependent, adding noise to skillHash-paired generation comparisons).
|
|
133
|
+
- **`skill` accepts `--allow-host-writes`** (and `critique` forwards it to BOTH turns) — the
|
|
134
|
+
hostloop writable-connected-folder consent that previously only `chat` (its own flag) and `run` (the
|
|
135
|
+
`allow_host_writes: true` scenario field) could grant; a plain `skill --fidelity hostloop --folder X`
|
|
136
|
+
was refused with no way to consent at all. Folder-less runs (skill dir + uploads) still need no consent —
|
|
137
|
+
uploads and skill/plugin mounts are read-only.
|
|
138
|
+
- **`critique`'s spaced flag parser no longer silently grabs the next flag as a value.** `critique
|
|
139
|
+
<folder> --prompt --output-format json` (a forgotten `--prompt` value) swallowed `--output-format` as the
|
|
140
|
+
prompt AND dropped the real flag, then ran a four-workload critique on the wrong input. The spaced form
|
|
141
|
+
now fails loud and points at the equals escape hatch (`--prompt=<value>`) for a value that intentionally
|
|
142
|
+
starts with `-`.
|
|
143
|
+
- **`critique`'s subprocess byte cap is now one combined stdout+stderr budget.** `boundedSpawn` kept two
|
|
144
|
+
independent per-stream counters, so a looping or hostile child splitting output across both streams could
|
|
145
|
+
buffer ~2× the documented cap before either tripped. It now charges both streams against a single budget
|
|
146
|
+
and slices the terminal chunk to the remaining room, so captured output never exceeds the cap.
|
|
147
|
+
- **`critique`'s verbatim citation-grounding is now case-sensitive.** The mechanical check that drops any
|
|
148
|
+
finding whose cited evidence is not a verbatim excerpt normalized whitespace AND folded case — so a
|
|
149
|
+
case-altered (paraphrased) citation resolved as "grounded," weakening the principal defense against
|
|
150
|
+
evaluator hallucination. It now matches case-exact; whitespace reflow stays tolerated (models reflow
|
|
151
|
+
spacing when quoting — they don't change case).
|
|
152
|
+
- **critique's "Attached inputs" evidence no longer reports `(none)` when the uploads dir is unreadable.**
|
|
153
|
+
`listAttachedInputs` caught every `readdirSync` failure identically, so an unreadable uploads dir
|
|
154
|
+
(`EACCES`/`ENOTDIR`/…) collapsed to the same `(none)` as a legitimately-absent one — telling the
|
|
155
|
+
evaluator "the agent correctly saw no file" when attachment presence was actually UNKNOWN. It now
|
|
156
|
+
distinguishes `ENOENT` (absent) from a genuine read fault and surfaces the uncertainty loudly.
|
|
157
|
+
- **critique flags a partly-corrupt archived turn-1 transcript instead of grading it as clean.**
|
|
158
|
+
`readTurn1Transcript` skipped malformed JSONL rows (resilient) but returned `degraded: false` on the
|
|
159
|
+
first valid transcript record regardless — so archive corruption vanished from evidence health and the
|
|
160
|
+
evaluator graded a silently-incomplete transcript as solid ground truth. It now scans the whole
|
|
161
|
+
(turn-1-only) archive and sets `turn1SliceDegraded` when a malformed row was skipped or the archive
|
|
162
|
+
doesn't hold exactly one transcript record; the transcript is still delivered, just flagged.
|
|
163
|
+
- **critique flags a crashed task turn instead of grading evidence from it.** `taskTurnInfraFailure` only
|
|
164
|
+
treated a *killed* (timed-out / byte-capped) task turn as an instrument failure — a task that exited
|
|
165
|
+
nonzero without ever printing a parseable result envelope (a crash after the early `[status]` line)
|
|
166
|
+
slipped through, and its `[status]`-recovered run dir was reflected on and graded as a legitimate task.
|
|
167
|
+
It now also flags a nonzero exit with no parseable envelope. Deliberately narrow: a completed run that
|
|
168
|
+
reported a failing verdict (`ok:false` / `result:"error"` with a valid envelope) stays gradeable.
|
|
169
|
+
- **A nested unreadable output subtree no longer persists a partial file list as complete.** The
|
|
170
|
+
`workspaceFiles` / `artifacts` list was recorded as UNAVAILABLE (`undefined`, the evidence-unavailable
|
|
171
|
+
convention) only when the workspace ROOT was unobservable; a nested `EACCES`/`EIO` subtree left the walk
|
|
172
|
+
partial (`walkComplete: false`) but still persisted the incompletely-enumerated list as if complete. An
|
|
173
|
+
authored file inside the unreadable subtree then vanished with no signal, so an absence-sensitive
|
|
174
|
+
consumer (`delivered_clean` / `ended_with_question`, the replay `diff`, `scaffold`, `file_exists`) could
|
|
175
|
+
read it as absent — a silent false-clean. All three RunResult producers (run, partial-salvage, chat) now
|
|
176
|
+
route through one shared `trustedWorkspaceFiles` gate that collapses a missing root OR any incomplete
|
|
177
|
+
walk to `undefined`, and the run lanes emit a `::warning::` naming the unreadable subtree.
|
|
178
|
+
|
|
9
179
|
## [1.7.0] — 2026-07-22
|
|
10
180
|
|
|
11
181
|
### Added
|
package/CONTRIBUTING.md
CHANGED
|
@@ -12,7 +12,7 @@ npm run format:check # prettier over src/ + test/ .ts — the most common first
|
|
|
12
12
|
npm run typecheck # tsc --noEmit
|
|
13
13
|
npm test # vitest
|
|
14
14
|
npm run build # -> dist/
|
|
15
|
-
npm run ci # typecheck + build + test — the core TS gate (CI's `
|
|
15
|
+
npm run ci # typecheck + build + test — the core TS gate (CI's `build` + `test` jobs run more; see below)
|
|
16
16
|
```
|
|
17
17
|
|
|
18
18
|
Before pushing: `npm run format:check` and `npm run ci` (see RELEASING.md's release checklist for the full pre-tag list).
|
|
@@ -25,10 +25,11 @@ Extra prerequisites for specific stages:
|
|
|
25
25
|
- **Docker (arm64)** — required for `boundary-check` and the **L1 `container`** + `hostloop` fidelity tiers (the container sandbox + agent image).
|
|
26
26
|
- **Lima (`limactl`, macOS arm64)** — required only for the **L2 `microvm`** tier and the `vm` commands; the guest runs on Apple Virtualization.framework (`vmType: vz`). `microvm` does **not** use Docker. (`cowork-harness doctor --tier microvm` checks for Lima, not Docker.)
|
|
27
27
|
|
|
28
|
-
CI Stage 1 (the `
|
|
29
|
-
gate steps individually — e.g. `check:versions`, `format:check`, `typecheck`, `
|
|
30
|
-
|
|
31
|
-
source-guard checks
|
|
28
|
+
CI Stage 1 (the `build` job in `.github/workflows/ci.yml`) does **not** invoke `npm run ci`. It runs the
|
|
29
|
+
gate steps individually — e.g. `check:versions`, `format:check`, `typecheck`, `build`, a CLI smoke
|
|
30
|
+
(`node dist/cli.js list`), three token-free `replay` gate fixtures, `verify-cassettes`, `lint`, and
|
|
31
|
+
source-guard checks — while the unit suite runs separately as the 4-shard `test` job (see `ci.yml` for
|
|
32
|
+
the authoritative list). Only `release.yml` calls `npm run ci`.
|
|
32
33
|
|
|
33
34
|
> **Cutting a release?** See [RELEASING.md](./RELEASING.md) for the branch → PR → tag → publish flow.
|
|
34
35
|
|
|
@@ -85,7 +86,7 @@ When a Desktop release moves something `sync` doesn't read, it reports an `unkno
|
|
|
85
86
|
## Commit & PR
|
|
86
87
|
|
|
87
88
|
- Conventional, imperative commit subjects (`add …`, `fix …`, `parity: sync to <ver>`).
|
|
88
|
-
- Open PRs against `main`. CI runs a
|
|
89
|
+
- Open PRs against `main`. CI runs a seven-stage pipeline (`build`, `test`, `action-self-test`, `python`, `boundary`, `scenarios`, `parity-drift` — see `ci.yml`) on every PR including forks (no secrets needed) except `scenarios`; live scenarios only run on same-repo PRs/pushes with `ANTHROPIC_API_KEY` set.
|
|
89
90
|
- Describe *what changed and why*; link issues.
|
|
90
91
|
|
|
91
92
|
## Reporting issues
|
package/DESIGN.md
CHANGED
|
@@ -38,7 +38,7 @@ flowchart TB
|
|
|
38
38
|
A Cowork session is the Desktop app driving an agent that runs **inside an Apple Virtualization.framework microVM**:
|
|
39
39
|
|
|
40
40
|
- VM bundle: `~/Library/Application Support/Claude/vm_bundles/claudevm.bundle/` (`rootfs.img`, `sessiondata.img`, `efivars.fd`, `machineIdentifier`, `gvisorMacAddress`, `vmIP`); a warm pool at `vm_bundles/warm/<sha>/`.
|
|
41
|
-
- In-VM agent: `~/Library/Application Support/Claude/claude-code-vm/<ver>/claude` (currently **2.1.
|
|
41
|
+
- In-VM agent: `~/Library/Application Support/Claude/claude-code-vm/<ver>/claude` (currently **2.1.217**, per `baselines/desktop-1.24012.1.json`, an **ELF aarch64** binary), spawned by the host **in cowork mode via the `CLAUDE_CODE_IS_COWORK=1` env var** — *not* a `--cowork` flag (that flag is plugin-scope and the staged agent rejects it; see the control-protocol note below). Each baseline records this ELF's `sha256` (`agentBinary.sha256`/`shaProvenance`), and the resolver integrity-checks the binary it's about to run against that hash by default (opt out `COWORK_HARNESS_VERIFY_AGENT_SHA=0`), so "the same pinned agent" is enforced, not just asserted. Old versions are re-downloadable + verifiable from the official release channel (see `docs/maintenance.md`).
|
|
42
42
|
- Network: `vm_network_mode: "gvisor"`, egress through a userspace netstack with a **compiled domain allowlist**; off-list partners rejected (`partner rejected: entry not on compiled allowlist`).
|
|
43
43
|
- Control plane: Electron renderer→main typed IPC on channels named `$eipc_message$_<per-build-UUID>_$_claude.web_$_<Class>_$_<method>`, every handler validating `event.senderFrame.url` against a trusted-origin allowlist. The session manager is `LocalAgentModeSessions` (80 methods: `start`, `sendMessage`, `setDraftSessionFolders`, `onToolPermissionRequest`, `respondToToolPermission`, `getTranscript`, `onEvent`, …), bridged to the renderer as `window.cowork`.
|
|
44
44
|
|
package/README.md
CHANGED
|
@@ -91,7 +91,7 @@ node dist/cli.js replay examples/replays/example-pdf-skill.cassette.json
|
|
|
91
91
|
|
|
92
92
|
> **Installed globally instead?** Once linked/installed, the same command is `cowork-harness replay
|
|
93
93
|
> <cassette>` — but the relative path above only resolves from a source checkout's `examples/replays/`.
|
|
94
|
-
> From a global install (`npm i -g "cowork-harness@>=1.
|
|
94
|
+
> From a global install (`npm i -g "cowork-harness@>=1.8.0"`), point at the package root instead:
|
|
95
95
|
> `cowork-harness replay "$(npm root -g)/cowork-harness/examples/replays/example-pdf-skill.cassette.json"`
|
|
96
96
|
> (or copy the cassette into your own project and pass that path).
|
|
97
97
|
|
|
@@ -101,7 +101,7 @@ Live `run`/`skill` need the prerequisites in the next section — note the `prot
|
|
|
101
101
|
> - **Replay only (zero setup):** `cowork-harness replay <cassette>` — no token, no Docker, no agent. The command above.
|
|
102
102
|
> - **`protocol` (real model, no Docker):** needs only the auth token (item 3 below).
|
|
103
103
|
> - **Live `container` / `microvm` / `hostloop` / `cowork`:** needs Docker (or Lima for `microvm`), a staged agent, and the token — run `cowork-harness doctor` first.
|
|
104
|
-
> - **Invocation:** from a source checkout, `node dist/cli.js <cmd>` (or `npm link` to get the `cowork-harness` command); from a global install, `cowork-harness <cmd>`; the companion skill falls back to `npx "cowork-harness@>=1.
|
|
104
|
+
> - **Invocation:** from a source checkout, `node dist/cli.js <cmd>` (or `npm link` to get the `cowork-harness` command); from a global install, `cowork-harness <cmd>`; the companion skill falls back to `npx "cowork-harness@>=1.8.0"`.
|
|
105
105
|
|
|
106
106
|
Two more worked examples worth knowing about: `examples/scenarios/protocol-smoke.yaml` (zero-Docker smoke
|
|
107
107
|
test) and `examples/scenarios/skill-loads.yaml` (container-tier acceptance check) — see
|
|
@@ -126,7 +126,7 @@ claude plugin marketplace add yaniv-golan/cowork-harness
|
|
|
126
126
|
claude plugin install cowork-harness@cowork-harness
|
|
127
127
|
```
|
|
128
128
|
|
|
129
|
-
The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx "cowork-harness@>=1.
|
|
129
|
+
The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx "cowork-harness@>=1.8.0"` (a version floor that fails loud rather than silently fetching a too-old CLI; Node ≥ 20). Tiers above `protocol` still need Docker/Lima and a Claude Desktop agent binary — see the prerequisites below.
|
|
130
130
|
|
|
131
131
|
It also follows the open [Agent Skills](https://agentskills.io) spec, so it installs cross-editor (Cursor, Codex, OpenCode, …) via [`npx skills`](https://github.com/vercel-labs/skills) (Vercel Labs' CLI implementation of that spec):
|
|
132
132
|
|
|
@@ -147,7 +147,7 @@ A global install is enough for CI `lint`, reading the teaching skill, and replay
|
|
|
147
147
|
To `run` the worked examples live or copy them as a starting point, use a source checkout. (The marketplace
|
|
148
148
|
skill install itself only pulls `.claude/skills/cowork-harness/` — SKILL.md + `references/` + `scenario.py`/
|
|
149
149
|
assertion keys, per `.claude-plugin/marketplace.json`'s `source` — not the rest of this table; the full set
|
|
150
|
-
above becomes available once the skill's first command self-bootstraps `npx "cowork-harness@>=1.
|
|
150
|
+
above becomes available once the skill's first command self-bootstraps `npx "cowork-harness@>=1.8.0"` — see
|
|
151
151
|
[above](#drive-it-from-claude-code-companion-skill) — which pulls the same npm package as the global-install row.)
|
|
152
152
|
|
|
153
153
|
### Prerequisites for anything above `protocol` fidelity
|
|
@@ -190,7 +190,11 @@ So **Linux live == `container` only**: `microvm` is Apple-VZ (macOS), and `hostl
|
|
|
190
190
|
|
|
191
191
|
> The `pytest` lane (`python/README.md`) has its own, slightly different prerequisite list (adds `pytest`
|
|
192
192
|
> + package importability) — see there if you're driving tests via `pytest` instead of the CLI directly.
|
|
193
|
-
3. **An auth token** — either `export CLAUDE_CODE_OAUTH_TOKEN=$(claude setup-token)` (the `claude setup-token` step needs the **`claude` CLI**: `npm i -g @anthropic-ai/claude-code`) or a **`.env`** file (copy `.env.example` → `.env`; gitignored).
|
|
193
|
+
3. **An auth token** — either `export CLAUDE_CODE_OAUTH_TOKEN=$(claude setup-token)` (the `claude setup-token` step needs the **`claude` CLI**: `npm i -g @anthropic-ai/claude-code`) or a **`.env`** file (copy `.env.example` → `.env`; gitignored). `.env.example` lists all three accepted vars — `CLAUDE_CODE_OAUTH_TOKEN`, then `ANTHROPIC_API_KEY`, then `ANTHROPIC_AUTH_TOKEN` — in that precedence order.
|
|
194
|
+
- **Resolution order:** exported env > `--dotenv <path>` > `./.env` (cwd) > `<install>/.env` (the package root) — so a `npm link`ed install works from any directory. (Use `--dotenv`, not `--env-file` — Node reserves the latter.)
|
|
195
|
+
- **Explicit beats best-effort:** that chain is about *precedence*, not error recovery. A `--dotenv` path you passed explicitly must be readable — if it is missing, unreadable, or a directory, the run stops with a usage error rather than quietly falling through to `./.env` and running against different credentials. The automatic `./.env` / `<install>/.env` locations remain best-effort.
|
|
196
|
+
- **Placement:** keep `.env` at a working-dir or install root, never inside a mounted skill/project folder.
|
|
197
|
+
- **Global install:** find the package root with `` `$(npm root -g)/cowork-harness` `` (e.g. `$(npm root -g)/cowork-harness/.env`) — or simpler, just use `--dotenv <path>` / `./.env` in your working directory, which take priority over the package root anyway.
|
|
194
198
|
|
|
195
199
|
> `sync` (below) is **optional for a first run** — the repo ships `baselines/desktop-*.json`, so `baseline: latest` already resolves. Run `sync` only to refresh the platform baseline after Claude Desktop updates. (`sync` is **macOS-only** today; on Linux/Windows use the committed baselines — they work cross-platform.)
|
|
196
200
|
|
|
@@ -398,7 +402,7 @@ Skill testing is the headline use, but the tool is a general harness over the Co
|
|
|
398
402
|
| `record` / `replay` | **Record a live run once → replay it token-free, Docker-free thereafter** (key flags below; `replay --explain` prints the evidence behind every passing assert) | **token-free, Docker-free CI** from a once-recorded run |
|
|
399
403
|
| `verify-cassettes <file\|dir>` | Token-free CI gate over committed cassettes: a privacy scan (email/currency/domain/path/machine-inventory) + a staleness check (allowlist and skip flags below); a dir argument scans `*.cassette.json` non-recursively | gating **committed cassettes** against PII leaks + "edited the skill, forgot to re-record" |
|
|
400
404
|
| `verify-run <run-dir> <scenario.yaml>` | Re-evaluate a scenario's `assert:` (and, when the scenario declares `answers:`, whether they still match the run's actual gates) against an already-kept run dir — **no live agent, no tokens, no Docker** (~1s) | iterating on a wrong assertion or a drifted `answer` without a full live re-record |
|
|
401
|
-
| `trace <run-id>` | Digest a run's `events.jsonl` (`--view tools\|questions\|dispatches\|tool-durations\|tool-errors\|files\|usage`; `--translate-paths` rewrites VM paths to host paths in the text `tools`/default views; `--full-results` captures the full input + result of every tool call — successful ones too, not just errors — so an external grader can ground a self-critique finding against the call it cites) | "how many sub-agents *actually* dispatched, and which?" — per-tool call/timing stats with `--view tool-durations`, full stderr per failed call with `--view tool-errors`, a workspace-file diff with `--view files`,
|
|
405
|
+
| `trace <run-id>` | Digest a run's `events.jsonl` (`--view tools\|questions\|dispatches\|tool-durations\|tool-errors\|files\|usage\|subagent-research`; `--translate-paths` rewrites VM paths to host paths in the text `tools`/default views; `--full-results` captures the full input + result of every tool call — successful ones too, not just errors — so an external grader can ground a self-critique finding against the call it cites) | "how many sub-agents *actually* dispatched, and which?" — per-tool call/timing stats with `--view tool-durations`, full stderr per failed call with `--view tool-errors`, a workspace-file diff with `--view files`, per-model token/cost with `--view usage`, or each dispatch's own WebSearch query+result with `--view subagent-research` (live/record lane) |
|
|
402
406
|
| `inspect <run-id>` | Show what a run **produced**: the artifacts + a shallow field preview of each JSON artifact (`--output-format json` for a digest). Works on a salvaged partial run too | "did it do the job?" — without hand-parsing `…/mnt/outputs/…` |
|
|
403
407
|
| `scaffold <run-id>` | Turn a kept run into a starter scenario YAML (gates→answers, artifacts→`file_exists`) | authoring a scenario from a real run instead of guessing |
|
|
404
408
|
| `python3 …/scenario.py scaffold --name <name> --skill <dir>` | Generate a starter scenario skeleton from scratch (the `…` is `.claude/skills/cowork-harness/scripts/`) | starting a new scenario when you have no prior run |
|
|
@@ -415,8 +419,8 @@ Skill testing is the headline use, but the tool is a general harness over the Co
|
|
|
415
419
|
| `boundary-check [baseline] [--session <file>]` | Prove the **L1 Docker** sandbox enforces Cowork's limitations (sealed FS + default-deny egress; `container`/`hostloop` share this sandbox — `microvm`'s guest firewall is not probed here); `--session` folds a session's `egress.extra_allow` into the probe allowlist | verifying the harness's own fidelity |
|
|
416
420
|
| `sync` / `list` | Derive/refresh (`sync [--diff] [--allow-empty\|--force]`) & list platform baselines from the Desktop install | after Claude Desktop updates (baselines ship, so it's optional otherwise) |
|
|
417
421
|
| `diff <a> <b>` | Compare two baselines, two runs, two cassettes, or a run+cassette — kind auto-detected by content (view/normalization flags below). Token-free, no live Desktop/Docker needed | "what changed between two runs/cassettes/baselines?" |
|
|
418
|
-
| `critique <skill-folder>` | **EXPERIMENTAL.** Run a skill, ask the agent what confused it, then grade that self-report against a frozen record of the run — blinded evaluator, mechanical citation checking. Accepts the `skill` flags a graded run needs — `--upload`/`--folder`/`--plugin` reach both spawned turns, `--answer`/`--decider-*` the graded turn only; anything that can't work is refused with a reason (full table in `critique --help`). Discovery instrument, never a gate (findings always exit 0).
|
|
419
|
-
| `doctor [--tier <t>]` | Read-only prerequisite check, per tier (Docker + agent image for `container`/`hostloop`/`cowork`; **Lima** for `microvm`; plus staged agent, token, baseline); prints the exact `docker build` line if the agent image is missing | "can I run the live tiers — what's missing?" before a first live run |
|
|
422
|
+
| `critique <skill-folder>` | **EXPERIMENTAL.** Run a skill, ask the agent what confused it, then grade that self-report against a frozen record of the run — blinded evaluator, mechanical citation checking. Accepts the `skill` flags a graded run needs — `--upload`/`--folder`/`--plugin` reach both spawned turns, `--answer`/`--decider-*` the graded turn only; anything that can't work is refused with a reason (full table in `critique --help`). Runs at `--fidelity container` (default) or `hostloop` (a writable `--folder` there additionally needs `--allow-host-writes`); `microvm`/`protocol`/`cowork` are refused, each with a stated reason. A multi-skill plugin needs `--skill <name>` (refused pre-spend otherwise). Persists `critique-report.json` + the armored evidence package to the run dir (`--out` copies the report; a salvage file lands on exit 2), reports per-critique cost across all four workloads, and stamps each finding with a cross-input `findingFingerprint`. Discovery instrument, never a gate (findings always exit 0). Advisory — a discovery lead, not an independent attestation (the skill under review can steer the grade). See [docs/critique.md](./docs/critique.md) | "what confused the agent about my skill?" — including "does it even see the attached document?" |
|
|
423
|
+
| `doctor [--tier <t>]` | Read-only prerequisite check, per tier (Docker + agent image for `container`/`hostloop`/`cowork`; **Lima** for `microvm`; plus staged agent, token, baseline); prints the exact `docker build` line if the agent image is missing, and — for a **pulled** agent image — warns (advisory, best-effort) when its digest doesn't match the current published GHCR `:2`, so you can re-pull | "can I run the live tiers — what's missing?" before a first live run |
|
|
420
424
|
| `migrate-run-dir [<runs-dir>] [--scenario <name>] [--write]` | Convert pre-layout run dirs (artifacts at the run-dir root) to the per-turn `turns/<N>/` layout, in place — **dry-run by default**, preserving the mtimes `stats`/`--latest-for` rank by, and refusing any dir it cannot resolve rather than guessing | upgrading a runs root written by an older version, so `verify-run`/`diff`/`inspect` can read it again; `--scenario` scopes it so you can migrate one, verify, then do the rest |
|
|
421
425
|
| `prune [<runs-dir>] [--keep-last <n>] [--pinned-older-than <N>d\|h\|m]` | Prune accumulated run dirs (keeps the N most recent per scenario; pinned `--session-id` runs are never pruned unless `--pinned-older-than` opts in to reclaiming stale ones by last-activity age); the optional positional overrides the runs root; `--dry-run` | the machine-global runs root has grown and you want space back |
|
|
422
426
|
| `rehash <dir/>` | Migrate cassette fingerprints to the current format version when the content is provably unchanged (`--dry-run`); no re-record needed | a cassette-format bump flagged committed fixtures as stale |
|
|
@@ -686,13 +690,14 @@ jobs:
|
|
|
686
690
|
anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
|
|
687
691
|
```
|
|
688
692
|
|
|
689
|
-
Every run writes a Markdown verdict table (scenario, pass/fail, signals, cost/turns when available, staleness findings, and the replay-skipped-assertions honesty line) to the job summary. Inputs: `command`, `path` (required), `version` (npm dist-tag/version, default `latest` — intentional so recipes track the current release; pin an exact version for reproducible CI. The companion skill's `cowork-harness@>=1.
|
|
693
|
+
Every run writes a Markdown verdict table (scenario, pass/fail, signals, cost/turns when available, staleness findings, and the replay-skipped-assertions honesty line) to the job summary. Inputs: `command`, `path` (required), `version` (npm dist-tag/version, default `latest` — intentional so recipes track the current release; pin an exact version for reproducible CI. The companion skill's `cowork-harness@>=1.8.0` floor guidance applies to ad-hoc CLI installs, not this input), `strict` (applies to `replay` (staleness findings), `lint`/`lint-skill` (WARN/INFO), and `analyze-skill` (any advisory finding); IGNORED — not forwarded — for `verify-cassettes`/`run`, which don't accept the flag), `fail-on-skill-drift` (**`replay`-only** — never forwarded to the analyzers), `extra-args`, `summary` (default `true`), `anthropic-api-key` (live lane only). See [`action.yml`](./action.yml) for the full input reference.
|
|
690
694
|
|
|
691
|
-
The provided [GitHub Actions workflow](.github/workflows/ci.yml) runs a **
|
|
695
|
+
The provided [GitHub Actions workflow](.github/workflows/ci.yml) runs a **seven-stage pipeline**. The **build** + **test** stages are the token-free gate you can copy into your skill repo; the `action-self-test`, `python`, `boundary`, `scenarios`, and `parity-drift` stages are this repo's own fidelity self-tests and are not directly portable (they build the harness's Docker image and run harness-specific e2e scenarios — see [`ci-recipe.md`](./.claude/skills/cowork-harness/references/ci-recipe.md) for the skill-repo template):
|
|
692
696
|
|
|
693
697
|
| Stage | Runs | Needs | Gates |
|
|
694
698
|
|---|---|---|---|
|
|
695
|
-
| **
|
|
699
|
+
| **build** | format check · version-lockstep guard · typecheck · source guards · build · CLI smoke · token-free `replay` · `verify-cassettes` · `lint` | nothing | every push/PR |
|
|
700
|
+
| **test** | the unit suite (vitest), sharded 4-way | nothing | every push/PR |
|
|
696
701
|
| **action-self-test** | packs this commit and runs the packaged `uses: ./` Action against the committed example-pdf-skill cassette (pass case) + a nonexistent path (fail case) | nothing | every push/PR |
|
|
697
702
|
| **python** | `pytest` helper self-checks (`python/`, run with `-m 'not cowork'` — the token-free subset; the Docker/token `@pytest.mark.cowork` tests are excluded) | nothing (token-free assertions only) | every push/PR |
|
|
698
703
|
| **boundary** | builds the pinned agent image, brings up the default-deny network, runs `boundary-check`, then `npm run test:live` (live contract tests that guard the binary-resolution assumptions, no token needed) | Docker, arm64 runner | proves the sandbox enforces Cowork's limits — **no API key** |
|
|
@@ -815,6 +820,7 @@ a global install has them locally too, not just on GitHub.
|
|
|
815
820
|
| [docs/chat.md](./docs/chat.md) | The interactive `chat` REPL — multi-turn debugging, flags, attaching files/folders. |
|
|
816
821
|
| [docs/debugging.md](./docs/debugging.md) | Debugging a run — `inspect`/`trace`/`verify-run`/`diff`/`chat` for a misbehaving skill; the false-green hunt for a green you don't trust; and the **iterate-across-fixes verification loop** (ground findings in run evidence; pair by `fingerprint.skillHash`). |
|
|
817
822
|
| [docs/cassette.md](./docs/cassette.md) | `record`/`replay` cassettes — what replay checks, which assertions are skipped. |
|
|
823
|
+
| [docs/critique.md](./docs/critique.md) | **EXPERIMENTAL** — `critique`: run a skill, ask the agent what confused it, then grade that self-report against a frozen record. Its verdict is an advisory lead, not an attestation. |
|
|
818
824
|
| [docs/run-status.md](./docs/run-status.md) | Checking whether a background run is alive — the `status.json` file + `cowork-harness status [--follow]`. |
|
|
819
825
|
| [docs/stats.md](./docs/stats.md) | The `stats` command + `index.jsonl` — querying pass rate, cost/duration/token/turn percentiles, and last-green across every past run. |
|
|
820
826
|
| [docs/gotchas.md](./docs/gotchas.md) | Troubleshooting FAQ — exit 127, empty skill mount, arm64 Docker issues, git-worktree token traps, scenarioDrift after an edit, plus skill-authoring/host-loop footguns. |
|
package/RELEASING.md
CHANGED
|
@@ -104,10 +104,12 @@ tagging `1.0.0`, deliberately review and freeze the surfaces with no machine-rea
|
|
|
104
104
|
`npx "cowork-harness@>=X.Y.Z"`).
|
|
105
105
|
5. `.claude/skills/cowork-harness/references/scenario-schema.md` → the
|
|
106
106
|
"Tracks `cowork-harness X.Y.Z`" line.
|
|
107
|
-
6. `.claude/skills/cowork-harness/references/fidelity-and-answers.md`
|
|
108
|
-
|
|
107
|
+
6. `.claude/skills/cowork-harness/references/fidelity-and-answers.md` and
|
|
108
|
+
`.claude/skills/cowork-harness/references/task-recipes.md` → the
|
|
109
|
+
"Tracks `cowork-harness X.Y.Z`" line in each.
|
|
109
110
|
7. The baseline these track (`tracks-harness … (baseline desktop-<ver>)`) — keep in sync with the
|
|
110
|
-
newest `baselines/desktop-*.json`.
|
|
111
|
+
newest `baselines/desktop-*.json`. The `check:versions` guard enforces this for SKILL.md, every
|
|
112
|
+
`references/*.md` baseline pin, and DESIGN.md's current-state sentence — a lagging pin reds CI.
|
|
111
113
|
8. `.claude/skills/cowork-harness/references/ci-recipe.md` → all `npm i -g "cowork-harness@>=X.Y.Z"` floors
|
|
112
114
|
(currently 3 occurrences).
|
|
113
115
|
9. `examples/replays/README.md` → the `npm i -g "cowork-harness@>=X.Y.Z"` floor.
|
|
@@ -185,5 +187,5 @@ tagging `1.0.0`, deliberately review and freeze the surfaces with no machine-rea
|
|
|
185
187
|
(`git tag -d vX.Y.Z`), re-create it on the correct commit, and push it.
|
|
186
188
|
- The live `scenario suite` CI stage runs on **same-repo** PRs (where the `ANTHROPIC_API_KEY` secret is
|
|
187
189
|
available) and on pushes to `main`; it is skipped only on **fork** PRs (or when the secret is unset). So
|
|
188
|
-
a same-repo release-branch PR DOES exercise the live suite and spend API budget — the `
|
|
189
|
-
stages alone are sufficient to gate a release if you'd rather not.
|
|
190
|
+
a same-repo release-branch PR DOES exercise the live suite and spend API budget — the `build` + `test` +
|
|
191
|
+
`boundary` stages alone are sufficient to gate a release if you'd rather not.
|