cowork-harness 0.31.0 → 0.33.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/cowork-harness/SKILL.md +23 -11
- package/.claude/skills/cowork-harness/references/ci-recipe.md +11 -8
- package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +6 -6
- package/.claude/skills/cowork-harness/references/scenario-schema.md +6 -6
- package/.claude/skills/cowork-harness/references/task-recipes.md +1 -1
- package/.claude/skills/cowork-harness/scripts/scenario.py +32 -19
- package/CHANGELOG.md +225 -0
- package/CONTRIBUTING.md +2 -0
- package/DESIGN.md +2 -2
- package/README.md +62 -41
- package/SPEC.md +6 -4
- package/dist/agent/session.js +11 -2
- package/dist/cli.js +21 -10
- package/dist/run/analyze-skill.js +520 -46
- package/dist/run/cassette.js +4 -2
- package/dist/run/doctor.js +2 -2
- package/dist/run/run.js +3 -3
- package/dist/run/scenario-tool.js +90 -25
- package/dist/run/subagent-reasoning.js +38 -1
- package/dist/runtime/argv.js +3 -0
- package/dist/session.js +9 -0
- package/dist/types.js +2 -2
- package/docs/README.md +5 -5
- package/docs/cassette.md +1 -1
- package/docs/chat.md +4 -0
- package/docs/debugging.md +11 -9
- package/docs/gotchas.md +5 -1
- package/docs/maintenance.md +1 -1
- package/docs/plugin-root.md +4 -0
- package/docs/protocol.md +5 -0
- package/docs/run-status.md +24 -0
- package/docs/scenario.md +10 -4
- package/docs/session.md +11 -4
- package/docs/subagents.md +167 -16
- package/examples/README.md +21 -1
- package/examples/replays/README.md +1 -1
- package/llms.txt +7 -2
- package/package.json +1 -1
- package/python/README.md +4 -1
- package/schema/run-result.json +10 -1
- package/schema/scenario.schema.json +2 -2
- package/schema/session.schema.json +7 -0
- package/scripts/check-versions.ts +35 -16
|
@@ -3,8 +3,8 @@ name: cowork-harness
|
|
|
3
3
|
description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
|
|
4
4
|
metadata:
|
|
5
5
|
author: cowork-harness
|
|
6
|
-
version: 0.
|
|
7
|
-
tracks-harness: cowork-harness 0.
|
|
6
|
+
version: 0.33.0
|
|
7
|
+
tracks-harness: cowork-harness 0.33.0 (baseline desktop-1.20186.1)
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
# cowork-harness
|
|
@@ -22,7 +22,7 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
|
|
|
22
22
|
allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
|
|
23
23
|
the highest-value part. Read it.
|
|
24
24
|
|
|
25
|
-
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.
|
|
25
|
+
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.33.0` (baseline
|
|
26
26
|
> `desktop-1.20186.1`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
|
|
27
27
|
> `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
|
|
28
28
|
|
|
@@ -38,10 +38,22 @@ cowork-harness skill ./my-skill "do X" # run the skill once against the sta
|
|
|
38
38
|
Before the first command, confirm the CLI is reachable and **fail loud** (never fake a pass) when a tier's dependencies are missing:
|
|
39
39
|
|
|
40
40
|
- **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
|
|
41
|
-
- **
|
|
41
|
+
- **Replay-only? Skip `doctor`.** Replaying committed cassettes needs no Docker, no staged agent, and no token — and every tier's `doctor` validates the auth token (the live tiers also Docker + the staged agent), so a ✗ there is expected, not a blocker. Go straight to `cowork-harness replay <cassette>`.
|
|
42
|
+
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.33.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=0.33.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=0.33.0"`. **Pin `@>=0.33.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
|
|
43
|
+
|
|
44
|
+
What the ≥ 0.33.0 floor gates, by release:
|
|
45
|
+
|
|
46
|
+
- **core set (pre-0.21.0 vintage, or mixed):** `assertions --list`, `scaffold <run-id>`, `trace --view dispatches`, `artifact_json` incl. the `in:` operator (passes when the resolved value deep-equals one of the listed members — value ∈ your list, not the reverse), `verify-cassettes` incl. the `--allow-domain`/`--allow-email`/`--allow-patterns-file` allows (`--allow-patterns-file <path>` is a FILE of patterns, one regex per line — not a path to allow, unlike `--allow <regex>`), batch `record <dir>`/`--rerecord-stale`, `record --concurrency <N>`, record-time redaction, multiSelect/`answer:`, `verify-run` answer-coverage, `record --max-artifact-bytes`, live record-time deciders, scenario `skills:` staleness scoping with `COWORK_HARNESS_AGENT_SCOPE=skill`, `chat --plugin`, and `/help` in the REPL.
|
|
47
|
+
- **0.21.0:** `verify-cassettes --allow-path` (`path` — local absolute filesystem paths — is the scanner's 4th class), and `hostloop`'s native host/VM process split with its `allow_host_writes:` consent field.
|
|
48
|
+
- **0.22.0:** `computer_links_resolve`.
|
|
49
|
+
- **0.28.0:** `semantic_matches` (an LLM judge grades a fixed `rubric` of claims against the run's answer — **live-only**, so it is evidence-unavailable / skipped-loud on replay, never a vacuous pass), and **glob-matched** `tool_called`/`tool_not_called`/`subagent_tool_used`/`subagent_tool_absent` (a pattern like `mcp__workspace__*` matches any tool in the family; exact names match exactly).
|
|
50
|
+
- **0.30.0:** resolved sub-agent identity on dispatch records (`resolvedAgentType`/`resolvedModel`), the five path-gate assertion keys `no_vm_path_file_op`/`vm_path_denied`/`path_denied`/`no_path_denied`/`subagent_file_write`, and the session-level `agent_env` knob.
|
|
51
|
+
- **0.31.0:** the `lint-skill` (static host-loop footgun + `subagent_type` resolution linter) / `analyze-skill` (advisory `/sessions`-path static scan, `--strict`, `analyze-skill: ignore` marker) / `probe-dispatch` (single-dispatch mechanics probe) commands, `status --latest-for` (resolve a scenario's newest run dir by run time, not directory mtime), the `subagent_dispatch_healthy` composite assertion, and the persisted `result.json` fields `verdict`, `subagents[].referencesRead`/`subagents[].reasoning`, plus the `toolCounts`/`toolErrors`/`toolDurations` shape distinction.
|
|
52
|
+
- **0.32.0:** `analyze-skill`'s directory scan now covering a skill/plugin's full contract surface (recursive `agents/`/`references/`/`commands/`, plugin-root-aware, symlink-following) with line/block-scoped `analyze-skill: ignore-next-line`/`ignore-start`/`ignore-end` markers and multi-path/glob input, and `lint-skill`'s provable in-plugin `subagent_type` typo now a WARN that gates under `--strict`.
|
|
53
|
+
- **0.33.0:** the `redacted` marker on display-omitted reasoning — `subagents[].reasoning` and the top-level `thinking[]` now carry `{text:"", redacted:true}` when the model returns a signed-but-empty thought (so "reasoned, text omitted" is distinct from "no thought"), plus the fenced `debug.thinking_display` escape hatch.
|
|
42
54
|
- **Agent binary (sandboxed live tiers — `container`/`microvm`/`hostloop`/`cowork`).** The staged Claude Code agent is **bind-mounted** from a local Claude Desktop install, or point `COWORK_AGENT_BINARY` at a `claude-code-vm/<ver>/claude` ELF. Nothing is bundled. `protocol` (L0) and `replay` need no staged agent; for the sandboxed tiers, no agent → no run; report that, don't skip silently.
|
|
43
55
|
- **Docker / Lima.** Only `--fidelity protocol` (L0) runs without them. `container` / `microvm` / `hostloop` / `cowork` need Docker (Lima for L2). If they're absent, drop to `--fidelity protocol` and **say so** — a green that never exercised the sandbox is not a sandbox pass.
|
|
44
|
-
- **Auth.** `CLAUDE_CODE_OAUTH_TOKEN` (preferred) or `ANTHROPIC_API_KEY`, via env or `.env`. Minting an OAuth token needs the **`claude` CLI** (`npm i -g @anthropic-ai/claude-code`, then `claude setup-token`).
|
|
56
|
+
- **Auth.** `CLAUDE_CODE_OAUTH_TOKEN` (preferred), or `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN`, via env or `.env`. Minting an OAuth token needs the **`claude` CLI** (`npm i -g @anthropic-ai/claude-code`, then `claude setup-token`).
|
|
45
57
|
- **`--dotenv` is a GLOBAL flag — put it BEFORE the subcommand.** `cowork-harness --dotenv .env record …`, never `cowork-harness record … --dotenv .env`. Every *other* flag is subcommand-level, so muscle memory fights this one; the harness rejects the misplaced form with an exact-fix error, but placing it first avoids the round-trip.
|
|
46
58
|
|
|
47
59
|
## Orient — the three loops
|
|
@@ -63,7 +75,7 @@ reproducible regression (Part II), and **debug** a run that misbehaved or greene
|
|
|
63
75
|
the path, `trace <run-id>` finds it). **Localize the failure post-hoc** from that evidence:
|
|
64
76
|
`cowork-harness trace <run-dir>`'s views + the emitted `result.json` to see what the run actually did,
|
|
65
77
|
then `verify-run` to re-check a suspect assertion — all token-free, no Docker, no re-record. This is
|
|
66
|
-
the loop 0.
|
|
78
|
+
the loop 0.32.0's observability is built for; the *Triage* and *Inspecting a run's observability
|
|
67
79
|
output* sections in **Part III — Debug** are the detail (the fuller human-facing map lives in
|
|
68
80
|
`docs/debugging.md` — repo-only, not shipped with the installed skill).
|
|
69
81
|
- **Multi-turn / interactive reproduction** → `cowork-harness chat` (interactive; gates answered at the
|
|
@@ -240,12 +252,12 @@ Full model in `references/scenario-schema.md`.
|
|
|
240
252
|
Don't hand-write the YAML from memory — that's how invented keys (`assertions:` vs `assert:`,
|
|
241
253
|
`json_file`, `answer_policy`) creep in. Start from the bundled generator, which emits the
|
|
242
254
|
known-good skeleton (right tier, scripted `answers:` + `on_unanswered: fail`, content assertions
|
|
243
|
-
separated from live-only ones, one concern per item) and **self-lints its own output
|
|
255
|
+
separated from live-only ones, one concern per item) and **self-lints its own output**. The
|
|
256
|
+
generator is the bundled `scripts/scenario.py` — installed as a plugin, point `S` at
|
|
257
|
+
`${CLAUDE_PLUGIN_ROOT}/scripts/scenario.py`; from a repo checkout, use the literal path below:
|
|
244
258
|
|
|
245
259
|
```bash
|
|
246
|
-
S="
|
|
247
|
-
# Working from a repo checkout instead of an installed plugin?
|
|
248
|
-
# S=".claude/skills/cowork-harness/scripts/scenario.py"
|
|
260
|
+
S=".claude/skills/cowork-harness/scripts/scenario.py"
|
|
249
261
|
python3 "$S" scaffold --name report-check --skill ./skills/report-gen \
|
|
250
262
|
--prompt "Generate the weekly report to outputs/report.md." \
|
|
251
263
|
--content 'weekly report' --artifact outputs/report.md \
|
|
@@ -372,7 +384,7 @@ than a stuck `"running"`.)
|
|
|
372
384
|
### Place assertions in the right CI lane
|
|
373
385
|
|
|
374
386
|
CI placement: a **token-free `replay` PR gate** (content/structure only) + a **nightly live `run`**
|
|
375
|
-
(filesystem/egress). Fastest setup: `uses: yaniv-golan/cowork-harness@
|
|
387
|
+
(filesystem/egress). Fastest setup: `uses: yaniv-golan/cowork-harness@main` (a packaged GitHub Action with a
|
|
376
388
|
PR job-summary reporter). See `references/ci-recipe.md` for the Action, the manual step-by-step form, and
|
|
377
389
|
the four-stage pipeline.
|
|
378
390
|
|
|
@@ -1,17 +1,20 @@
|
|
|
1
1
|
# CI recipe — replay vs live lanes
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 0.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 0.33.0` (baseline `desktop-1.20186.1`).
|
|
4
4
|
|
|
5
5
|
**Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
|
|
6
6
|
job-summary reporter (verdict table, staleness findings, cost/turns when available):
|
|
7
7
|
|
|
8
8
|
```yaml
|
|
9
|
-
- uses: yaniv-golan/cowork-harness@
|
|
9
|
+
- uses: yaniv-golan/cowork-harness@main
|
|
10
10
|
with:
|
|
11
11
|
command: replay
|
|
12
12
|
path: cassettes/
|
|
13
13
|
```
|
|
14
14
|
|
|
15
|
+
The Action's `version` input defaults to `latest` — intentional so a copy-pasted recipe tracks the current
|
|
16
|
+
release; pin an exact version (e.g. `version: "0.33.0"`) for reproducible CI.
|
|
17
|
+
|
|
15
18
|
Reach for the manual multi-step form below only when you need per-step control the Action's inputs don't
|
|
16
19
|
cover (a custom flag combination, a different runner matrix per step, or `lint`/`verify-cassettes` gated
|
|
17
20
|
independently instead of as one action run per command).
|
|
@@ -29,13 +32,13 @@ jobs:
|
|
|
29
32
|
- uses: actions/checkout@v4
|
|
30
33
|
- name: Stage the agent binary (official channel, sha256-verified — see docs/maintenance.md)
|
|
31
34
|
run: |
|
|
32
|
-
V=2.1.
|
|
35
|
+
V=2.1.205 # match your scenario's pinned baseline's agentVersion
|
|
33
36
|
curl -fSL "https://downloads.claude.ai/claude-code-releases/$V/linux-arm64/claude" -o "$RUNNER_TEMP/claude-$V"
|
|
34
37
|
chmod +x "$RUNNER_TEMP/claude-$V"
|
|
35
38
|
# verify against the committed baseline's sha256 (baselines/desktop-*.json → agentBinary.sha256)
|
|
36
39
|
# before trusting it — see docs/maintenance.md's "Agent-binary provenance" section.
|
|
37
40
|
echo "COWORK_AGENT_BINARY=$RUNNER_TEMP/claude-$V" >> "$GITHUB_ENV"
|
|
38
|
-
- uses: yaniv-golan/cowork-harness@
|
|
41
|
+
- uses: yaniv-golan/cowork-harness@main
|
|
39
42
|
with:
|
|
40
43
|
command: run
|
|
41
44
|
path: scenarios/
|
|
@@ -54,7 +57,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
|
|
|
54
57
|
GitHub-hosted runners, no token/Docker/agent:
|
|
55
58
|
|
|
56
59
|
```yaml
|
|
57
|
-
- run: npm i -g "cowork-harness@>=0.
|
|
60
|
+
- run: npm i -g "cowork-harness@>=0.33.0"
|
|
58
61
|
- run: cowork-harness lint scenarios/*.yaml # no silent false-greens
|
|
59
62
|
- run: cowork-harness verify-cassettes cassettes/ # privacy + staleness
|
|
60
63
|
- run: cowork-harness replay cassettes/ # token-free content/structure
|
|
@@ -174,7 +177,7 @@ A typical skill repo runs four stages, fastest/cheapest first:
|
|
|
174
177
|
|
|
175
178
|
## GitHub Actions sketch
|
|
176
179
|
|
|
177
|
-
The PR gate below is the manual, step-by-step version of what `uses: yaniv-golan/cowork-harness@
|
|
180
|
+
The PR gate below is the manual, step-by-step version of what `uses: yaniv-golan/cowork-harness@main` does
|
|
178
181
|
in one step (see the top of this doc) — reach for this form when you need independent per-command
|
|
179
182
|
gating/annotations rather than one action run per command. The nightly live job has no packaged-Action
|
|
180
183
|
equivalent yet (the Action's `command: run` mode needs a self-hosted runner with Docker + the agent binary
|
|
@@ -194,7 +197,7 @@ jobs:
|
|
|
194
197
|
with: { node-version: '20' }
|
|
195
198
|
- uses: actions/setup-python@v5
|
|
196
199
|
with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
|
|
197
|
-
- run: npm i -g "cowork-harness@>=0.
|
|
200
|
+
- run: npm i -g "cowork-harness@>=0.33.0"
|
|
198
201
|
- run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
|
|
199
202
|
- run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
|
|
200
203
|
- run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
|
|
@@ -223,7 +226,7 @@ jobs:
|
|
|
223
226
|
echo "live=true" >> "$GITHUB_OUTPUT"
|
|
224
227
|
fi
|
|
225
228
|
- if: steps.guard.outputs.live == 'true'
|
|
226
|
-
run: npm i -g "cowork-harness@>=0.
|
|
229
|
+
run: npm i -g "cowork-harness@>=0.33.0"
|
|
227
230
|
- if: steps.guard.outputs.live == 'true'
|
|
228
231
|
run: cowork-harness run scenarios/ --output-format json
|
|
229
232
|
env:
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Fidelity tiers & answer paths
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 0.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 0.33.0` (baseline `desktop-1.20186.1`).
|
|
4
4
|
|
|
5
5
|
## Fidelity tiers (`fidelity:` in the scenario)
|
|
6
6
|
|
|
@@ -13,10 +13,10 @@ Self-contained reference. Tracks `cowork-harness 0.31.0` (baseline `desktop-1.20
|
|
|
13
13
|
| `cowork` | Auto-picks `hostloop` vs `container` the way Cowork itself does for the synced release | "Do what real Cowork does for this release." |
|
|
14
14
|
|
|
15
15
|
- `hostloop` / `cowork` are the production-faithful path; `container` is the practical default.
|
|
16
|
-
- Boundary assertions (`egress_*`, `expect_denied`) are enforced at `container`, `microvm`,
|
|
17
|
-
`
|
|
18
|
-
`hostloop`'s native file tools run with no container at all,
|
|
19
|
-
Only `protocol` is rejected.
|
|
16
|
+
- Boundary assertions (`egress_*`, `expect_denied`) are enforced at `container`, `microvm`, `hostloop`,
|
|
17
|
+
and `cowork` (`cowork` auto-resolves to a sandboxed tier; `container`'s and `hostloop`'s `bash` share
|
|
18
|
+
the same Docker sandbox + egress proxy — `hostloop`'s native file tools run with no container at all,
|
|
19
|
+
gated instead by a path-containment hook). Only `protocol` is rejected.
|
|
20
20
|
- A `hostloop` scenario with a **writable** connected folder (`mode: rw`/`rwd`) needs `allow_host_writes:
|
|
21
21
|
true` — with no container around the native file tools, that combination gives the agent genuine,
|
|
22
22
|
software-checked-only host filesystem access. Read-only folders and folder-less runs need no opt-in.
|
|
@@ -176,7 +176,7 @@ and `scaffold` refuse to treat a partial run's half-finished output as a passing
|
|
|
176
176
|
- `COWORK_HARNESS_RUNS_DIR` (or `--run-dir <path>`) — override the default run-output root `~/.cowork-harness/runs` (out of any working tree). flag > env > default.
|
|
177
177
|
- `COWORK_HARNESS_DECIDER_CMD_TIMEOUT_MS` / `COWORK_HARNESS_LLM_TIMEOUT_MS` — decider backstops
|
|
178
178
|
(default 600 s; **fail loud** on timeout).
|
|
179
|
-
- `COWORK_HARNESS_DECIDER_DIR_POLL_MS` / `_TIMEOUT_MS` — the `--decider-dir` rendezvous.
|
|
179
|
+
- `COWORK_HARNESS_DECIDER_DIR_POLL_MS` / `_TIMEOUT_MS` — the `--decider-dir` rendezvous (poll defaults: 300 ms for the run-side rendezvous, 500 ms for `gates --follow`).
|
|
180
180
|
- `COWORK_HARNESS_DIALOG_TIMEOUT_MS` — dialog auto-cancel (default 6 s).
|
|
181
181
|
- `COWORK_HARNESS_LLM_MAX_BYTES` — stdout bound on `--decider-llm` (default 8 MiB).
|
|
182
182
|
- `COWORK_HARNESS_LLM_RETRIES` — bounded retries for a transient non-zero `claude -p` exit on the
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Scenario & session schema, assertion catalog, web_fetch, full gotchas
|
|
2
2
|
|
|
3
|
-
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.
|
|
3
|
+
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.33.0`
|
|
4
4
|
(baseline `desktop-1.20186.1`). If your checkout is newer, prefer the live `docs/scenario.md`,
|
|
5
5
|
`docs/session.md`, and `SPEC.md`.
|
|
6
6
|
|
|
@@ -37,7 +37,7 @@ fidelity: container # protocol | container | microvm | hostloop
|
|
|
37
37
|
execution: local # OPTIONAL — orthogonal to fidelity (a privilege/sandbox tier, all
|
|
38
38
|
# local): local (default) | cloud-describe (RESERVED — no runner
|
|
39
39
|
# exists yet; authoring it is a load-time error, not a silent no-op)
|
|
40
|
-
on_unanswered: fail # policy for unscripted gates: fail | prompt | first | llm
|
|
40
|
+
on_unanswered: fail # policy for unscripted gates: fail | prompt | first | llm — run rejects prompt
|
|
41
41
|
# ("agent" is retired — no longer a valid value)
|
|
42
42
|
|
|
43
43
|
prompt: | # the user turn
|
|
@@ -312,7 +312,7 @@ same set live from the schema.
|
|
|
312
312
|
| `egress_allowed: <host>` | the host was allowed through |
|
|
313
313
|
| `no_mcp_error: true` | no MCP round-trip failed (`RunResult.mcpErrors` is empty — no unhandled server, no handler throw) — live-only: MCP round-trips are harness-computed, not in the SDK stdout stream, so evidence-unavailable on replay (never a vacuous pass). **Only `true` is valid** |
|
|
314
314
|
| `max_peak_rss_bytes: <N>` | peak sampled RSS of the agent sandbox ≤ N bytes (`RunResult.resources.peakRssBytes`) — live-only: replay never spawns a sandbox to sample, so evidence-unavailable on replay/protocol (never a vacuous pass); also evidence-unavailable when sampling captured no RSS value |
|
|
315
|
-
| `semantic_matches: {rubric: [...], min_pass?, judge_model?}` | a pinned LLM judge grades each fixed `rubric` claim against the run's answer — the **union of the agent's final result text (`RunResult.finalMessage`), the transcript, and the final on-disk content of any files the agent authored during the run** — so a claim about content the skill led the agent to *write to a file* grades as reliably as one about inlined prose (authored-file evidence is unavailable on the **microvm** tier — no pre-run manifest to diff against, so no files are captured; `container`/`hostloop` do capture them). Beyond the microvm case, when the authored-file evidence backing the judged document is **incomplete** — a file dropped at the capture-size cap, unreadable at read-back, or (on `--resume`) the scratchpad walk skipped — the assert fails evidence-unavailable rather than trusting a judge grade over a partial document; this is separate from the malformed-grade `judgeInvalid` path below. The assert passes iff ≥ `min_pass` claims pass (default: all — avoid for a gating scenario). Results align by claim index and are recorded per-claim in `RunResult.assertions[].semanticClaims` (`[{index, claim, pass}]`, so a consumer can diff the per-claim profile across runs); a rep whose grade can't be parsed (after one retry) is marked `RunResult.assertions[].judgeInvalid` and **never silently dropped** — it is excluded from the pass denominator, and the guard against a misleading score from that exclusion is the gate's minimum-valid-rep floor (`MIN_VALID` ≥ 4) plus this visibility, not a claim that denominator-shrinking inflation is impossible. Within a rep, a grade that's still unparseable after the retry **fails that assert outright** (evidence-unavailable, not a vacuous pass) — a persistently-flaky judge reds the run rather than silently passing. `judge_model` pins the grader (a dated id keeps a before/after comparison reproducible). Live-only: the judge is a live model call, so evidence-unavailable / skipped-loud on replay (never a vacuous pass) |
|
|
315
|
+
| `semantic_matches: {rubric: [...], min_pass?, judge_model?}` | a pinned LLM judge grades each fixed `rubric` claim against the run's answer — the **union of the agent's final result text (`RunResult.finalMessage`), the transcript, and the final on-disk content of any files the agent authored during the run** — so a claim about content the skill led the agent to *write to a file* grades as reliably as one about inlined prose (authored-file evidence is unavailable on the **microvm** tier — no pre-run manifest to diff against, so no files are captured; `container`/`hostloop` do capture them). Beyond the microvm case, when the authored-file evidence backing the judged document is **incomplete** — a file dropped at the capture-size cap, unreadable at read-back, or (on `--resume`) the scratchpad walk skipped — the assert fails evidence-unavailable rather than trusting a judge grade over a partial document; this is separate from the malformed-grade `judgeInvalid` path below. The assert passes iff ≥ `min_pass` claims pass (default: all — avoid for a gating scenario). Results align by claim index and are recorded per-claim in `RunResult.assertions[].semanticClaims` (`[{index, claim, pass}]`, so a consumer can diff the per-claim profile across runs); a rep whose grade can't be parsed (after one retry) is marked `RunResult.assertions[].judgeInvalid` and **never silently dropped** — it is excluded from the pass denominator, and the guard against a misleading score from that exclusion is the gate's minimum-valid-rep floor (`MIN_VALID` ≥ 4) plus this visibility, not a claim that denominator-shrinking inflation is impossible. Within a rep, a grade that's still unparseable after the retry **fails that assert outright** (evidence-unavailable, not a vacuous pass) — a persistently-flaky judge reds the run rather than silently passing. `judge_model` pins the grader (default when neither it nor `COWORK_HARNESS_JUDGE_MODEL` is set: `claude-opus-4-8`; a dated id keeps a before/after comparison reproducible). Live-only: the judge is a live model call, so evidence-unavailable / skipped-loud on replay (never a vacuous pass) |
|
|
316
316
|
| `artifact_json: {artifact, path, …}` | assert a JSON artifact's contents — `equals`/`gt`/`in`/`exists`/`absent`/`is_null` over a dotted `path` (`in` = membership in a list, for a stochastic/LLM value; `absent` ≠ `is_null`; an unresolved intermediate fails loud) |
|
|
317
317
|
| `computer_links_resolve: true` | every `computer://` link in the model-visible transcript resolves to an artifact that exists in the run's collected outputs/mounts — a dangling link fails, naming which target was checked (a live host path, the collected work tree, or the replay manifest). **Requires ≥1 link** (zero links fails — use `computer_links_resolve_if_present` for the presence-free variant). **Only `true` is valid** (`false` is rejected by the schema) |
|
|
318
318
|
| `computer_links_resolve_if_present: true` | like `computer_links_resolve` but passes vacuously when the transcript has zero `computer://` links — the presence-free variant. **Only `true` is valid** |
|
|
@@ -329,7 +329,7 @@ paraphrases (that re-records red). Structured JSON → assert it in YAML with **
|
|
|
329
329
|
`path` + operator); use the pytest lane (`assert_artifact_json`) only for predicates too complex for a
|
|
330
330
|
dotted path.
|
|
331
331
|
|
|
332
|
-
**VerdictSignals in `result.signals`:** `computeVerdict` pushes signals into `result.signals`; most
|
|
332
|
+
**VerdictSignals in `result.verdict.signals`:** `computeVerdict` pushes signals into `result.verdict.signals`; most
|
|
333
333
|
are **fail**-severity (they flip the run's pass/exit code even though `result.result` itself stays
|
|
334
334
|
`"success"`) and only three are **warn**-severity (informational, never flip pass/fail). Current signal
|
|
335
335
|
codes (`VerdictSignal["code"]` in `src/run/verdict.ts`):
|
|
@@ -353,7 +353,7 @@ codes (`VerdictSignal["code"]` in `src/run/verdict.ts`):
|
|
|
353
353
|
|
|
354
354
|
A **fail**-severity signal does not change `result.result` (still `"success"`), but it DOES fail the
|
|
355
355
|
overall run verdict and exit code — `assert result: success` alone won't catch it; check
|
|
356
|
-
`result.signals[].severity` or the run's exit code. Only the three **warn** codes are truly benign.
|
|
356
|
+
`result.verdict.signals[].severity` or the run's exit code. Only the three **warn** codes are truly benign.
|
|
357
357
|
|
|
358
358
|
## Replay class
|
|
359
359
|
|
|
@@ -491,7 +491,7 @@ reference omits). Neither list is a strict superset of the other — reach for t
|
|
|
491
491
|
the stochastic path flags the run `nonDeterministic`. The LLM decider is one mechanism, two
|
|
492
492
|
spellings: `on_unanswered: llm` (YAML) and `--decider-llm` (CLI). The bare `--on-unanswered llm`
|
|
493
493
|
is rejected (use `--decider-llm`). `agent` is **retired** — `on_unanswered: agent` is rejected by
|
|
494
|
-
the schema. (`src/types.ts
|
|
494
|
+
the schema. (`src/types.ts` — the `on_unanswered` enum; `src/cli.ts:899` — the CLI-side
|
|
495
495
|
`--on-unanswered` value check.)
|
|
496
496
|
|
|
497
497
|
4. **`--on-unanswered first` is non-deterministic too** — it picks option 1 and is flagged
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Each recipe composes facts that live scattered across SKILL.md and the other references into one
|
|
4
4
|
decision path. Every one answers a question a real fleet owner had to work out the hard way. Facts track the harness version in SKILL.md's
|
|
5
|
-
front-matter (currently 0.
|
|
5
|
+
front-matter (currently 0.32.0). Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
|
|
6
6
|
Recipe 3's `init-redact` shipped in 0.24.0 and are part of the current feature set — no version gate
|
|
7
7
|
needed if your CLI meets SKILL.md's version floor.
|
|
8
8
|
|
|
@@ -1047,9 +1047,10 @@ def _finding_subagent_not_found_in_plugin(path, line, value, plugin_name, agent,
|
|
|
1047
1047
|
own enclosing plugin) but whose agent isn't in its enumerated agents/*.md set. Unlike
|
|
1048
1048
|
`subagent-type-unknown`, this can never be another binary's built-in — the namespace prefix
|
|
1049
1049
|
already commits it to this plugin, and the plugin's agent set was fully enumerable — so it's a
|
|
1050
|
-
provable typo, not an unconfirmable unknown.
|
|
1050
|
+
provable typo, not an unconfirmable unknown. Provable means it can be a hard gate: WARN (not
|
|
1051
|
+
INFO), so `lint-skill --strict` fails on it."""
|
|
1051
1052
|
return Finding(
|
|
1052
|
-
"
|
|
1053
|
+
"WARN",
|
|
1053
1054
|
"subagent-type-not-found-in-plugin",
|
|
1054
1055
|
f"pinned type `{value}` names this plugin (`{plugin_name}`) but `{agent}` is not among its "
|
|
1055
1056
|
f"agents [{', '.join(sorted_agents)}] — likely a typo.",
|
|
@@ -1063,20 +1064,23 @@ def _finding_subagent_not_found_in_plugin(path, line, value, plugin_name, agent,
|
|
|
1063
1064
|
|
|
1064
1065
|
def _classify_subagent_type(value, plugin_name, plugin_agent_types):
|
|
1065
1066
|
"""subagent_type severity ladder. Returns a Finding-builder (path, line, value) -> Finding, or None if
|
|
1066
|
-
clean.
|
|
1067
|
+
clean. Only ONE outcome is a hard gate: `subagent-type-not-found-in-plugin` is WARN (a provable
|
|
1068
|
+
typo, so `lint-skill --strict` fails on it); the two unconfirmable outcomes
|
|
1069
|
+
(`subagent-type-unresolvable`, `subagent-type-unknown`) stay INFO — see the HONEST LIMIT note
|
|
1070
|
+
above this section for why those two can never be proven.
|
|
1067
1071
|
|
|
1068
1072
|
Precedence:
|
|
1069
1073
|
1. Resolves in-plugin, or is `general-purpose` -> clean (None).
|
|
1070
1074
|
2. Has a `<prefix>:<agent>` shape whose prefix EQUALS the resolved plugin's own name AND the
|
|
1071
|
-
plugin's agent set was non-empty (i.e. enumerable) -> `subagent-type-not-found-in-plugin
|
|
1072
|
-
A value namespaced under this plugin's own
|
|
1073
|
-
once the plugin is fully enumerated a miss
|
|
1074
|
-
unknown.
|
|
1075
|
+
plugin's agent set was non-empty (i.e. enumerable) -> `subagent-type-not-found-in-plugin`
|
|
1076
|
+
(WARN — provable typo, gates under `--strict`). A value namespaced under this plugin's own
|
|
1077
|
+
name can never be another binary's built-in, so once the plugin is fully enumerated a miss
|
|
1078
|
+
here is a provable typo, not an unconfirmable unknown.
|
|
1075
1079
|
3. Has a `<prefix>:<agent>` shape whose prefix does NOT equal the resolved plugin's name (or
|
|
1076
|
-
there's no resolved plugin at all) -> `subagent-type-unresolvable` (belongs to another
|
|
1080
|
+
there's no resolved plugin at all) -> `subagent-type-unresolvable` (INFO — belongs to another
|
|
1077
1081
|
plugin, or an empty plugin_name means we truly can't tell whose namespace it is).
|
|
1078
1082
|
4. Otherwise (no colon, or same-plugin prefix but the plugin set was empty/unenumerable) ->
|
|
1079
|
-
`subagent-type-unknown` — genuinely can't confirm statically.
|
|
1083
|
+
`subagent-type-unknown` (INFO) — genuinely can't confirm statically.
|
|
1080
1084
|
"""
|
|
1081
1085
|
if value == "general-purpose" or value in plugin_agent_types:
|
|
1082
1086
|
return None
|
|
@@ -1170,10 +1174,10 @@ def cmd_lint_skill(args):
|
|
|
1170
1174
|
_print_findings(all_findings, n_files, kind="skill file", clean_suffix=" — no Cowork host-loop footguns.")
|
|
1171
1175
|
has_error = any(x.severity == "ERROR" for x in all_findings)
|
|
1172
1176
|
# --strict fails on WARN too, per its own --help text ("exit non-zero on WARN too, not just ERROR")
|
|
1173
|
-
# — but NEVER on INFO.
|
|
1174
|
-
#
|
|
1175
|
-
#
|
|
1176
|
-
# must always be surfaced, never failed, even under --strict.
|
|
1177
|
+
# — but NEVER on INFO. Of the subagent_type ladder, only `subagent-type-not-found-in-plugin` is
|
|
1178
|
+
# WARN (a provable in-plugin typo — see the subparser help above); `subagent-type-unresolvable` and
|
|
1179
|
+
# `subagent-type-unknown` stay INFO by design (there is no harness registry to disprove an unknown
|
|
1180
|
+
# value against), so those two must always be surfaced, never failed, even under --strict.
|
|
1177
1181
|
has_warn = any(x.severity == "WARN" for x in all_findings)
|
|
1178
1182
|
if has_error or (args.strict and has_warn):
|
|
1179
1183
|
return 1
|
|
@@ -1364,17 +1368,26 @@ def main(argv=None):
|
|
|
1364
1368
|
"Also statically resolves any pinned `subagent_type` value in the SKILL.md against the "
|
|
1365
1369
|
"enclosing plugin's `agents/*.md` (see `resolve-agent-types`): a value that resolves in-plugin "
|
|
1366
1370
|
"or is `general-purpose` is clean; a `<this-plugin>:<agent>` whose agent is missing from an "
|
|
1367
|
-
"enumerable plugin is a provable typo, reported as `subagent-type-not-found-in-plugin
|
|
1368
|
-
"
|
|
1369
|
-
"
|
|
1370
|
-
"`subagent-type-
|
|
1371
|
-
"
|
|
1371
|
+
"enumerable plugin is a provable typo, reported as `subagent-type-not-found-in-plugin` "
|
|
1372
|
+
"(WARN — gates under `--strict`, since the namespace prefix already commits it to this "
|
|
1373
|
+
"plugin and its agent set was fully enumerated); a `<other-plugin>:<agent>` is reported as "
|
|
1374
|
+
"`subagent-type-unresolvable` (INFO); any other unresolved value (including a same-plugin "
|
|
1375
|
+
"prefix the linter couldn't enumerate) is `subagent-type-unknown` (INFO) — those two stay "
|
|
1376
|
+
"INFO, never WARN, since there is no harness registry of built-in agent types to disprove "
|
|
1377
|
+
"an unknown value against.\n\n"
|
|
1378
|
+
"Plain `lint-skill` (no `--strict`) is ADVISORY — it prints findings but exits 0 on "
|
|
1379
|
+
"WARN/INFO. CI should invoke `lint-skill --strict` to actually gate on the WARN-class "
|
|
1380
|
+
"findings above (the two host-loop footguns and the provable subagent_type typo)."
|
|
1372
1381
|
),
|
|
1373
1382
|
formatter_class=argparse.RawDescriptionHelpFormatter,
|
|
1374
1383
|
)
|
|
1375
1384
|
lsp.add_argument("paths", nargs="+", help="SKILL.md file(s) or skill director(ies) to inspect")
|
|
1376
1385
|
lsp.add_argument("--json", action="store_true", help="emit findings as JSON")
|
|
1377
|
-
lsp.add_argument(
|
|
1386
|
+
lsp.add_argument(
|
|
1387
|
+
"--strict",
|
|
1388
|
+
action="store_true",
|
|
1389
|
+
help="exit non-zero on WARN too, not just ERROR (CI-recommended invocation; plain lint-skill is advisory-only)",
|
|
1390
|
+
)
|
|
1378
1391
|
lsp.set_defaults(func=cmd_lint_skill)
|
|
1379
1392
|
|
|
1380
1393
|
rap = sub.add_parser(
|
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,226 @@ All notable changes to this project are documented here. The format is based on
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [0.33.0] — 2026-07-13
|
|
10
|
+
|
|
11
|
+
### Fixed
|
|
12
|
+
|
|
13
|
+
- **Empty thinking was surfaced as if the model hadn't reasoned — both `subagents[].reasoning` and the
|
|
14
|
+
top-level `thinking[]`.** The 0.31.0 sub-agent field captured TEXT turns fine but every `thinking` turn
|
|
15
|
+
came through as `{kind:"thinking", text:""}`, and the older top-level `thinking[]` field does the same
|
|
16
|
+
on current-gen models — a consumer couldn't tell "reasoned but text unavailable" from "no thought."
|
|
17
|
+
Root cause (binary-verified against the staged 2.1.205 agent, corpus-corroborated): it is a
|
|
18
|
+
**request-side display mode**, not a persist-time strip — the API's `thinking.display` resolves to
|
|
19
|
+
`"omitted"` (empty thinking text + a signature) for sub-agent turns always, and for the MAIN loop too
|
|
20
|
+
on models whose API default is `"omitted"` (Opus 4.8, Sonnet 5; Sonnet 4.6 defaulted to `"summarized"`).
|
|
21
|
+
Corpus: main-loop thinking text is 1810/1810 on Sonnet-4.6 but **0/2 on Opus-4.8 and 0/15 on Sonnet-5**,
|
|
22
|
+
every empty carrying a signature; sub-agents 0/230. The harness passes no `--thinking-display` —
|
|
23
|
+
faithfully to real Cowork, whose spawn passes none either. Both fields now mark such blocks
|
|
24
|
+
`{text:"", redacted:true}` (`kind:"thinking"` for the sub-agent turn shape) so they read as "reasoned,
|
|
25
|
+
text omitted by request"; TEXT turns are never redacted, and `redacted` is omitted (not `false`) on
|
|
26
|
+
blocks that carry text.
|
|
27
|
+
- **New `debug.thinking_display` escape hatch** (fenced, non-Cowork — like `debug.max_thinking_tokens`).
|
|
28
|
+
Set it to `"summarized"` to emit `--thinking-display summarized`, which flips **both** loops to
|
|
29
|
+
summarized thinking text (the API returns no raw chain-of-thought — `summarized` is the ceiling). It
|
|
30
|
+
diverges from real Cowork (which passes no such flag) and adds token cost, so it is a debug-only opt-in;
|
|
31
|
+
the default stays `"omitted"`, byte-identical argv. `docs/subagents.md`, `docs/session.md`,
|
|
32
|
+
`schema/run-result.json`, and the `RunResult`/`SessionConfig` types state the real capture semantics;
|
|
33
|
+
the released 0.31.0 note is corrected in place.
|
|
34
|
+
|
|
35
|
+
- **Docs pointed the GitHub Action at a ref that doesn't exist.** README, `SKILL.md`, and
|
|
36
|
+
`references/ci-recipe.md` all said `uses: yaniv-golan/cowork-harness@v1`, but no `v1` tag has ever
|
|
37
|
+
been published — a copy-pasted workflow failed with "unable to resolve action" before running
|
|
38
|
+
anything. All six references now bind `@main`; a moving major tag is a 1.0.0 question. A new
|
|
39
|
+
token-free guard (`test/action-docs-sync.test.ts`) locks the ref policy and also validates that
|
|
40
|
+
every documented `with:` key is a real `action.yml` input and every documented `command:` value is
|
|
41
|
+
one the Action describes.
|
|
42
|
+
- **`llms.txt` command list was three commands stale** — `lint-skill`, `analyze-skill`, and
|
|
43
|
+
`probe-dispatch` were missing. Now lists all 29, locked by a new COMMANDS ↔ llms.txt sync test
|
|
44
|
+
(set-equality, so stale names fail too), and the README "Commands at a glance" guard gained the
|
|
45
|
+
reverse direction (a README-only row now fails).
|
|
46
|
+
- **README observability prose still called a `subagents[]` field `model`** — renamed to
|
|
47
|
+
`dispatchModel` (beside `resolvedModel`) in 0.30.0; the prose now names the real fields. Also
|
|
48
|
+
fixed: the effort enum shorthand now lists `xhigh` (`extra` is the accepted alias) in both places,
|
|
49
|
+
the CI stage table matches `ci.yml`'s declaration order, the exit-code summary notes
|
|
50
|
+
`verify-cassettes`' distinct exit-`3` meaning and the reserved exit `4`, a duplicated `--keep`
|
|
51
|
+
example was collapsed, and `on_unanswered: llm` is explicitly marked as a scenario-YAML key (not a
|
|
52
|
+
CLI flag).
|
|
53
|
+
- **`status --latest-for` was undocumented in its own guide** — now covered in `docs/run-status.md`
|
|
54
|
+
(with the real output shape), the `docs/README.md` index row, and the top-level `--help` status
|
|
55
|
+
entry (with its own `0` found / `2` not found exit semantics).
|
|
56
|
+
- **`probe-dispatch --help` didn't start with a `usage:` line** like every other command; it does
|
|
57
|
+
now.
|
|
58
|
+
- **`debugging.md` showed only the scenario run-dir layout** — it now also names `chat`'s
|
|
59
|
+
`runs/chat/<sessionId>/` path. `docs/session.md`'s "all four tiers" now spells out the four
|
|
60
|
+
execution tiers and that `fidelity: cowork` resolves to one of them. `examples/README.md`'s
|
|
61
|
+
answer-policies pointer is now a real link. `effectiveFidelity` (the recorded resolved tier behind
|
|
62
|
+
the `resolved-tier`/`unverifiable-tier` staleness classes) is now mentioned in the README fidelity
|
|
63
|
+
section and `python/README.md`, not only in `docs/cassette.md`.
|
|
64
|
+
- **The docs told a false "default `8899`" story for `COWORK_VM_PROXY_PORT`** — when the var is
|
|
65
|
+
unset, the host binds the egress proxy on an OS-assigned free port and threads that value into the
|
|
66
|
+
guest firewall + `HTTP(S)_PROXY`; `8899` is only the guest-config fallback when a VM is spawned
|
|
67
|
+
without an explicit port, which never happens on a normal run. README and `docs/scenario.md` now
|
|
68
|
+
say so. Also closed: `COWORK_HARNESS_STATUS_CORRUPT_TIMEOUT_MS` (30s corrupt-`status.json`
|
|
69
|
+
backstop on `status --follow`) was the one `COWORK_*` env var documented nowhere — now in
|
|
70
|
+
`docs/run-status.md` + the README knob list; the `COWORK_HARNESS_DECIDER_DIR_POLL_MS` docs name
|
|
71
|
+
the per-subsystem defaults (300 ms rendezvous / 500 ms `gates --follow`); the `semantic_matches`
|
|
72
|
+
judge docs name the pinned default (`claude-opus-4-8`) in README, `docs/scenario.md`, and the
|
|
73
|
+
skill's schema reference; the heartbeat bullet states its 30s default instead of pointing at a
|
|
74
|
+
source file.
|
|
75
|
+
- **`replay --explain` was invisible from the command catalog** — the flagship false-green
|
|
76
|
+
diagnosis flag was documented only in debugging prose; it's now in the `record`/`replay` command
|
|
77
|
+
table row and leads the record/replay "Flags worth knowing" bullet.
|
|
78
|
+
- **`examples/README.md` ships in the npm tarball while the trees it describes don't** — it now
|
|
79
|
+
opens with a "Reading this on npm?" callout (clone for `scenarios/`/`sessions/`/`skills/`/`data/`),
|
|
80
|
+
documents `probes/` (previously invisible: used by `test/live-contract.test.ts` but absent from
|
|
81
|
+
the layout and from schema validation — `examples/probes` is now in `test/examples.test.ts`'s
|
|
82
|
+
scenario sweep), and links the matrices worked example; the README documentation-table row carries
|
|
83
|
+
the same source-checkout caveat.
|
|
84
|
+
- **The companion skill's preflight sent replay-only users through `doctor`**, whose token check
|
|
85
|
+
hard-fails on every tier — a new "Replay-only? Skip `doctor`" bullet carries the same carve-out
|
|
86
|
+
`docs/README.md` already had. The 3,200-character single-bullet 0.32.0 feature parenthetical is
|
|
87
|
+
now a scannable by-release sub-list, with every release bucket tag-verified against git history —
|
|
88
|
+
which caught and fixed a long-standing wrong tag: `semantic_matches` was labeled "new in 0.27.0"
|
|
89
|
+
but shipped in 0.28.0; the five path-gate keys, `agent_env`, and the `hostloop` split/
|
|
90
|
+
`allow_host_writes:` now sit under their real releases (0.30.0 / 0.21.0) too.
|
|
91
|
+
- **`DESIGN.md` still said the staged in-VM agent is 2.1.202** — the `desktop-1.20186.1` baseline
|
|
92
|
+
re-synced it to 2.1.205; DESIGN.md and `docs/protocol.md` now carry dated patch-only notes for
|
|
93
|
+
1.20186.1 while the 2026-07-11 live pass stays scoped to `desktop-1.20186.0` (not restamped).
|
|
94
|
+
|
|
95
|
+
### Added
|
|
96
|
+
|
|
97
|
+
- **`test/docs-index-sync.test.ts`** — three token-free guards: every `COWORK_*` env var read in
|
|
98
|
+
`src/` (dot-access **or** helper-read string literal) must be documented in README/`docs/*.md`;
|
|
99
|
+
the judge-model default id in the docs must match `semantic-judge.ts` (fails loud on a const
|
|
100
|
+
rename); `llms.txt` must link every top-level `docs/*.md` guide (both directions — `gotchas.md`,
|
|
101
|
+
`subagents.md`, and `plugin-root.md` were missing and are now linked).
|
|
102
|
+
|
|
103
|
+
- **The Action's `version` default (`latest`) is now documented as intentional** — in `action.yml`'s
|
|
104
|
+
input description, the README inputs sentence, and `references/ci-recipe.md` — with
|
|
105
|
+
pin-an-exact-version guidance for reproducible CI, and why it deliberately differs from the
|
|
106
|
+
companion skill's `@>=0.32.0` floor for ad-hoc CLI installs.
|
|
107
|
+
- **`--help` and unknown-flag structural-guard test coverage now spans all 29 commands** (previously
|
|
108
|
+
16 and 17 respectively); `lint`/`lint-skill` are asserted against their `scenario.py` passthrough
|
|
109
|
+
usage lines and skip cleanly when `python3` is absent.
|
|
110
|
+
|
|
111
|
+
### Documentation
|
|
112
|
+
|
|
113
|
+
- **`on_unanswered: prompt` was described as "only valid for `chat`" — wrong on two counts.** `chat`
|
|
114
|
+
never reads a scenario YAML (it runs an inline interactive scenario), and `prompt` is really a
|
|
115
|
+
`skill`-command policy (the adaptive-TTY default, or explicit `skill --on-unanswered prompt`);
|
|
116
|
+
`run` rejects it. The schema `.describe()` (and regenerated `scenario.schema.json`) now say so.
|
|
117
|
+
- **`ANTHROPIC_AUTH_TOKEN` is an accepted auth source but was under-documented.** It resolves
|
|
118
|
+
identically to `ANTHROPIC_API_KEY` (used only when no OAuth token is set) and was already in
|
|
119
|
+
`.env.example`, but the README auth text, the `record`/`doctor` `--help` blurbs, the `doctor`
|
|
120
|
+
no-token detail/remedy, the two `record` credential messages, `docs/cassette.md`, and the companion
|
|
121
|
+
skill's Auth note named only the other two — all now list it as the third alternative.
|
|
122
|
+
- **`llms.txt` exit-code line** now flags that `3`/`1` carry per-command meanings (e.g.
|
|
123
|
+
`verify-cassettes` `3` = could-not-verify, `sync` hard-fail = `1`) and links the authoritative
|
|
124
|
+
SPEC §11 text, instead of implying one global meaning for `3`.
|
|
125
|
+
- **README** gains a platform × tier support matrix (making explicit that Linux live runs are
|
|
126
|
+
`container`-only), doc-index rows for the spawn contract and `docs/decisions/`, the
|
|
127
|
+
`probe-dispatch` fidelity set, and a clearer global-install-ships-`examples/replays/`-only warning;
|
|
128
|
+
`docs/chat.md` notes that scaffolding a `chat` run yields an empty `assert:` block.
|
|
129
|
+
|
|
130
|
+
## [0.32.0] — 2026-07-13
|
|
131
|
+
|
|
132
|
+
### Added
|
|
133
|
+
|
|
134
|
+
- **`assertions --list` now documents that `subagent_dispatch_healthy`'s `type` matches the dispatch
|
|
135
|
+
description** (not just resolvedAgentType/dispatchAgentType), so a regex can narrow to one dispatch in a
|
|
136
|
+
same-agent-type fleet — previously only in `docs/scenario.md`.
|
|
137
|
+
- **`analyze-skill` line- and block-scoped ignore markers** — `analyze-skill: ignore-next-line` and
|
|
138
|
+
`analyze-skill: ignore-start` / `analyze-skill: ignore-end`, alongside the existing file-wide
|
|
139
|
+
`analyze-skill: ignore`. The file-wide marker silences EVERY finding in the file, which was too coarse
|
|
140
|
+
for a SKILL.md that carries just one `/sessions`-addressed teaching example (e.g. a "don't do this"
|
|
141
|
+
callout in `references/`/`agents/`): the whole file went blind to any OTHER, genuine finding. The two
|
|
142
|
+
new markers suppress only the exact line (`ignore-next-line`) or fenced range (`ignore-start`/
|
|
143
|
+
`ignore-end`, inclusive) they scope, reusing the same line-anchored marker matching as the file-wide
|
|
144
|
+
marker (bare, `#`-prefixed, list-bullet, HTML comment, or markdown reference-link comment — never
|
|
145
|
+
triggered by a marker merely documented mid-prose). An `ignore-start` with no matching `ignore-end`
|
|
146
|
+
before EOF emits its own gating `unclosed-ignore-fence` finding — it still suppresses to EOF (fail-open),
|
|
147
|
+
but the missing-`ignore-end` mistake itself prints and fails under `--strict` like any other finding,
|
|
148
|
+
never a silent notice on a green exit.
|
|
149
|
+
|
|
150
|
+
- **`analyze-skill <dir>` now scans the UNION of every contract-bearing markdown file in the directory,
|
|
151
|
+
not just `SKILL.md`.** A plugin's `/sessions` dispatch/output contracts often live in `agents/**` or
|
|
152
|
+
`references/**` — scanning only `SKILL.md` there was a false green (a consumer's `agents/sub/x.md`
|
|
153
|
+
could hand a `/sessions` path straight to a file tool and `analyze-skill` would never see it). A
|
|
154
|
+
directory target is now expanded to every shape it matches, deduped by resolved absolute path: a
|
|
155
|
+
top-level `SKILL.md` (+ its `references/**`); a plugin root (`.claude-plugin/plugin.json` or
|
|
156
|
+
`plugin.json`, + `agents/**`, `references/**`, `commands/**`, and each `skills/*/SKILL.md` + that
|
|
157
|
+
skill's own `references/**`); and a skill dir living inside a plugin also pulls in the enclosing
|
|
158
|
+
plugin's `agents/**`, `references/**`, and `commands/**`. Every walk is RECURSIVE (Claude Code discovers
|
|
159
|
+
namespaced commands/agents in subdirectories — `commands/tasks/build.md`, `agents/sub/x.md` — so a
|
|
160
|
+
single-level listing silently narrowed the scan) and FOLLOWS directory symlinks, loop-guarded against a
|
|
161
|
+
self-referencing symlink via a realpath visited-set, so a symlinked `references/`/`skills/<name>` is
|
|
162
|
+
scanned rather than silently dropped. Zero scannable files under a directory target is now a usage error
|
|
163
|
+
(exit 2) that enumerates the shapes it looked for — never a silent clean pass.
|
|
164
|
+
|
|
165
|
+
- **`analyze-skill` accepts MULTIPLE positionals + a simple `*` glob, matching `lint-skill`'s
|
|
166
|
+
`nargs="+"`.** A consumer with several explicit paths (or a directory's flat `*.md` set) no longer has
|
|
167
|
+
to loop `analyze-skill` once per path: `analyze-skill a/SKILL.md b/SKILL.md`, `analyze-skill skill-a/
|
|
168
|
+
skill-b/`, and `analyze-skill "plug/agents/*.md"` are all one call now. Each positional resolves
|
|
169
|
+
independently — a file or directory via the existing rules above, or a `*`-bearing target via a small
|
|
170
|
+
HAND-ROLLED glob matcher (no new dependency, no `engines.node` bump): `dir/*.md` matches shallowly,
|
|
171
|
+
`dir/**/*.md` matches recursively (reusing the same symlink-following, loop-guarded walker the
|
|
172
|
+
directory-target union scan already uses). Every positional's files are UNIONed and deduped by
|
|
173
|
+
resolved absolute path across the WHOLE invocation, extending the existing single-target dedup to
|
|
174
|
+
span all of them — a file reached
|
|
175
|
+
both directly and through a dir/glob positional is analyzed once. Zero scannable files across ALL
|
|
176
|
+
positionals remains the usage error (exit 2); a bad single positional (missing path, unrecognized glob
|
|
177
|
+
shape) fails the whole invocation.
|
|
178
|
+
|
|
179
|
+
### Changed (breaking, pre-1.0)
|
|
180
|
+
|
|
181
|
+
- **`lint-skill`'s `subagent-type-not-found-in-plugin` finding is now WARN, not INFO** — so
|
|
182
|
+
`lint-skill --strict` newly gates on it. This is the one `subagent_type` outcome that's a *provable*
|
|
183
|
+
typo: a `<this-plugin>:<agent>` value whose prefix names the SKILL.md's own enclosing plugin, where
|
|
184
|
+
that plugin's `agents/*.md` was fully enumerated and doesn't contain the agent — the namespace prefix
|
|
185
|
+
already commits the value to this plugin, so a miss can never be another binary's built-in hiding
|
|
186
|
+
from static analysis. The other two `subagent_type` outcomes stay INFO, unchanged, because they
|
|
187
|
+
genuinely can't be disproven: `subagent-type-unresolvable` (a `<other-plugin>:<agent>` — belongs to
|
|
188
|
+
a plugin this linter didn't scan) and `subagent-type-unknown` (a bare value with no in-plugin match —
|
|
189
|
+
may be an agent-binary built-in). A `lint-skill --strict` run that previously passed clean may now
|
|
190
|
+
fail if it carries this exact typo shape; fix the agent name (or add the missing `agents/<agent>.md`)
|
|
191
|
+
to restore a clean `--strict` run.
|
|
192
|
+
- **`analyze-skill <dir> --strict` may newly fail where it passed clean before**, if `references/`/
|
|
193
|
+
`agents/`/`commands/` carries a `/sessions`-addressed teaching example that the old SKILL.md-only scan
|
|
194
|
+
never looked at — including one nested under a subdirectory or reached only via a symlink, both now
|
|
195
|
+
scanned. Annotate the example with `analyze-skill: ignore-next-line` (or an `ignore-start`/`ignore-end`
|
|
196
|
+
fence) to restore a clean `--strict` run — there is no `--skill-md-only` compatibility flag; the
|
|
197
|
+
narrower scan was the bug this closes.
|
|
198
|
+
- **`analyze-skill --output-format json`'s single-file shape changed.** The flat
|
|
199
|
+
`{file, findings, suppressed, strict}` payload is now always `{files: [{file, findings, suppressed}],
|
|
200
|
+
scanned, unscanned, strict}`, for uniformity with a directory target's multi-file result. An external
|
|
201
|
+
`jq '.findings'` / `jq '.file'` recipe needs `jq '.files[0].findings'` / `jq '.files[0].file'` instead.
|
|
202
|
+
- **`analyze-skill`'s JSON envelope `ok` now mirrors the exit code, not "zero findings."** Previously
|
|
203
|
+
`ok = findings.length === 0`, so an ADVISORY run (the default — findings print but exit 0) reported
|
|
204
|
+
`ok:false` even though the process exited 0 and `action.yml`'s own documented contract (`ok` "mirrors
|
|
205
|
+
the command's exit code") said otherwise; the packaged Action's job-summary reporter then rendered
|
|
206
|
+
"Overall: ❌ fail" for a run that hadn't actually failed. `ok` is now `true` on an advisory exit-0 run
|
|
207
|
+
even with findings present, and `false` only under `--strict` with findings or the exit-2 usage error
|
|
208
|
+
— the same rule `lint`/`lint-skill --output-format json` follow (see the Fixed entry below). A `jq
|
|
209
|
+
'.ok'` consumer that was treating `ok:false` as "any findings at all" should switch to checking
|
|
210
|
+
`.files[].findings` directly; the exit code (and `ok`) now only reflect `--strict` gating.
|
|
211
|
+
|
|
212
|
+
### Fixed
|
|
213
|
+
|
|
214
|
+
- **The packaged Action's `lint`/`lint-skill` lanes were broken.** `action.yml` always appends
|
|
215
|
+
`--output-format json` to every command it drives, but the bundled `scenario.py`'s `lint`/`lint-skill`
|
|
216
|
+
only understand `--json` — `--output-format` was an unrecognized argument to python's argparse (exit 2,
|
|
217
|
+
empty stdout), so `command: lint` / `command: lint-skill` in the Action always failed with an
|
|
218
|
+
unhelpful `ok:false` and no findings. `cowork-harness lint`/`lint-skill` now detect json mode
|
|
219
|
+
themselves, strip `--output-format json|text|=…` before forwarding to python, pass python's own
|
|
220
|
+
`--json` in json mode, and re-wrap the child's bare findings array in the harness's standard
|
|
221
|
+
`jsonPayloadEnvelope("lint"/"lint-skill", ok, { findings })` (`ok` mirrors the child's exit code). Text
|
|
222
|
+
mode (the default) is unaffected in output — `stdio: "inherit"` as before — but now also strips the
|
|
223
|
+
flag first, since python didn't know the text form either. A python-level usage error in json mode
|
|
224
|
+
(e.g. a missing required path — argparse exit 2, empty stdout) now surfaces as a `jsonError("usage",
|
|
225
|
+
…)` envelope instead of a raw `JSON.parse` crash. `action.yml` also now documents `lint-skill` and
|
|
226
|
+
`analyze-skill` in its `command` input, and no longer forwards the replay-only `--fail-on-skill-drift`
|
|
227
|
+
flag to any command other than `replay`.
|
|
228
|
+
|
|
9
229
|
## [0.31.0] — 2026-07-12
|
|
10
230
|
|
|
11
231
|
### Added
|
|
@@ -26,6 +246,11 @@ All notable changes to this project are documented here. The format is based on
|
|
|
26
246
|
`thinking[]` field is (~50 entries, ~10KB/entry each), with `reasoningElided` counting the overflow. A
|
|
27
247
|
missing or malformed child transcript never fails the run — the affected dispatch's `reasoning` is just
|
|
28
248
|
left absent.
|
|
249
|
+
- **Correction (see Unreleased):** in practice only the TEXT turns carry content — the harness's
|
|
250
|
+
non-interactive spawn forces the API's `thinking.display` to `"omitted"` for sub-agent turns, so
|
|
251
|
+
`thinking` turns come through empty (signature-only). The Unreleased `redacted` marker distinguishes
|
|
252
|
+
"reasoned, text omitted by request" from "no thought"; the "THINKING … turns" wording above
|
|
253
|
+
overstated what this captures by default.
|
|
29
254
|
|
|
30
255
|
- **`status --latest-for <scenario-name-or-slug>`** — resolves and prints the NEWEST run dir for a
|
|
31
256
|
scenario by actual run time, replacing the fragile `ls -td runs/<scenario>/* | head -1` idiom: bare
|