cowork-harness 0.29.0 → 0.30.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/cowork-harness/SKILL.md +25 -11
- package/.claude/skills/cowork-harness/references/ci-recipe.md +4 -4
- package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +1 -1
- package/.claude/skills/cowork-harness/references/scenario-schema.md +29 -9
- package/.claude/skills/cowork-harness/references/task-recipes.md +2 -2
- package/.claude/skills/cowork-harness/scripts/assertion-keys.json +6 -1
- package/.claude/skills/cowork-harness/scripts/scenario.py +5 -0
- package/CHANGELOG.md +169 -0
- package/CONTRIBUTING.md +1 -1
- package/README.md +11 -28
- package/RELEASING.md +27 -0
- package/SPEC.md +14 -9
- package/baselines/desktop-1.18286.2.json +1 -0
- package/baselines/desktop-1.19367.0.json +1 -0
- package/baselines/desktop-1.20186.0.json +1 -0
- package/baselines/desktop-1.20186.1.json +380 -0
- package/baselines/prompts/cowork-system-prompt-fingerprints.json +4 -0
- package/baselines/prompts/desktop-1.18286.2/subagent-append-hl.md +16 -0
- package/dist/agent/session.js +70 -9
- package/dist/agent/timeline.js +6 -1
- package/dist/assert.js +95 -5
- package/dist/cli.js +48 -7
- package/dist/decide/decider.js +1 -1
- package/dist/hostloop/canusetool-gate.js +63 -0
- package/dist/hostloop/pretooluse-path-hook.js +62 -22
- package/dist/prompt.js +33 -10
- package/dist/run/cassette.js +236 -103
- package/dist/run/chat-result.js +2 -0
- package/dist/run/chat.js +30 -11
- package/dist/run/execute.js +702 -656
- package/dist/run/renderer.js +1 -1
- package/dist/run/run.js +267 -44
- package/dist/run/trace-view.js +27 -7
- package/dist/run/verdict.js +16 -2
- package/dist/runtime/argv.js +6 -1
- package/dist/runtime/container.js +7 -2
- package/dist/runtime/hostloop-prompt.js +5 -5
- package/dist/runtime/hostloop.js +51 -13
- package/dist/runtime/microvm.js +3 -1
- package/dist/runtime/protocol.js +17 -1
- package/dist/session.js +42 -0
- package/dist/sync/cowork-sync.js +296 -5
- package/dist/types.js +48 -2
- package/docker/Dockerfile.agent +1 -1
- package/docs/README.md +2 -1
- package/docs/boundary.md +21 -2
- package/docs/cassette.md +35 -17
- package/docs/cowork-spawn-contract-1.12603.1.md +3 -2
- package/docs/fidelity-gaps.md +85 -0
- package/docs/gotchas.md +0 -3
- package/docs/invariants.md +4 -2
- package/docs/maintenance.md +19 -0
- package/docs/plugin-root.md +16 -7
- package/docs/scenario.md +31 -6
- package/docs/session.md +37 -0
- package/docs/subagents.md +154 -0
- package/examples/replays/README.md +1 -1
- package/examples/replays/example-multiselect-gate.cassette.json +1 -1
- package/examples/replays/example-pdf-skill.cassette.json +1 -1
- package/examples/replays/hostloop-computer-links.cassette.json +1 -1
- package/package.json +3 -1
- package/schema/cassette.v10.json +4 -0
- package/schema/protocol.v1.json +1 -0
- package/schema/run-result.json +65 -7
- package/schema/scenario.schema.json +72 -2
- package/schema/session.schema.json +20 -0
- package/scripts/check-versions.ts +8 -15
- package/dist/canary/grants.js +0 -37
- package/dist/critique/evaluator.js +0 -321
- package/dist/critique/evidence.js +0 -175
- package/dist/critique/package-evidence.js +0 -198
- package/docs/assets/banner.png +0 -0
- package/fixtures/subagent-grants.json +0 -5
- package/schema/cassette.v2.json +0 -19
- package/schema/cassette.v3.json +0 -30
- package/schema/cassette.v5.json +0 -107
- package/schema/cassette.v6.json +0 -131
- package/schema/cassette.v7.json +0 -156
- package/schema/cassette.v8.json +0 -166
- package/scripts/boot-rootfs-vz.ts +0 -99
|
@@ -3,8 +3,8 @@ name: cowork-harness
|
|
|
3
3
|
description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
|
|
4
4
|
metadata:
|
|
5
5
|
author: cowork-harness
|
|
6
|
-
version: 0.
|
|
7
|
-
tracks-harness: cowork-harness 0.
|
|
6
|
+
version: 0.30.0
|
|
7
|
+
tracks-harness: cowork-harness 0.30.0 (baseline desktop-1.20186.1)
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
# cowork-harness
|
|
@@ -22,7 +22,7 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
|
|
|
22
22
|
allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
|
|
23
23
|
the highest-value part. Read it.
|
|
24
24
|
|
|
25
|
-
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.
|
|
25
|
+
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.30.0` (baseline
|
|
26
26
|
> `desktop-1.20186.0`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
|
|
27
27
|
> `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
|
|
28
28
|
|
|
@@ -38,7 +38,7 @@ cowork-harness skill ./my-skill "do X" # run the skill once against the sta
|
|
|
38
38
|
Before the first command, confirm the CLI is reachable and **fail loud** (never fake a pass) when a tier's dependencies are missing:
|
|
39
39
|
|
|
40
40
|
- **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
|
|
41
|
-
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.
|
|
41
|
+
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.30.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=0.30.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=0.30.0"`. **Pin `@>=0.30.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published. (≥ 0.30.0 is what gates the commands/assertions this skill teaches: `assertions --list`, `scaffold <run-id>`, `trace --view dispatches`, `artifact_json` incl. the `in:` operator (passes when the resolved value deep-equals one of the listed members — value ∈ your list, not the reverse), `verify-cassettes`, batch `record <dir>`/`--rerecord-stale`, `record --concurrency <N>`, record-time redaction, multiSelect/`answer:`, `verify-run` answer-coverage, `record --max-artifact-bytes`, live record-time deciders, `verify-cassettes --allow-domain`/`--allow-email`/`--allow-path`/`--allow-patterns-file` (`path` — local absolute filesystem paths — is the scanner's 4th class, new in 0.21.0; `--allow-patterns-file <path>` is a FILE of patterns, one regex per line — not a path to allow, unlike `--allow <regex>`), scenario `skills:` staleness scoping with `COWORK_HARNESS_AGENT_SCOPE=skill`, `chat --plugin`, `/help` in the REPL, `hostloop`'s native host/VM process split with its `allow_host_writes:` consent field, `computer_links_resolve` (new in 0.22.0), `semantic_matches` (an LLM judge grades a fixed `rubric` of claims against the run's answer — **live-only**, so it is evidence-unavailable / skipped-loud on replay, never a vacuous pass; new in 0.27.0), **glob-matched** `tool_called`/`tool_not_called`/`subagent_tool_used`/`subagent_tool_absent` (a pattern like `mcp__workspace__*` matches any tool in the family; exact names match exactly; new in 0.28.0), the five path-gate assertion keys `no_vm_path_file_op`/`vm_path_denied`/`path_denied`/`no_path_denied`/`subagent_file_write`, the session-level `agent_env` knob, and resolved sub-agent identity on dispatch records (`resolvedAgentType`/`resolvedModel`; new in 0.30.0).)
|
|
42
42
|
- **Agent binary (sandboxed live tiers — `container`/`microvm`/`hostloop`/`cowork`).** The staged Claude Code agent is **bind-mounted** from a local Claude Desktop install, or point `COWORK_AGENT_BINARY` at a `claude-code-vm/<ver>/claude` ELF. Nothing is bundled. `protocol` (L0) and `replay` need no staged agent; for the sandboxed tiers, no agent → no run; report that, don't skip silently.
|
|
43
43
|
- **Docker / Lima.** Only `--fidelity protocol` (L0) runs without them. `container` / `microvm` / `hostloop` / `cowork` need Docker (Lima for L2). If they're absent, drop to `--fidelity protocol` and **say so** — a green that never exercised the sandbox is not a sandbox pass.
|
|
44
44
|
- **Auth.** `CLAUDE_CODE_OAUTH_TOKEN` (preferred) or `ANTHROPIC_API_KEY`, via env or `.env`. Minting an OAuth token needs the **`claude` CLI** (`npm i -g @anthropic-ai/claude-code`, then `claude setup-token`).
|
|
@@ -63,7 +63,7 @@ reproducible regression (Part II), and **debug** a run that misbehaved or greene
|
|
|
63
63
|
the path, `trace <run-id>` finds it). **Localize the failure post-hoc** from that evidence:
|
|
64
64
|
`cowork-harness trace <run-dir>`'s views + the emitted `result.json` to see what the run actually did,
|
|
65
65
|
then `verify-run` to re-check a suspect assertion — all token-free, no Docker, no re-record. This is
|
|
66
|
-
the loop 0.
|
|
66
|
+
the loop 0.30.0's observability is built for; the *Triage* and *Inspecting a run's observability
|
|
67
67
|
output* sections in **Part III — Debug** are the detail (the fuller human-facing map lives in
|
|
68
68
|
`docs/debugging.md` — repo-only, not shipped with the installed skill).
|
|
69
69
|
- **Multi-turn / interactive reproduction** → `cowork-harness chat` (interactive; gates answered at the
|
|
@@ -292,7 +292,9 @@ scenario's gates or their labels: `--keep` ONE run, then `cowork-harness trace <
|
|
|
292
292
|
your `answers:` against that kept run, then record once. **But the kept run is a snapshot:** if you change the
|
|
293
293
|
skill's gate phrasing afterward, re-`--keep` — verify-run's answer-coverage *refuses* (exit 2, "predates the
|
|
294
294
|
current skill") rather than vouch against stale labels, but the trace/inspect path can't warn you, so re-keep
|
|
295
|
-
deliberately. (
|
|
295
|
+
deliberately. (Same fail-closed family: corrupt gate evidence — unparseable `events.jsonl` lines, or fewer
|
|
296
|
+
gates than `trace.json` recorded questions — and a structurally invalid `result.json` also refuse rather
|
|
297
|
+
than certify.) (A token-free probe of "which gates fire" isn't possible — gates are model-decided per run.)
|
|
296
298
|
|
|
297
299
|
Run artifacts are written to `~/.cowork-harness/runs/…` by default — **outside any working tree**, so a run
|
|
298
300
|
launched from a repo root never drops sensitive skill inputs/outputs into it. Pass `--run-dir <path>` (or set
|
|
@@ -420,8 +422,8 @@ decide which assertions from *Assertions: two orthogonal axes* are worth adding)
|
|
|
420
422
|
- **`cowork-harness stats [--metric <m>]`** — aggregate across the run index: `cost`, `duration`,
|
|
421
423
|
`tokens`, `cache-tokens`, `model-cost`, `turns`, `pass-rate`.
|
|
422
424
|
- **`result.json` carries the raw fields** the assertions read: `toolDurations`, `models`, `toolErrors`,
|
|
423
|
-
`redundantToolCalls`, `modelUsage`, `thinking`, `skillActivity`, `subagents[]` (prompt
|
|
424
|
-
`attributedSkillId`, `outputTruncated`), `context` (tools/mcpServers/availableSkills), `tasks`,
|
|
425
|
+
`redundantToolCalls`, `modelUsage`, `thinking`, `skillActivity`, `subagents[]` (prompt/`dispatchModel`/
|
|
426
|
+
`resolvedModel`/output/`attributedSkillId`, `outputTruncated`), `context` (tools/mcpServers/availableSkills), `tasks`,
|
|
425
427
|
`workspaceFiles`, `presentedFiles`, `hookEvents`, `mcpErrors`, `contextEvents`, `resources`
|
|
426
428
|
(`probeFailures` distinguishes a failed sample from a tier that was never sampleable). Provenance/
|
|
427
429
|
evidence-health fields: `command` (`run`/`skill`/`record`/`chat`/`replay` — finer than `mode`),
|
|
@@ -439,7 +441,7 @@ decide which assertions from *Assertions: two orthogonal axes* are worth adding)
|
|
|
439
441
|
than debugging; `"transport"`/`"agent"` means something actually broke, worth localizing before
|
|
440
442
|
re-running.
|
|
441
443
|
- **Attributing cost to sub-agent work.** `subagents[]` gives the dispatch tree — each sub-agent's
|
|
442
|
-
`
|
|
444
|
+
`dispatchModel`/`resolvedModel`, `toolsUsed`, `prompt`/`output`, and `attributedSkillId` — but **not** its own token/cost;
|
|
443
445
|
aggregate cost is per-**model** in `modelUsage` (and `trace --view usage`), not per-sub-agent. So a
|
|
444
446
|
cost spike from fan-out reads as `trace --view dispatches` (how many, which agent) against that model's
|
|
445
447
|
per-model usage — the harness doesn't line-item each sub-agent's tokens.
|
|
@@ -514,8 +516,20 @@ repeats the assertion/replay-relevant ones alongside the schema (a scoped subset
|
|
|
514
516
|
to prove an attempt; it proves the tool actually ran.
|
|
515
517
|
|
|
516
518
|
5. **`subagent_declared_but_unused` fires on declared-but-didn't-use-THAT-tool**, even if the
|
|
517
|
-
sub-agent used other tools.
|
|
518
|
-
|
|
519
|
+
sub-agent used other tools. `subagent_dispatched` / `subagent_output_contains` match on dispatch
|
|
520
|
+
type (`dispatchAgentType`), the binary-*resolved* type (`resolvedAgentType`), *or* the dispatch
|
|
521
|
+
**description** — so a type-less dispatch that resolved to e.g. `general-purpose` is still
|
|
522
|
+
selectable, by either the resolved type or the description. A `Task` dispatch that carries NO
|
|
523
|
+
`subagent_type` at all falls back to the built-in `general-purpose` agent with a **wildcard tool
|
|
524
|
+
surface** (`tools:["*"]`, including workspace bash) — faithful production behavior, and it fires
|
|
525
|
+
routinely. The harness warns loudly on this fallback and records `subagents[].dispatchTypeOmitted`;
|
|
526
|
+
an *explicit* `subagent_type: "general-purpose"` is a deliberate author choice and does not warn.
|
|
527
|
+
Implication: `subagent_tool_absent` on a type-less dispatch is weaker evidence (wildcard surface) —
|
|
528
|
+
pin `subagent_type` explicitly when you need a tight tool-absence guarantee.
|
|
529
|
+
|
|
530
|
+
**Cross-tier "no shell" caveat.** On `hostloop`, native `Bash` calls route through the
|
|
531
|
+
`mcp__workspace__bash` alias, so a "sub-agent used no shell" check must glob **both** `Bash` and
|
|
532
|
+
`mcp__workspace__*` to hold across every tier.
|
|
519
533
|
|
|
520
534
|
6. **`dispatch_count_max` is an author-chosen budget, not a production cap.** It's a post-hoc count
|
|
521
535
|
assertion: passing means "happened to dispatch ≤N this run," nothing more. Cowork imposes **no**
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# CI recipe — replay vs live lanes
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 0.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 0.30.0` (baseline `desktop-1.20186.0`).
|
|
4
4
|
|
|
5
5
|
**Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
|
|
6
6
|
job-summary reporter (verdict table, staleness findings, cost/turns when available):
|
|
@@ -54,7 +54,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
|
|
|
54
54
|
GitHub-hosted runners, no token/Docker/agent:
|
|
55
55
|
|
|
56
56
|
```yaml
|
|
57
|
-
- run: npm i -g "cowork-harness@>=0.
|
|
57
|
+
- run: npm i -g "cowork-harness@>=0.30.0"
|
|
58
58
|
- run: cowork-harness lint scenarios/*.yaml # no silent false-greens
|
|
59
59
|
- run: cowork-harness verify-cassettes cassettes/ # privacy + staleness
|
|
60
60
|
- run: cowork-harness replay cassettes/ # token-free content/structure
|
|
@@ -188,7 +188,7 @@ jobs:
|
|
|
188
188
|
with: { node-version: '20' }
|
|
189
189
|
- uses: actions/setup-python@v5
|
|
190
190
|
with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
|
|
191
|
-
- run: npm i -g "cowork-harness@>=0.
|
|
191
|
+
- run: npm i -g "cowork-harness@>=0.30.0"
|
|
192
192
|
- run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
|
|
193
193
|
- run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
|
|
194
194
|
- run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
|
|
@@ -217,7 +217,7 @@ jobs:
|
|
|
217
217
|
echo "live=true" >> "$GITHUB_OUTPUT"
|
|
218
218
|
fi
|
|
219
219
|
- if: steps.guard.outputs.live == 'true'
|
|
220
|
-
run: npm i -g "cowork-harness@>=0.
|
|
220
|
+
run: npm i -g "cowork-harness@>=0.30.0"
|
|
221
221
|
- if: steps.guard.outputs.live == 'true'
|
|
222
222
|
run: cowork-harness run scenarios/ --output-format json
|
|
223
223
|
env:
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Fidelity tiers & answer paths
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 0.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 0.30.0` (baseline `desktop-1.20186.0`).
|
|
4
4
|
|
|
5
5
|
## Fidelity tiers (`fidelity:` in the scenario)
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Scenario & session schema, assertion catalog, web_fetch, full gotchas
|
|
2
2
|
|
|
3
|
-
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.
|
|
3
|
+
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.30.0`
|
|
4
4
|
(baseline `desktop-1.20186.0`). If your checkout is newer, prefer the live `docs/scenario.md`,
|
|
5
5
|
`docs/session.md`, and `SPEC.md`.
|
|
6
6
|
|
|
@@ -105,6 +105,12 @@ agent_max_turns: 500 # optional turn ceiling -> agent --max-turns;
|
|
|
105
105
|
permission_mode: default # default | acceptEdits | plan | bypassPermissions
|
|
106
106
|
permission_parity: cowork # cowork (unscripted tool calls allowed) | strict (deny unscripted)
|
|
107
107
|
|
|
108
|
+
# sub-agent / tool-search env knob (tier-uniform; maps to agent env vars)
|
|
109
|
+
agent_env:
|
|
110
|
+
subagent_model: claude-opus-4-8 # -> CLAUDE_CODE_SUBAGENT_MODEL
|
|
111
|
+
tool_search: auto # auto | off -> ENABLE_TOOL_SEARCH; omit = binary default (ToolSearch ON)
|
|
112
|
+
disable_experimental_betas: false # true -> CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1 (also disables ToolSearch)
|
|
113
|
+
|
|
108
114
|
# fenced debug escape hatch (NOT reachable via Cowork's UI)
|
|
109
115
|
debug:
|
|
110
116
|
max_thinking_tokens: 50000 # overrides --max-thinking-tokens directly; bypasses extended_thinking's
|
|
@@ -254,9 +260,11 @@ same set live from the schema.
|
|
|
254
260
|
| `tool_result_not_contains: <str>` | no tool result includes the literal string (content / replay-checkable; fails loud when tool results are absent) |
|
|
255
261
|
| `subagent_tool_used: <glob>` | a sub-agent used a tool matching this glob (same `*`/`?`, anchored, case-sensitive semantics as `tool_called`, including the empty/regex-ish rejection) |
|
|
256
262
|
| `subagent_tool_absent: <glob>` | no sub-agent used a tool matching this glob (same rejection) |
|
|
257
|
-
| `
|
|
263
|
+
| `no_vm_path_file_op: true` | **`fidelity: hostloop` only** — NO gated file tool attempted a `/sessions`(-prefixed) path (`RunResult.fileToolAttempts`) — content-class, replay-checkable without `controlOut`; any other tier FAILS "cannot verify" (`/sessions/...` is valid there). **Only `true` is valid** |
|
|
264
|
+
| `subagent_file_write: {path?, path_suffix?, tool?}` | a sub-agent-origin write attempt whose raw path equals `path` (exact) or ends with `path_suffix` has a paired non-error tool_result — the causal half of a delivery probe; requires one of `path`/`path_suffix`; `tool` defaults to Write/Edit/MultiEdit; content-class; tier-agnostic |
|
|
265
|
+
| `subagent_dispatched: <regex>` | a sub-agent whose `dispatchAgentType`, binary-*resolved* `resolvedAgentType`, **or dispatch description** matches |
|
|
258
266
|
| `subagent_declared_but_unused: <Tool>` | a sub-agent declared the tool but never used **that** tool (even if it used others) |
|
|
259
|
-
| `subagent_output_contains: {match?, contains}` | a dispatched sub-agent's own output contains the substring `contains` — `match` (optional regex over `
|
|
267
|
+
| `subagent_output_contains: {match?, contains}` | a dispatched sub-agent's own output contains the substring `contains` — `match` (optional regex over `dispatchAgentType`/`resolvedAgentType`/`description`) narrows to specific dispatch(es); omitted, checks whether ANY dispatch's output contains it (existence check, not "all"); a miss against an output that was **truncated at the assert cap** reports evidence-unavailable instead of a proven absence — the substring could lie past the cut |
|
|
260
268
|
| `dispatch_count_max: <N>` | at most N sub-agents dispatched — an author-chosen budget (Cowork imposes no in-conversation Task-dispatch cap; records only, enforces nothing — see gotcha 12) |
|
|
261
269
|
| `skill_triggered: <regex>` | a skill matching the regex (invoked id, e.g. `"plugin:skill"`) was invoked via the `Skill` tool — evidence-unavailable (not a normal fail) if the agent's init tools have no `Skill` tool |
|
|
262
270
|
| `no_skill_triggered: <regex>` | no invoked skill id matched — the negative-control / description-collision catcher; evidence-unavailable (never a vacuous pass) if invocation data is absent or the `Skill` tool is unobservable |
|
|
@@ -291,6 +299,9 @@ same set live from the schema.
|
|
|
291
299
|
| `gate_answer_count_min: <N>` | at least N AskUserQuestion gates fired AND were delivered non-error — presence companion to `gate_answers_delivered`'s vacuous-pass |
|
|
292
300
|
| `hook_blocked: <regex>` | a PreToolUse hook blocked a tool whose name matches the regex (`RunResult.hookEvents`) — evidence-unavailable if hook telemetry is absent. Replay: needs a `controlOut` cassette (a custom hook's decision lives only there, not the recorded stream) |
|
|
293
301
|
| `no_hook_blocked: true` | no tool was hook-blocked during the run (distinguishes a real tool crash from an intentional hook block) — evidence-unavailable if hook telemetry is absent. Replay: needs a `controlOut` cassette. **Only `true` is valid** |
|
|
302
|
+
| `vm_path_denied: true` | **`fidelity: hostloop` only** — at least one recorded path denial (`RunResult.pathDenials`, any source) targeted a `/sessions` VM path — evidence-unavailable if path-denial telemetry is absent. Replay: needs a `controlOut` cassette. Any other tier FAILS "cannot verify". **Only `true` is valid** |
|
|
303
|
+
| `path_denied: {tool?, path_matches?, source?, agent_scope?}` | **`fidelity: hostloop` only** — a path denial matching ALL given matchers (`tool` glob, `path_matches` regex, `source` ∈ pretooluse/can_use_tool/permission_denied, `agent_scope` ∈ main/subagent/any) was recorded. Replay: needs a `controlOut` cassette. Any other tier FAILS "cannot verify" |
|
|
304
|
+
| `no_path_denied: true` | **`fidelity: hostloop` only** — NO path denial was recorded at all (the channel is already path-scoped, unlike `no_hook_blocked`'s indiscriminate reject). Replay: needs a `controlOut` cassette. Any other tier FAILS "cannot verify". **Only `true` is valid** |
|
|
294
305
|
| `allow_permissive_auto_allow: true` | verdict modifier — suppresses the default-fail when the run recorded a cowork-parity permissive auto-allow; for tests that deliberately assert Cowork's permissive behavior |
|
|
295
306
|
| `allow_missing_capability: true` | verdict modifier — suppresses the default-fail when the (partial "core") agent image omits a capability the skill used but real Cowork ships (OCR/LibreOffice/markitdown/opencv/PDF-tables). Assert only when the skill's fallback is genuinely equivalent; otherwise rebuild full parity (`--build-arg COWORK_FULL_PARITY=1`). Also opts out of the `requires_capabilities` declared-need check. Live tiers only |
|
|
296
307
|
| `allow_l0_plugin_divergence: true` | verdict modifier — opt into L0/protocol plugin divergence: suppresses the default-fail when a plugin behaves differently at `protocol` (L0) fidelity than under a sandboxed tier. Live tiers only |
|
|
@@ -319,7 +330,7 @@ dotted path.
|
|
|
319
330
|
|
|
320
331
|
**VerdictSignals in `result.signals`:** `computeVerdict` pushes signals into `result.signals`; most
|
|
321
332
|
are **fail**-severity (they flip the run's pass/exit code even though `result.result` itself stays
|
|
322
|
-
`"success"`) and only
|
|
333
|
+
`"success"`) and only three are **warn**-severity (informational, never flip pass/fail). Current signal
|
|
323
334
|
codes (`VerdictSignal["code"]` in `src/run/verdict.ts`):
|
|
324
335
|
|
|
325
336
|
| Code | Severity | Meaning |
|
|
@@ -337,10 +348,11 @@ codes (`VerdictSignal["code"]` in `src/run/verdict.ts`):
|
|
|
337
348
|
| `stalled` | fail | The run ended on an unanswered question (opt out: `allow_stall`) |
|
|
338
349
|
| `non_deterministic` | warn | The run was LLM/external/human-decided — not reproducible |
|
|
339
350
|
| `prompt_asset_missing` | warn | The run proceeded with a missing prompt asset (`COWORK_HARNESS_ALLOW_MISSING_PROMPT=1`); fidelity is degraded |
|
|
351
|
+
| `scan_unavailable` | warn | Post-run scan evidence unavailable (`RunResult.scan` undefined) — the host-path and outputs-delete guards did not run this run |
|
|
340
352
|
|
|
341
353
|
A **fail**-severity signal does not change `result.result` (still `"success"`), but it DOES fail the
|
|
342
354
|
overall run verdict and exit code — `assert result: success` alone won't catch it; check
|
|
343
|
-
`result.signals[].severity` or the run's exit code. Only the
|
|
355
|
+
`result.signals[].severity` or the run's exit code. Only the three **warn** codes are truly benign.
|
|
344
356
|
|
|
345
357
|
## Replay class
|
|
346
358
|
|
|
@@ -356,7 +368,8 @@ on recording-shaping drift (`prompt`/`answers`/`baseline`/`skills`) or skill-con
|
|
|
356
368
|
sourcing ≠ evaluation (replay warns when you edit one). `verify-run` is the on-disk-`assert:` path for a kept
|
|
357
369
|
*run dir*; `--assert-from` is the equivalent for a *cassette*.
|
|
358
370
|
|
|
359
|
-
**Evaluated on replay (content):** `transcript_*`, `tool_*`, `subagent_*`, `
|
|
371
|
+
**Evaluated on replay (content):** `transcript_*`, `tool_*`, `subagent_*`, `subagent_file_write`,
|
|
372
|
+
`no_vm_path_file_op`, `dispatch_count_max`,
|
|
360
373
|
`skill_triggered`, `no_skill_triggered`, `skill_available`, `connector_available`, `tool_available`,
|
|
361
374
|
`skill_tool_used`, `max_cost_usd`, `max_tokens`, `tool_calls_max`, `tool_no_error`,
|
|
362
375
|
`max_tool_errors`, `max_redundant_tool_calls`, `max_turns`, `compaction_occurred`, `all_tasks_completed`, `task_status`, `task_count_min`, `no_scratchpad_leak`, `present_files_called`, `result`
|
|
@@ -365,7 +378,9 @@ modifiers `allow_permissive_auto_allow` / `allow_missing_capability` / `allow_l0
|
|
|
365
378
|
`allow_stall` are also kept on replay, evaluated as no-op passes.
|
|
366
379
|
|
|
367
380
|
**Gate keys — replay only with a `controlOut` cassette:** `question_asked`, `questions_count_max`,
|
|
368
|
-
`gate_answers_delivered`, `gate_answer_count_min`, `hook_blocked`, `no_hook_blocked
|
|
381
|
+
`gate_answers_delivered`, `gate_answer_count_min`, `hook_blocked`, `no_hook_blocked`, `vm_path_denied`,
|
|
382
|
+
`path_denied`, `no_path_denied` (the latter three are also `fidelity: hostloop`-only — see the assertion
|
|
383
|
+
table). With `controlOut` present they evaluate; on an old
|
|
369
384
|
cassette without it, a **loud warning** fires and they are **excluded** (not vacuously passed). Re-record to enable them.
|
|
370
385
|
`questions_count_max` counts sub-questions, not gates/tool-calls — see the catalog row above and
|
|
371
386
|
`trace --view questions`. `hook_blocked`/`no_hook_blocked` need `controlOut` for a different reason than
|
|
@@ -498,8 +513,13 @@ reference omits). Neither list is a strict superset of the other — reach for t
|
|
|
498
513
|
`artifact_json` (dotted `path` + operator); use the pytest lane (`assert_artifact_json` with a real
|
|
499
514
|
predicate) only for checks too complex for a dotted path. Find field paths via `--keep`.
|
|
500
515
|
|
|
501
|
-
10. **`subagent_dispatched` matches by `description` too** —
|
|
502
|
-
`subagent_type`
|
|
516
|
+
10. **`subagent_dispatched` matches by `resolvedAgentType` or `description` too** — a `Task` dispatch
|
|
517
|
+
with no `subagent_type` at all falls back to the built-in `general-purpose` agent (a WILDCARD tool
|
|
518
|
+
surface, `tools:["*"]`, incl. workspace bash) rather than leaving the type unresolved; the harness
|
|
519
|
+
warns loudly on this fallback and records `subagents[].dispatchTypeOmitted`, so match on the
|
|
520
|
+
resolved type or the dispatch description. `subagent_tool_absent` on a type-less dispatch is
|
|
521
|
+
correspondingly weaker evidence (wildcard surface) — pin `subagent_type` explicitly when you need a
|
|
522
|
+
tight guarantee.
|
|
503
523
|
|
|
504
524
|
11. **`subagent_declared_but_unused` fires on declared-but-didn't-use-THAT-tool**, even if the
|
|
505
525
|
sub-agent used other tools.
|
|
@@ -64,8 +64,8 @@ Top-level fields of a `*.cassette.json` (schema `schema/cassette.v10.json`):
|
|
|
64
64
|
| `preRunHashes` | Pre-run per-path sha256 baseline for `input_unmodified` (added 0.27.0, no cassetteVersion bump); a `null` value marks a path whose recorded artifact body was secret-scrubbed (evidence-unavailable, never a false "modified") |
|
|
65
65
|
| `scenarioSource` | Relative path to the authored YAML this was recorded from |
|
|
66
66
|
| `authoring` | Present iff a live decider answered ≥1 gate during recording (`nonDeterministic: true`) |
|
|
67
|
-
| `sessionFingerprint` | v9
|
|
68
|
-
| `folderPrefixMap` | v9+: the record-time connected-folder host-path → mount-name map. Replay's `computer_links_resolve` uses THIS (never the current session file)
|
|
67
|
+
| `sessionFingerprint` | Optional even on v9+ (the minimum readable version): hash of the session's content-relevant SHAPE (folders/plugins/skills/mcp/egress). Checked ONLY by `verify-cassettes`, never the default replay verdict; absent → not checked |
|
|
68
|
+
| `folderPrefixMap` | Optional even on v9+: the record-time connected-folder host-path → mount-name map. Replay's `computer_links_resolve` uses THIS (never the current session file); absent → the link is treated as evidence-unavailable, never reconstructed from the current session |
|
|
69
69
|
| `timeline`, `timelineHeader` | The recorded per-event timeline (harness-observation timestamps for tool_use/tool_result/subagent_dispatch/thinking/decision/result, in total order) plus its header (`startedAtWall`/`startedAtMono` anchors); informational only — never affects the replay verdict. Absent on a cassette recorded before this field existed |
|
|
70
70
|
| `environment` | Recording provenance: `location` (`"local"` on every cassette this harness produces), plus the resolved `tier` and `agentBinaryFormat` |
|
|
71
71
|
|
|
@@ -28,9 +28,12 @@
|
|
|
28
28
|
"no_delete_in_outputs",
|
|
29
29
|
"no_hook_blocked",
|
|
30
30
|
"no_mcp_error",
|
|
31
|
+
"no_path_denied",
|
|
31
32
|
"no_scratchpad_leak",
|
|
32
33
|
"no_skill_triggered",
|
|
33
34
|
"no_unexpected_files",
|
|
35
|
+
"no_vm_path_file_op",
|
|
36
|
+
"path_denied",
|
|
34
37
|
"present_files_called",
|
|
35
38
|
"question_asked",
|
|
36
39
|
"questions_count_max",
|
|
@@ -43,6 +46,7 @@
|
|
|
43
46
|
"skill_triggered",
|
|
44
47
|
"subagent_declared_but_unused",
|
|
45
48
|
"subagent_dispatched",
|
|
49
|
+
"subagent_file_write",
|
|
46
50
|
"subagent_output_contains",
|
|
47
51
|
"subagent_tool_absent",
|
|
48
52
|
"subagent_tool_used",
|
|
@@ -61,7 +65,8 @@
|
|
|
61
65
|
"transcript_no_host_path",
|
|
62
66
|
"transcript_not_contains",
|
|
63
67
|
"transcript_not_matches",
|
|
64
|
-
"user_visible_artifact"
|
|
68
|
+
"user_visible_artifact",
|
|
69
|
+
"vm_path_denied"
|
|
65
70
|
],
|
|
66
71
|
"topLevelKeys": [
|
|
67
72
|
"allow_host_writes",
|
|
@@ -64,6 +64,8 @@ CONTENT_KEYS = {
|
|
|
64
64
|
"subagent_dispatched",
|
|
65
65
|
"subagent_declared_but_unused",
|
|
66
66
|
"subagent_output_contains",
|
|
67
|
+
"no_vm_path_file_op",
|
|
68
|
+
"subagent_file_write",
|
|
67
69
|
"dispatch_count_max",
|
|
68
70
|
"skill_triggered",
|
|
69
71
|
"no_skill_triggered",
|
|
@@ -94,6 +96,9 @@ GATE_KEYS = {
|
|
|
94
96
|
"gate_answer_count_min",
|
|
95
97
|
"hook_blocked",
|
|
96
98
|
"no_hook_blocked",
|
|
99
|
+
"vm_path_denied",
|
|
100
|
+
"path_denied",
|
|
101
|
+
"no_path_denied",
|
|
97
102
|
}
|
|
98
103
|
# manifest-backed: replay-checkable when the cassette carries an `artifacts` manifest (record snapshots one);
|
|
99
104
|
# a manifest-less cassette skips them. Since the 0.3.0 artifact-manifest these are NOT always live-only.
|
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,175 @@ All notable changes to this project are documented here. The format is based on
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [0.30.0] — 2026-07-12
|
|
10
|
+
|
|
11
|
+
> **Upgrade notes.**
|
|
12
|
+
> - **Breaking (pre-1.0): `subagents[].agentType` renamed to `dispatchAgentType`, and
|
|
13
|
+
> `subagents[].model` renamed to `dispatchModel`.** A consumer reading either old field name off
|
|
14
|
+
> `result.json` now gets `undefined`. Both keep their original (dispatch-input) meaning — the
|
|
15
|
+
> resolved counterparts (`resolvedAgentType`/`resolvedModel`, below) are new, separate fields.
|
|
16
|
+
|
|
17
|
+
### Added
|
|
18
|
+
|
|
19
|
+
- **Parity baseline for Claude Desktop 1.20186.1.** `baseline: latest` now resolves to `desktop-1.20186.1`
|
|
20
|
+
— a patch-only Desktop release (egress allowlist, spawn config, and the Cowork system-prompt
|
|
21
|
+
fingerprint are unchanged from 1.20186.0). The pinned VM agent ELF was re-synced from 2.1.202 to
|
|
22
|
+
2.1.205 after a live Desktop self-update within the same Desktop build, restoring agreement between
|
|
23
|
+
the pinned agent version and what's staged on disk so the host-loop live lane resolves its binary.
|
|
24
|
+
- **Cassette prompt-asset fingerprint.** Prompt identity was previously keyed on `appVersion` alone, so
|
|
25
|
+
editing a committed prompt-asset file (`spawn.promptTemplate` / `subagentAppend` /
|
|
26
|
+
`subagentAppendHostLoop`) under the *same* app version silently replayed old-prompt behavior. A cassette
|
|
27
|
+
now records a hash over the referenced asset files (`fingerprint.promptAssetsHash`); a mismatch is a new
|
|
28
|
+
warn-by-default staleness finding (`prompt-assets`, fails under `--strict`), an unhashable pointer is a
|
|
29
|
+
distinct can't-verify finding (`unverifiable-prompt-assets`), and a cassette recorded before this field
|
|
30
|
+
existed surfaces a non-failing informational note instead of either.
|
|
31
|
+
- **Host-loop sub-agent tool aliasing and gated `web_fetch`.** The host-loop initialize request now
|
|
32
|
+
carries production's `Bash → mcp__workspace__bash` / `WebFetch → mcp__workspace__web_fetch` aliases
|
|
33
|
+
(single-hop; an alias never grants a tool a sub-agent's frontmatter didn't already bind).
|
|
34
|
+
- **Five new assertion keys for sub-agent path fidelity.** `no_vm_path_file_op: true` (hostloop-only —
|
|
35
|
+
no gated file tool attempted a `/sessions`-exact-or-prefixed path); `vm_path_denied: true`
|
|
36
|
+
(hostloop-only — at least one recorded denial targeted a `/sessions` path; needs `controlOut` on
|
|
37
|
+
replay); `path_denied: {tool?, path_matches?, source?, agent_scope?}` (hostloop-only — a denial
|
|
38
|
+
matching all given matchers, `source` ∈ `pretooluse`/`can_use_tool`/`permission_denied`); `no_path_denied:
|
|
39
|
+
true` (hostloop-only — no path denial recorded at all); and `subagent_file_write: {path?, path_suffix?,
|
|
40
|
+
tool?}` (**tier-agnostic** — a sub-agent-origin write/edit attempt at the given path, exact or suffix,
|
|
41
|
+
with a paired non-error tool result — the causal half of a delivery probe, pairs with `artifact_json` for
|
|
42
|
+
content). The four hostloop-only keys fail "cannot verify" (never vacuous-pass) on any other tier, since
|
|
43
|
+
`/sessions/...` is a valid path there.
|
|
44
|
+
- **`RunResult.fileToolAttempts`** — attempt-level telemetry (raw `file_path`/`path` as sent, the
|
|
45
|
+
first-match gate path, and `origin: main|subagent|unknown`) for every gated file-tool call
|
|
46
|
+
(Read/Write/Edit/Glob/Grep/MultiEdit), re-derivable on replay from the frozen tool_use stream.
|
|
47
|
+
- **`RunResult.pathDenials`** — decision-level path-denial telemetry merged from all three producers that
|
|
48
|
+
can deny a gated file-tool call on path grounds: the PreToolUse path gate's own callback, a denied
|
|
49
|
+
`can_use_tool` ask on a gated file tool with a path, and a pre-ask `permission_denied` correlated to a
|
|
50
|
+
recorded attempt. The `can_use_tool` source is reconstructible on replay only when the cassette carries
|
|
51
|
+
`controlOut`; without it the field is undefined (evidence-unavailable), never a vacuous `[]`.
|
|
52
|
+
- **`RunResult.subagents[].resolvedAgentType` / `.resolvedModel`.** The agent stream states the
|
|
53
|
+
binary-*resolved* child type (via `task_started`) and the resolved child model (via the dispatch's own
|
|
54
|
+
`tool_use_result` envelope) — previously discarded. Both are joined strictly by `toolUseId` and
|
|
55
|
+
corroborate each other without overwriting stronger evidence. `subagent_dispatched` and
|
|
56
|
+
`subagent_output_contains` now match on dispatch type, resolved type, *or* description, so a type-less
|
|
57
|
+
dispatch that resolved to e.g. `general-purpose` is selectable by either assertion.
|
|
58
|
+
- **A loud warning on type-omitted sub-agent dispatches.** A `Task` dispatch that carries no
|
|
59
|
+
`subagent_type` at all falls back to the built-in `general-purpose` agent (`tools:["*"]` — the wildcard
|
|
60
|
+
surface, including workspace bash) — faithful production behavior, and one that fires routinely, not
|
|
61
|
+
just at the edges. The harness now warns loudly when this fallback is observed (an *explicit*
|
|
62
|
+
`subagent_type: "general-purpose"` is a deliberate author choice and does not warn) and records the
|
|
63
|
+
omission on the run (`subagents[].dispatchTypeOmitted`), so it's visible in the run's own record, not
|
|
64
|
+
only in a live terminal warning.
|
|
65
|
+
- **`agent_env` session config knob** (`subagent_model`, `tool_search: "auto"|"off"`,
|
|
66
|
+
`disable_experimental_betas`) applies `CLAUDE_CODE_SUBAGENT_MODEL` / `ENABLE_TOOL_SEARCH` /
|
|
67
|
+
`CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS` **uniformly across all four fidelity tiers**. Previously an
|
|
68
|
+
operator-exported value of any of these three leaked into hostloop/protocol (which spawn over the full
|
|
69
|
+
operator shell env) but never into container/microvm (a constructed allowlist) — the same session
|
|
70
|
+
behaved differently depending only on which tier it ran at. The three keys are now also scrubbed from
|
|
71
|
+
the operator layer on the two inheriting tiers before any baseline/knob overlay, so a stray shell value
|
|
72
|
+
can never leak through on some tiers and not others; setting any `agent_env` field moves the session
|
|
73
|
+
fingerprint so `verify-cassettes` surfaces the drift on an older cassette.
|
|
74
|
+
- **`sync` now guards sub-agent prompt and path-gate fidelity.** Two new structural sentinels pin the
|
|
75
|
+
two-branch (`subagent_env_hl`/`subagent_env_vm`) sub-agent append — the branch-selection ternary,
|
|
76
|
+
the per-branch content fingerprint, and the substitution map's keys *and* values (a host/VM cwd swap
|
|
77
|
+
fails) — and the host-loop path gate's module-bounded shape (gated/excluded tool sets, every deny text,
|
|
78
|
+
the read-only-guard-before-containment order). A separate pinned gate hard-stops the sync (not just
|
|
79
|
+
warns) if production flips on a server-delivered override of the sub-agent append text, since that flip
|
|
80
|
+
is invisible to the content sentinel and the harness has no captured override text to fall back to.
|
|
81
|
+
- **`docs/subagents.md`** — the sub-agent capability/path reference: the tier-qualified outputs-addressing
|
|
82
|
+
contract (host-loop's cwd-relative `artifacts/...` vs. the VM loop's `mnt/outputs/artifacts/...` —
|
|
83
|
+
there is no single cross-tier literal form), a full capability/path matrix, the sub-agent tool-composition
|
|
84
|
+
rules (frontmatter allowlists, the `"*"` wildcard, `disallowedTools`, tool aliases, never-subagent
|
|
85
|
+
tools), the type-less dispatch trap, model-resolution precedence, and lifecycle notes (no resume, a
|
|
86
|
+
depth cap of 5 with no fan-out cap, fork-dispatch environment-append exclusion).
|
|
87
|
+
|
|
88
|
+
### Changed
|
|
89
|
+
|
|
90
|
+
- **Cassette v9 read floor.** `readCassette` now refuses cassettes recorded below v9
|
|
91
|
+
(`MIN_SUPPORTED_CASSETTE_VERSION`) with a clear re-record error instead of silently tolerating
|
|
92
|
+
legacy formats; the pre-v9 reconstruction branches (`contentSigAlgoOf`, `buildFolderPrefixMap`'s
|
|
93
|
+
session-fallback, `cmdRehash`'s pre-v3/pre-v6 checks) are deleted as unreachable, and SPEC.md's
|
|
94
|
+
stability contract states the floor. The superseded `schema/cassette.v2`–`v8.json` files are
|
|
95
|
+
removed (v9/v10 remain).
|
|
96
|
+
- **`verify-run` fails closed on degraded evidence.** A `result.json` that parses but is
|
|
97
|
+
structurally invalid (no `"success" | "error"` result field) refuses instead of being certified
|
|
98
|
+
as success; an `events.jsonl` with unparseable lines, or one yielding fewer gates than
|
|
99
|
+
`trace.json` recorded questions, refuses instead of passing answer coverage at a hollow 0/0.
|
|
100
|
+
- **Verdict honesty for missing scan evidence.** When post-run scan evidence is unavailable, the
|
|
101
|
+
`host-path` / `outputs-delete` guards report `?` (unverified) instead of a false ✓, and a new
|
|
102
|
+
warn-severity `scan_unavailable` signal surfaces in the verdict JSON.
|
|
103
|
+
- **Release gates hardened.** The CI scenario suite hard-fails on the canonical repo when
|
|
104
|
+
`ANTHROPIC_API_KEY` is missing on non-PR events (fork and Dependabot PRs keep the warn+skip);
|
|
105
|
+
image publishing now waits on ci.yml success for the tagged commit and verifies tag↔package
|
|
106
|
+
lockstep via a composite action shared with the npm release gate (`workflow_dispatch` remains
|
|
107
|
+
the documented manual-republish escape hatch). An admin can deliberately skip the live scenario
|
|
108
|
+
suite for a single release by setting a `SKIP_LIVE_SCENARIOS` repo variable to that exact commit
|
|
109
|
+
SHA — an explicit, auditable, per-commit override (a later commit can't inherit it), not a silent
|
|
110
|
+
skip; see [RELEASING.md](./RELEASING.md).
|
|
111
|
+
- `ResourceSummary.probeFailures` (shipped at runtime in 0.29.0) is now declared on the RunResult
|
|
112
|
+
type and JSON Schema; assertion-side consumption remains deferred.
|
|
113
|
+
- **Breaking (pre-1.0): `subagents[].agentType` renamed to `dispatchAgentType`, and `subagents[].model`
|
|
114
|
+
renamed to `dispatchModel`.** Both keep their original (dispatch-input) meaning — the rename exists only
|
|
115
|
+
because a *resolved* type/model now lands beside each (`resolvedAgentType`/`resolvedModel`, above) and
|
|
116
|
+
the old names would have been ambiguous between "what the dispatch asked for" and "what the binary
|
|
117
|
+
actually ran." No deprecation window; a consumer reading the old field names gets `undefined`.
|
|
118
|
+
- **The host-loop `PreToolUse` path gate is re-ported to Desktop 1.20186.1 semantics.** Plugin/skill
|
|
119
|
+
content loses its blanket path exemption and becomes a read-only category instead (a mutating tool
|
|
120
|
+
targeting it is now denied, with its own message, rather than silently passing through); the read-only
|
|
121
|
+
guard gains the two other production categories — an uploads hardlink write-block and spooled tool
|
|
122
|
+
results (`projects`) — each with per-session-type deny text; the spool dir joins the readable roots. The
|
|
123
|
+
harness's own `mode: r` connected-folder extension and the `/sessions` MultiEdit message-selection
|
|
124
|
+
nuance are preserved across the re-port.
|
|
125
|
+
|
|
126
|
+
### Fixed
|
|
127
|
+
|
|
128
|
+
- **Host-loop sub-agent dispatches now receive the correct environment description.** The harness always
|
|
129
|
+
sent sub-agents the VM-branch environment text, telling them their files exist only in a sandbox while
|
|
130
|
+
their file tools actually reach the real host filesystem — one of two prompt bugs that could seed
|
|
131
|
+
`/sessions/...` paths into a host-loop sub-agent's dispatch. A new host-loop-specific append
|
|
132
|
+
(`subagent_env_hl`) is now selected whenever `effectiveFidelity === "hostloop"` (container/microvm keep
|
|
133
|
+
the existing VM-branch text; `protocol` sends neither, a documented divergence), delivered on **both**
|
|
134
|
+
the `run`/`skill` and `chat` lanes (previously `chat` sent no sub-agent append at all).
|
|
135
|
+
- **The host-loop "Shell access" prompt's outputs bullet no longer teaches VM paths to native file tools.**
|
|
136
|
+
Its file-tool side substituted the VM session root where production substitutes the host outputs
|
|
137
|
+
directory — the other of the two prompt bugs behind stray `/sessions/...` sub-agent paths.
|
|
138
|
+
- **`web_fetch` on host-loop is now gated through a single `can_use_tool` decision, matching production**,
|
|
139
|
+
instead of being pre-approved directly by the workspace handler. Bash stays pre-approved for the whole
|
|
140
|
+
session; a fetch now emits a real permission ask, answered once by the shared provenance/domain
|
|
141
|
+
decision (which marks provenance on allow) — closing the prior two-decision, self-approving shape.
|
|
142
|
+
- **Raw run logs are scrubbed on every exit path.** Previously only the success and
|
|
143
|
+
unanswered-gate paths scrubbed `events.jsonl`/`control-out.jsonl`, and `agent.stderr.log` was
|
|
144
|
+
never scrubbed — a mid-run fault kept raw (potentially secret-bearing) logs on disk. An
|
|
145
|
+
outermost finally now scrubs all three on success, salvage, and every rethrown fault, in the
|
|
146
|
+
run and chat lanes alike; leak/capability scanners still read the raw stream first.
|
|
147
|
+
- A truncated `present_files` result (fewer returned paths than inputs, or extras) now counts as
|
|
148
|
+
malformed evidence, so `no_scratchpad_leak` fails "cannot verify" instead of green-lighting
|
|
149
|
+
only the pairs that came back.
|
|
150
|
+
- Baseline pins caught up to `desktop-1.20186.1` (the parity sync landed the baseline file
|
|
151
|
+
without bumping them, leaving `check:versions` red).
|
|
152
|
+
- `docker/Dockerfile.agent` preinstalls `pyyaml==6.0.3`, matching real rootfs provisioning (the
|
|
153
|
+
container tier no longer exercises the vendored-YAML fallback the real environment never needs).
|
|
154
|
+
|
|
155
|
+
### Removed
|
|
156
|
+
|
|
157
|
+
- The subagent-grant canary (`src/canary/`, its fixture, test, and empty snapshot) — never wired
|
|
158
|
+
to `sync`, and its drift snapshot could never fail.
|
|
159
|
+
- `scripts/boot-rootfs-vz.ts` — the tested-infeasible generic-VZ boot attempt; its finding is
|
|
160
|
+
preserved in `docs/fidelity-gaps.md`.
|
|
161
|
+
- From the npm tarball: the 1.5 MB README banner and the compiled critique evaluator (relocated
|
|
162
|
+
to `scripts/lib/critique/`, dev-only) — the package drops from 2.57 MB to ~1.05 MB compressed.
|
|
163
|
+
|
|
164
|
+
### Docs
|
|
165
|
+
|
|
166
|
+
- README's `## Status` section collapsed to the baseline-pin sentence (the perishable
|
|
167
|
+
verification narrative lives in this changelog; the feature catalogue lives in the sections
|
|
168
|
+
above it).
|
|
169
|
+
- `docs/cowork-spawn-contract-1.12603.1.md` is frozen as historical research and no longer a
|
|
170
|
+
release version-pin; `check:versions` gates on the SKILL.md and README pins only.
|
|
171
|
+
- Dropped the stale first-run egress-race gotcha (proxy startup has been synchronous for a
|
|
172
|
+
while); documented `verify-run`'s fail-closed refusal family in `docs/scenario.md` and the
|
|
173
|
+
companion skill.
|
|
174
|
+
- `docs/subagents.md` (new — see Added) is cross-linked from `docs/boundary.md` and `docs/scenario.md`;
|
|
175
|
+
`docs/plugin-root.md`'s tier table is corrected — `CLAUDE_PLUGIN_ROOT` is absent from a Bash-tool
|
|
176
|
+
subprocess's env on **every** tier, not host-loop only, as the table previously implied.
|
|
177
|
+
|
|
9
178
|
## [0.29.0] — 2026-07-11
|
|
10
179
|
|
|
11
180
|
### Added
|
package/CONTRIBUTING.md
CHANGED
|
@@ -51,7 +51,7 @@ test/ vitest unit tests
|
|
|
51
51
|
baselines/ committed platform baselines (one per Desktop release)
|
|
52
52
|
examples/ user-facing worked examples (CI-verified): scenarios/ sessions/ skills/ data/
|
|
53
53
|
e2e/ the harness's own fidelity self-tests: scenarios/ sessions/
|
|
54
|
-
fixtures/ harness runtime fixtures (
|
|
54
|
+
fixtures/ harness runtime fixtures (protocol/ golden control-response vectors)
|
|
55
55
|
docs/ guides + references
|
|
56
56
|
```
|
|
57
57
|
|
package/README.md
CHANGED
|
@@ -91,7 +91,7 @@ node dist/cli.js replay examples/replays/example-pdf-skill.cassette.json
|
|
|
91
91
|
|
|
92
92
|
> **Installed globally instead?** Once linked/installed, the same command is `cowork-harness replay
|
|
93
93
|
> <cassette>` — but the relative path above only resolves from a source checkout's `examples/replays/`.
|
|
94
|
-
> From a global install (`npm i -g "cowork-harness@>=0.
|
|
94
|
+
> From a global install (`npm i -g "cowork-harness@>=0.30.0"`), point at the package root instead:
|
|
95
95
|
> `cowork-harness replay "$(npm root -g)/cowork-harness/examples/replays/example-pdf-skill.cassette.json"`
|
|
96
96
|
> (or copy the cassette into your own project and pass that path).
|
|
97
97
|
|
|
@@ -101,7 +101,7 @@ Live `run`/`skill` need the prerequisites in the next section — note the `prot
|
|
|
101
101
|
> - **Replay only (zero setup):** `cowork-harness replay <cassette>` — no token, no Docker, no agent. The command above.
|
|
102
102
|
> - **`protocol` (real model, no Docker):** needs only the auth token (item 3 below).
|
|
103
103
|
> - **Live `container` / `microvm` / `hostloop` / `cowork`:** needs Docker (or Lima for `microvm`), a staged agent, and the token — run `cowork-harness doctor` first.
|
|
104
|
-
> - **Invocation:** from a source checkout, `node dist/cli.js <cmd>` (or `npm link` to get the `cowork-harness` command); from a global install, `cowork-harness <cmd>`; the companion skill falls back to `npx "cowork-harness@>=0.
|
|
104
|
+
> - **Invocation:** from a source checkout, `node dist/cli.js <cmd>` (or `npm link` to get the `cowork-harness` command); from a global install, `cowork-harness <cmd>`; the companion skill falls back to `npx "cowork-harness@>=0.30.0"`.
|
|
105
105
|
|
|
106
106
|
Two more worked examples worth knowing about: `examples/scenarios/protocol-smoke.yaml` (zero-Docker smoke
|
|
107
107
|
test) and `examples/scenarios/skill-loads.yaml` (container-tier acceptance check) — see
|
|
@@ -126,7 +126,7 @@ claude plugin marketplace add yaniv-golan/cowork-harness
|
|
|
126
126
|
claude plugin install cowork-harness@cowork-harness
|
|
127
127
|
```
|
|
128
128
|
|
|
129
|
-
The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx "cowork-harness@>=0.
|
|
129
|
+
The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx "cowork-harness@>=0.30.0"` (a version floor that fails loud rather than silently fetching a too-old CLI; Node ≥ 20). Tiers above `protocol` still need Docker/Lima and a Claude Desktop agent binary — see the prerequisites below.
|
|
130
130
|
|
|
131
131
|
It also follows the open [Agent Skills](https://agentskills.io) spec, so it installs cross-editor (Cursor, Codex, OpenCode, …) via [`npx skills`](https://github.com/vercel-labs/skills) (Vercel Labs' CLI implementation of that spec):
|
|
132
132
|
|
|
@@ -147,7 +147,7 @@ A global install is enough for CI `lint`, reading the teaching skill, and replay
|
|
|
147
147
|
To `run` the worked examples live or copy them as a starting point, use a source checkout. (The marketplace
|
|
148
148
|
skill install itself only pulls `.claude/skills/cowork-harness/` — SKILL.md + `references/` + `scenario.py`/
|
|
149
149
|
assertion keys, per `.claude-plugin/marketplace.json`'s `source` — not the rest of this table; the full set
|
|
150
|
-
above becomes available once the skill's first command self-bootstraps `npx "cowork-harness@>=0.
|
|
150
|
+
above becomes available once the skill's first command self-bootstraps `npx "cowork-harness@>=0.30.0"` — see
|
|
151
151
|
[above](#drive-it-from-claude-code-companion-skill) — which pulls the same npm package as the global-install row.)
|
|
152
152
|
|
|
153
153
|
### Prerequisites for anything above `protocol` fidelity
|
|
@@ -225,8 +225,8 @@ cowork-harness lint examples/scenarios/*.yaml
|
|
|
225
225
|
>
|
|
226
226
|
> | When | Assertions |
|
|
227
227
|
> |---|---|
|
|
228
|
-
> | Always | `transcript_*`, `tool_*`, `subagent_*`, `dispatch_count_max`, `skill_triggered`/`no_skill_triggered`, `max_cost_usd`/`max_tokens`/`tool_calls_max`/`max_turns` (against the *frozen recording's* spend, not fresh spend — a live `run` catches a real budget regression), `max_tool_errors`, `max_redundant_tool_calls`, `skill_available`, `connector_available`, `skill_tool_used`, `compaction_occurred`, `all_tasks_completed`, `task_count_min`, `task_status`, `no_scratchpad_leak`, `present_files_called`, `result`, the verdict modifiers |
|
|
229
|
-
> | Only if the cassette carries `controlOut` | `question_asked`, `questions_count_max`, `gate_answers_delivered`, `gate_answer_count_min`, `hook_blocked`, `no_hook_blocked` |
|
|
228
|
+
> | Always | `transcript_*`, `tool_*`, `subagent_*`, `no_vm_path_file_op`, `dispatch_count_max`, `skill_triggered`/`no_skill_triggered`, `max_cost_usd`/`max_tokens`/`tool_calls_max`/`max_turns` (against the *frozen recording's* spend, not fresh spend — a live `run` catches a real budget regression), `max_tool_errors`, `max_redundant_tool_calls`, `skill_available`, `connector_available`, `skill_tool_used`, `compaction_occurred`, `all_tasks_completed`, `task_count_min`, `task_status`, `no_scratchpad_leak`, `present_files_called`, `result`, the verdict modifiers |
|
|
229
|
+
> | Only if the cassette carries `controlOut` | `question_asked`, `questions_count_max`, `gate_answers_delivered`, `gate_answer_count_min`, `hook_blocked`, `no_hook_blocked`, `vm_path_denied`, `path_denied`, `no_path_denied` |
|
|
230
230
|
> | Only if the cassette carries an `artifacts` manifest | `file_exists`, `user_visible_artifact`, `artifact_json`, `computer_links_resolve`, `computer_links_resolve_if_present`, `no_unexpected_files`, `input_unmodified` |
|
|
231
231
|
> | Always skipped (live-only) | `egress_*`, `expect_denied`, `no_delete_in_outputs`, `self_heal_ran`, `transcript_no_host_path`, `no_mcp_error`, `max_peak_rss_bytes`, `semantic_matches` — keep these in a periodic live `run` |
|
|
232
232
|
>
|
|
@@ -766,6 +766,7 @@ a global install has them locally too, not just on GitHub.
|
|
|
766
766
|
| [docs/boundary.md](./docs/boundary.md) | The limitations model — sealed FS, default-deny egress, MCP-only crossing; how each tier enforces it; how to verify. |
|
|
767
767
|
| [docs/session.md](./docs/session.md) | Every `sessions/*.yaml` field and its Cowork mapping. |
|
|
768
768
|
| [docs/scenario.md](./docs/scenario.md) | `scenarios/*.yaml` — prompt, scripted answers, assertions. |
|
|
769
|
+
| [docs/subagents.md](./docs/subagents.md) | The sub-agent capability/path model — tier-qualified outputs contract, tool-composition rules, the type-less dispatch trap. |
|
|
769
770
|
| [docs/chat.md](./docs/chat.md) | The interactive `chat` REPL — multi-turn debugging, flags, attaching files/folders. |
|
|
770
771
|
| [docs/debugging.md](./docs/debugging.md) | Debugging a run — `inspect`/`trace`/`chat`/`verify-run` for a misbehaving skill; the false-green hunt for a green you don't trust. |
|
|
771
772
|
| [docs/cassette.md](./docs/cassette.md) | `record`/`replay` cassettes — what replay checks, which assertions are skipped. |
|
|
@@ -798,25 +799,7 @@ inputs/outputs. Human-readable terminal text is explicitly **not** part of the c
|
|
|
798
799
|
|
|
799
800
|
## Status
|
|
800
801
|
|
|
801
|
-
|
|
802
|
-
|
|
803
|
-
|
|
804
|
-
|
|
805
|
-
- `hostloop` (production split-execution: native host agent process, file tools routed host-side via the workspace SDK-MCP server into a Docker VM sidecar) and `cowork` (auto-picks `hostloop` vs `container` the way Cowork does, via gate `1143815894`).
|
|
806
|
-
- Egress enforced at `container`/`microvm`/`hostloop`; `boundary-check` reports **ALL CONSTRAINTS ENFORCED**.
|
|
807
|
-
- ✅ **Three-seam driver** — `AgentSession` (control protocol) → `Decider` (scripted + `on_unanswered` policy, no silent false-greens) → `Run` (turn loop, sub-agent dispatch tree, `RunRecord`). Multi-turn `chat`, deterministic cassette `record`/`replay` (no token), `run.jsonl`/`trace.json` logging with secret-scrub.
|
|
808
|
-
- ✅ **Answering live questions, every way you'd need** — four independent channels, all incapable of silently picking option 1 (an unhandled question always fails loud):
|
|
809
|
-
- `--decider-llm --intent "<one line>"` — a model picks per question, steered by your test intent (the ergonomic default); defaults to **Sonnet** (a weaker model prose-declines ambiguous judgment gates → fail-loud); override with `--decider-model <id>` to pin a cheaper or stronger model. It replies with the option **number** (mapped to the exact label, so it can't whiff by parroting label text); an out-of-set reply fails loud, and non-determinism is flagged so a green isn't mistaken for a scripted pass. ⚠️ A *decided* gate whose answer feeds a **semantic** assertion can still false-green — script those instead.
|
|
810
|
-
- `--answer-policy <yaml>` / `--answer "rx=c"` — declarative regex→label, deterministic for CI (accepts a **stable partial anchor** for skills whose labels drift run-to-run).
|
|
811
|
-
- `--decider-cmd '<helper>'` — custom logic via the Python `serve_decider(fn)` adapter, which pre-builds the wire loop so the helper writes only the decision.
|
|
812
|
-
- `--decider-dir <dir>` — the *driving agent* answers each gate **in-band** with full context: it arms a Monitor on `cowork-harness gates <dir> --follow` and replies with `cowork-harness answer <dir> --gate <N> --choose <label>`; the session-under-test stays live, no resume/re-worded question (binary-verified).
|
|
813
|
-
- **multiSelect gates** are answerable on every channel — repeat `--choose` (or, for `--decider-llm`, a comma-list of option numbers; or a JSON-array reply from a `--decider-cmd` helper / hand-written `resp-N.json`); all deliver the binary-verified `", "`-joined wire shape. Every channel keeps stdout free, so all compose with `--output-format json`. Validate any decider in ~2s with `cowork-harness decide`.
|
|
814
|
-
- ✅ **Sub-agent aggregation + `trace`** — recognizes the real `Agent` dispatch tool (binary-verified; `Task` is its alias) so `subagent_dispatched`/`dispatch_count_max` fire under `--fidelity cowork`, excluding the `TaskCreate` todo list; `cowork-harness trace <id> --view tools` digests `events.jsonl` (each tool row now shows its **result status** `ok`/`error`), and `trace <id> --view questions` shows the gate lifecycle (**question → injected answer → delivered result**); `result.json`/`--keep` surface the deep `mnt/outputs` deliverable path.
|
|
815
|
-
- ✅ **Answer delivery is verified, not assumed** — an AskUserQuestion answer is injected as the binary's full tool input (`{questions, answers}`, ELF-verified), so the answer actually reaches the model; `RunResult.gateDeliveries[]` + the `gate_answers_delivered` assertion catch any delivery failure, and `RunResult.toolCounts` gives the **truthful** per-tool call count (host-routed `WebSearch` shows here, not the always-0 `usage.server_tool_use`).
|
|
816
|
-
- ✅ **Binary-grounded fidelity** — cwd `/sessions/<id>`, the three-channel MCP model (`--mcp-config` honored in plain cowork mode; host/API-routed `web_fetch`; SDK-server delivery), host-loop tool partition, auth-env token-only drop, and production GrowthBook gates pinned per release.
|
|
817
|
-
- ✅ The agent binary is **bind-mounted from your own install** at run time — nothing Anthropic-owned is in any image or distributed.
|
|
818
|
-
- ✅ **File provision & session resume** — `--upload <file>` / `--folder <dir>` attach files & connect folders (`mnt/uploads`, `mnt/<folder-name>`); `--session-id <id>` + `--resume` persist and continue a session via the agent's *native* resume (binary-verified), so checkpoint-and-resume gated skills are testable. Demonstrated end-to-end in the live-contract suite (`test/live-contract.test.ts`: codeword established → resumed → recalled).
|
|
819
|
-
- ✅ **The full unit suite + the live-contract suite green**; a `cowork` pytest lane (`python/`) for skill authors.
|
|
820
|
-
- ℹ️ **Auth:** a `claude setup-token` OAuth token (or `ANTHROPIC_API_KEY`), provided via the env, `--dotenv <path>`, `./.env`, or the install's own `.env` (gitignored; keep it out of mounted folders). It's passed into the sandbox **off the process argv** (Docker: `-e KEY` inherit-by-name; microVM: a stdin prologue) so the token isn't visible via `ps`/`/proc`, scrubbed from logs, never persisted in a runtime path; the token-only path mirrors the desktop. A fresh `CLAUDE_CONFIG_DIR` alone breaks local OAuth.
|
|
821
|
-
|
|
822
|
-
See [SPEC.md](./SPEC.md) (the testable contract), [DESIGN.md](./DESIGN.md) (architecture + parity), and [CHANGELOG.md](./CHANGELOG.md).
|
|
802
|
+
The latest shipped baseline — what `baseline: latest` resolves to (`cowork-harness list`) — is
|
|
803
|
+
**`desktop-1.20186.1`**. Release-by-release verification notes (what was re-verified against
|
|
804
|
+
which live agent/asar) are recorded in [CHANGELOG.md](./CHANGELOG.md); the feature catalogue
|
|
805
|
+
this section used to duplicate lives in the sections above.
|