cowork-harness 0.28.0 → 0.29.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (61) hide show
  1. package/.claude/skills/cowork-harness/SKILL.md +22 -12
  2. package/.claude/skills/cowork-harness/references/ci-recipe.md +4 -4
  3. package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +1 -1
  4. package/.claude/skills/cowork-harness/references/scenario-schema.md +16 -15
  5. package/.claude/skills/cowork-harness/references/task-recipes.md +1 -1
  6. package/.claude/skills/cowork-harness/scripts/scenario.py +46 -2
  7. package/AGENTS.md +5 -4
  8. package/CHANGELOG.md +138 -0
  9. package/DESIGN.md +2 -2
  10. package/README.md +30 -13
  11. package/SPEC.md +13 -6
  12. package/baselines/desktop-1.20186.0.json +373 -0
  13. package/baselines/prompts/cowork-system-prompt-fingerprints.json +16 -0
  14. package/dist/agent/session.js +112 -18
  15. package/dist/agent/timeline.js +12 -8
  16. package/dist/assert.js +90 -25
  17. package/dist/baseline.js +1 -1
  18. package/dist/cli.js +26 -6
  19. package/dist/critique/evaluator.js +147 -25
  20. package/dist/critique/evidence.js +135 -20
  21. package/dist/critique/package-evidence.js +79 -22
  22. package/dist/decide/llm-transport.js +26 -3
  23. package/dist/decide/semantic-judge.js +15 -5
  24. package/dist/egress/sidecar.js +6 -7
  25. package/dist/prompt.js +16 -0
  26. package/dist/run/artifacts.js +186 -47
  27. package/dist/run/cassette.js +104 -15
  28. package/dist/run/chat-result.js +12 -2
  29. package/dist/run/chat.js +21 -1
  30. package/dist/run/execute.js +41 -12
  31. package/dist/run/matrix.js +1 -1
  32. package/dist/run/pre-run-manifest.js +72 -13
  33. package/dist/run/renderer.js +9 -1
  34. package/dist/run/run-index.js +10 -1
  35. package/dist/run/run-status.js +49 -6
  36. package/dist/run/run.js +73 -8
  37. package/dist/run/scenario-tool.js +4 -8
  38. package/dist/run/trace-view.js +9 -1
  39. package/dist/run/verdict.js +10 -1
  40. package/dist/runtime/argv.js +4 -2
  41. package/dist/runtime/resource-sampler.js +78 -21
  42. package/dist/sync/cowork-sync.js +214 -19
  43. package/dist/types.js +16 -10
  44. package/dist/usage-limit.js +40 -0
  45. package/docs/boundary.md +5 -0
  46. package/docs/cassette.md +36 -10
  47. package/docs/chat.md +8 -0
  48. package/docs/cowork-spawn-contract-1.12603.1.md +2 -2
  49. package/docs/invariants.md +4 -0
  50. package/docs/maintenance.md +11 -0
  51. package/docs/protocol.md +24 -7
  52. package/docs/run-status.md +8 -2
  53. package/docs/scenario.md +16 -14
  54. package/docs/stats.md +7 -0
  55. package/examples/replays/README.md +1 -1
  56. package/examples/replays/example-multiselect-gate.cassette.json +1 -1
  57. package/examples/replays/example-pdf-skill.cassette.json +1 -1
  58. package/examples/replays/hostloop-computer-links.cassette.json +1 -1
  59. package/package.json +1 -1
  60. package/schema/run-result.json +29 -2
  61. package/schema/scenario.schema.json +23 -11
@@ -3,8 +3,8 @@ name: cowork-harness
3
3
  description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
4
4
  metadata:
5
5
  author: cowork-harness
6
- version: 0.28.0
7
- tracks-harness: cowork-harness 0.28.0 (baseline desktop-1.19367.0)
6
+ version: 0.29.0
7
+ tracks-harness: cowork-harness 0.29.0 (baseline desktop-1.20186.0)
8
8
  ---
9
9
 
10
10
  # cowork-harness
@@ -22,8 +22,8 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
22
22
  allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
23
23
  the highest-value part. Read it.
24
24
 
25
- > **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.28.0` (baseline
26
- > `desktop-1.19367.0`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
25
+ > **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.29.0` (baseline
26
+ > `desktop-1.20186.0`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
27
27
  > `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
28
28
 
29
29
  ## Preflight — make sure the harness can actually run
@@ -38,7 +38,7 @@ cowork-harness skill ./my-skill "do X" # run the skill once against the sta
38
38
  Before the first command, confirm the CLI is reachable and **fail loud** (never fake a pass) when a tier's dependencies are missing:
39
39
 
40
40
  - **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
41
- - **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.28.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=0.28.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=0.28.0"`. **Pin `@>=0.28.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published. (≥ 0.28.0 is what gates the commands/assertions this skill teaches: `assertions --list`, `scaffold <run-id>`, `trace --view dispatches`, `artifact_json` incl. the `in:` operator (passes when the resolved value deep-equals one of the listed members — value ∈ your list, not the reverse), `verify-cassettes`, batch `record <dir>`/`--rerecord-stale`, `record --concurrency <N>`, record-time redaction, multiSelect/`answer:`, `verify-run` answer-coverage, `record --max-artifact-bytes`, live record-time deciders, `verify-cassettes --allow-domain`/`--allow-email`/`--allow-path`/`--allow-patterns-file` (`path` — local absolute filesystem paths — is the scanner's 4th class, new in 0.21.0; `--allow-patterns-file <path>` is a FILE of patterns, one regex per line — not a path to allow, unlike `--allow <regex>`), scenario `skills:` staleness scoping with `COWORK_HARNESS_AGENT_SCOPE=skill`, `chat --plugin`, `/help` in the REPL, `hostloop`'s native host/VM process split with its `allow_host_writes:` consent field, `computer_links_resolve` (new in 0.22.0), and `semantic_matches` (an LLM judge grades a fixed `rubric` of claims against the run's answer — **live-only**, so it is evidence-unavailable / skipped-loud on replay, never a vacuous pass; new in 0.27.0), and **glob-matched** `tool_called`/`tool_not_called`/`subagent_tool_used`/`subagent_tool_absent` (a pattern like `mcp__workspace__*` matches any tool in the family; exact names match exactly; new in 0.28.0).)
41
+ - **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.29.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=0.29.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=0.29.0"`. **Pin `@>=0.29.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published. (≥ 0.29.0 is what gates the commands/assertions this skill teaches: `assertions --list`, `scaffold <run-id>`, `trace --view dispatches`, `artifact_json` incl. the `in:` operator (passes when the resolved value deep-equals one of the listed members — value ∈ your list, not the reverse), `verify-cassettes`, batch `record <dir>`/`--rerecord-stale`, `record --concurrency <N>`, record-time redaction, multiSelect/`answer:`, `verify-run` answer-coverage, `record --max-artifact-bytes`, live record-time deciders, `verify-cassettes --allow-domain`/`--allow-email`/`--allow-path`/`--allow-patterns-file` (`path` — local absolute filesystem paths — is the scanner's 4th class, new in 0.21.0; `--allow-patterns-file <path>` is a FILE of patterns, one regex per line — not a path to allow, unlike `--allow <regex>`), scenario `skills:` staleness scoping with `COWORK_HARNESS_AGENT_SCOPE=skill`, `chat --plugin`, `/help` in the REPL, `hostloop`'s native host/VM process split with its `allow_host_writes:` consent field, `computer_links_resolve` (new in 0.22.0), and `semantic_matches` (an LLM judge grades a fixed `rubric` of claims against the run's answer — **live-only**, so it is evidence-unavailable / skipped-loud on replay, never a vacuous pass; new in 0.27.0), and **glob-matched** `tool_called`/`tool_not_called`/`subagent_tool_used`/`subagent_tool_absent` (a pattern like `mcp__workspace__*` matches any tool in the family; exact names match exactly; new in 0.28.0).)
42
42
  - **Agent binary (sandboxed live tiers — `container`/`microvm`/`hostloop`/`cowork`).** The staged Claude Code agent is **bind-mounted** from a local Claude Desktop install, or point `COWORK_AGENT_BINARY` at a `claude-code-vm/<ver>/claude` ELF. Nothing is bundled. `protocol` (L0) and `replay` need no staged agent; for the sandboxed tiers, no agent → no run; report that, don't skip silently.
43
43
  - **Docker / Lima.** Only `--fidelity protocol` (L0) runs without them. `container` / `microvm` / `hostloop` / `cowork` need Docker (Lima for L2). If they're absent, drop to `--fidelity protocol` and **say so** — a green that never exercised the sandbox is not a sandbox pass.
44
44
  - **Auth.** `CLAUDE_CODE_OAUTH_TOKEN` (preferred) or `ANTHROPIC_API_KEY`, via env or `.env`. Minting an OAuth token needs the **`claude` CLI** (`npm i -g @anthropic-ai/claude-code`, then `claude setup-token`).
@@ -63,7 +63,7 @@ reproducible regression (Part II), and **debug** a run that misbehaved or greene
63
63
  the path, `trace <run-id>` finds it). **Localize the failure post-hoc** from that evidence:
64
64
  `cowork-harness trace <run-dir>`'s views + the emitted `result.json` to see what the run actually did,
65
65
  then `verify-run` to re-check a suspect assertion — all token-free, no Docker, no re-record. This is
66
- the loop 0.28.0's observability is built for; the *Triage* and *Inspecting a run's observability
66
+ the loop 0.29.0's observability is built for; the *Triage* and *Inspecting a run's observability
67
67
  output* sections in **Part III — Debug** are the detail (the fuller human-facing map lives in
68
68
  `docs/debugging.md` — repo-only, not shipped with the installed skill).
69
69
  - **Multi-turn / interactive reproduction** → `cowork-harness chat` (interactive; gates answered at the
@@ -206,7 +206,7 @@ them by what you're trying to prove:
206
206
  | a skill actually **ran** (or must NOT) | `skill_triggered: <regex>`, `no_skill_triggered: <regex>` |
207
207
  | a tool ran **inside** a skill's scope | `skill_tool_used: {skill, tool}` |
208
208
  | a sub-agent did the work | `subagent_output_contains: {contains}`, `subagent_dispatched: <regex>`, `dispatch_count_max: <N>` |
209
- | a pre-existing input wasn't mutated | `input_unmodified: [<glob>]` (live/verify-run; not microvm) |
209
+ | a pre-existing input wasn't mutated (incl. `uploads/**`) | `input_unmodified: <glob>` or `[<glob>, …]` (live/verify-run; not microvm) |
210
210
  | a resource ceiling held | `max_peak_rss_bytes: <N>` (**live-only**) |
211
211
  | a hook blocked / didn't block a tool | `hook_blocked: <regex>`, `no_hook_blocked: true` (replay needs a `controlOut` cassette) |
212
212
  | every MCP round-trip succeeded | `no_mcp_error: true` (**live-only**) |
@@ -421,13 +421,23 @@ decide which assertions from *Assertions: two orthogonal axes* are worth adding)
421
421
  `tokens`, `cache-tokens`, `model-cost`, `turns`, `pass-rate`.
422
422
  - **`result.json` carries the raw fields** the assertions read: `toolDurations`, `models`, `toolErrors`,
423
423
  `redundantToolCalls`, `modelUsage`, `thinking`, `skillActivity`, `subagents[]` (prompt/model/output/
424
- `attributedSkillId`), `context` (tools/mcpServers/availableSkills), `tasks`, `workspaceFiles`,
425
- `presentedFiles`, `hookEvents`, `mcpErrors`, `contextEvents`, `resources`. (Full per-field semantics:
426
- the README's "Observability fields" section — repo-only; `schema/run-result.json` is the machine
427
- source.)
424
+ `attributedSkillId`, `outputTruncated`), `context` (tools/mcpServers/availableSkills), `tasks`,
425
+ `workspaceFiles`, `presentedFiles`, `hookEvents`, `mcpErrors`, `contextEvents`, `resources`
426
+ (`probeFailures` distinguishes a failed sample from a tier that was never sampleable). Provenance/
427
+ evidence-health fields: `command` (`run`/`skill`/`record`/`chat`/`replay` — finer than `mode`),
428
+ `gateProvenance` (per-gate `scripted`/`decided(llm|external)`/`first-option`/`prompt` with a
429
+ `bySource` histogram), `evidenceErrors` (dropped/malformed telemetry lines per stream, incl.
430
+ `egressParse`), `fingerprint.frozen` (replay only — marks the shown staleness fingerprint as the
431
+ cassette's record-time value, not a fresh recompute), and `assertTextTruncated` (companion to
432
+ `outputTruncated` on a matched tool result). (Full per-field semantics: the README's "Observability
433
+ fields" section — repo-only; `schema/run-result.json` is the machine source.)
428
434
  - **Opaque failure?** A failed run also records **`errorSource`** (where the failure originated) and
429
435
  **`stderrLogPath`** (the captured agent stderr) — read those and `trace <run-dir>` *before* re-running;
430
- a re-record rarely tells you more than the captured stderr already does.
436
+ a re-record rarely tells you more than the captured stderr already does. Also check
437
+ **`resultErrorKind`** (`"transport" | "agent" | "usage_limit"`) before spending another paid run: a
438
+ `"usage_limit"` failure is a quota exhaustion, not a skill bug — retry after the limit resets rather
439
+ than debugging; `"transport"`/`"agent"` means something actually broke, worth localizing before
440
+ re-running.
431
441
  - **Attributing cost to sub-agent work.** `subagents[]` gives the dispatch tree — each sub-agent's
432
442
  `model`, `toolsUsed`, `prompt`/`output`, and `attributedSkillId` — but **not** its own token/cost;
433
443
  aggregate cost is per-**model** in `modelUsage` (and `trace --view usage`), not per-sub-agent. So a
@@ -1,6 +1,6 @@
1
1
  # CI recipe — replay vs live lanes
2
2
 
3
- Self-contained reference. Tracks `cowork-harness 0.28.0` (baseline `desktop-1.19367.0`).
3
+ Self-contained reference. Tracks `cowork-harness 0.29.0` (baseline `desktop-1.20186.0`).
4
4
 
5
5
  **Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
6
6
  job-summary reporter (verdict table, staleness findings, cost/turns when available):
@@ -54,7 +54,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
54
54
  GitHub-hosted runners, no token/Docker/agent:
55
55
 
56
56
  ```yaml
57
- - run: npm i -g "cowork-harness@>=0.28.0"
57
+ - run: npm i -g "cowork-harness@>=0.29.0"
58
58
  - run: cowork-harness lint scenarios/*.yaml # no silent false-greens
59
59
  - run: cowork-harness verify-cassettes cassettes/ # privacy + staleness
60
60
  - run: cowork-harness replay cassettes/ # token-free content/structure
@@ -188,7 +188,7 @@ jobs:
188
188
  with: { node-version: '20' }
189
189
  - uses: actions/setup-python@v5
190
190
  with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
191
- - run: npm i -g "cowork-harness@>=0.28.0"
191
+ - run: npm i -g "cowork-harness@>=0.29.0"
192
192
  - run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
193
193
  - run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
194
194
  - run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
@@ -217,7 +217,7 @@ jobs:
217
217
  echo "live=true" >> "$GITHUB_OUTPUT"
218
218
  fi
219
219
  - if: steps.guard.outputs.live == 'true'
220
- run: npm i -g "cowork-harness@>=0.28.0"
220
+ run: npm i -g "cowork-harness@>=0.29.0"
221
221
  - if: steps.guard.outputs.live == 'true'
222
222
  run: cowork-harness run scenarios/ --output-format json
223
223
  env:
@@ -1,6 +1,6 @@
1
1
  # Fidelity tiers & answer paths
2
2
 
3
- Self-contained reference. Tracks `cowork-harness 0.28.0` (baseline `desktop-1.19367.0`).
3
+ Self-contained reference. Tracks `cowork-harness 0.29.0` (baseline `desktop-1.20186.0`).
4
4
 
5
5
  ## Fidelity tiers (`fidelity:` in the scenario)
6
6
 
@@ -1,7 +1,7 @@
1
1
  # Scenario & session schema, assertion catalog, web_fetch, full gotchas
2
2
 
3
- Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.28.0`
4
- (baseline `desktop-1.19367.0`). If your checkout is newer, prefer the live `docs/scenario.md`,
3
+ Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.29.0`
4
+ (baseline `desktop-1.20186.0`). If your checkout is newer, prefer the live `docs/scenario.md`,
5
5
  `docs/session.md`, and `SPEC.md`.
6
6
 
7
7
  **Minimal scenario** — `prompt` is the only required field:
@@ -245,18 +245,18 @@ same set live from the schema.
245
245
  | `file_exists: <path>` | the path exists under the run's `work/` (anchored at `mnt/`, e.g. `outputs/x.md`). For a user-facing deliverable prefer `user_visible_artifact` — with a connected folder the file lands in `mnt/<folder>` (= `{{workspaceFolder}}`), not `mnt/outputs`, so `file_exists: outputs/x.md` misses it |
246
246
  | `user_visible_artifact: <path>` | exists **and** under a user-visible root (`outputs/` + each connected folder's mount name) — the right primitive for a workspace deliverable when a folder is connected |
247
247
  | `no_delete_in_outputs: true` | no delete op touched `mnt/outputs` — **only `true` is valid**; `false` is rejected (omit to allow deletes) |
248
- | `no_unexpected_files: [<glob>, …]` | every **newly created** file under a user-visible root matches ≥1 glob (workRoot-relative paths; `**` = whole path segment for any depth — use `outputs/handoff/**` for per-run subdirs); `[]` = no new files; **new-files-only** — overwriting a pre-existing file in place is invisible (use content-level producer stamping); live/verify-run without a pre-run manifest ⇒ evidence-unavailable hard-fail (live runs capture the baseline only when this key is asserted; recordings always capture); **microvm cannot capture** (use container/hostloop); replay needs `cassette.preRunPaths` (≥0.24 container/hostloop recordings) — cassettes without it **exclude** the key with a loud warning |
249
- | `input_unmodified: [<glob>, …]` | every **pre-existing** file whose workRoot-relative path matches ≥1 glob keeps an unchanged content hash after the run — the in-place-mutation companion to `no_unexpected_files`'s new-files check (`[]` is rejected by the schema — list at least one glob); a matched file that was deleted counts as a content change (fails); live/verify-run without a pre-run hash manifest ⇒ evidence-unavailable hard-fail; **microvm cannot capture**; replay needs `cassette.preRunHashes` — cassettes without it **exclude** the key with a loud warning; on replay it compares against the manifest's recorded `sha256`, never a re-hash of the materialized tree |
248
+ | `no_unexpected_files: [<glob>, …]` | every **newly created** file under a user-visible root matches ≥1 glob (workRoot-relative paths; `**` = whole path segment for any depth — use `outputs/handoff/**` for per-run subdirs); `[]` = no new files; **new-files-only** — overwriting a pre-existing file in place is invisible (use content-level producer stamping); live/verify-run without a pre-run manifest ⇒ evidence-unavailable hard-fail (live runs capture the baseline only when this key is asserted; recordings always capture); an **incomplete post-run filesystem walk** (an unreadable subtree — a permission/I-O error) also fails evidence-unavailable rather than reporting "no strays" over a partial tree — distinct from the missing-manifest/microvm case, which fails for a different reason; **microvm cannot capture** (use container/hostloop); replay needs `cassette.preRunPaths` (≥0.24 container/hostloop recordings) — cassettes without it **exclude** the key with a loud warning |
249
+ | `input_unmodified: <glob>` or `[<glob>, …]` | a single glob or a list; every **pre-existing** file (incl. uploaded files under `uploads/**`) whose workRoot-relative path matches ≥1 glob keeps an unchanged content hash after the run — the in-place-mutation companion to `no_unexpected_files`'s new-files check (`[]` is rejected by the schema — list at least one glob); a matched file that was deleted counts as a content change (fails); live/verify-run without a pre-run hash manifest ⇒ evidence-unavailable hard-fail; **microvm cannot capture**; replay needs `cassette.preRunHashes` — cassettes without it **exclude** the key with a loud warning; on replay it compares against the manifest's recorded `sha256`, never a re-hash of the materialized tree |
250
250
  | `self_heal_ran: <bool>` | a plugin-root self-heal script was (not) invoked |
251
- | `tool_called: <glob>` | a tool the agent ran matched this **glob** — `*` = any run, `?` = one char, exact when literal, anchored + case-sensitive. Exact name (`Write`) matches only that tool; `mcp__workspace__*` matches any workspace tool. GLOB, not regex (`.` is literal; `.*`/`\|` match nothing → warned) |
252
- | `tool_not_called: <glob>` | NO tool the agent ran matched this glob (`mcp__*` = "no MCP tool ran"). Same glob semantics as `tool_called` |
251
+ | `tool_called: <glob>` | a tool the agent ran matched this **glob** — `*` = any run, `?` = one char, exact when literal, anchored + case-sensitive. Exact name (`Write`) matches only that tool; `mcp__workspace__*` matches any workspace tool. GLOB, not regex (`.` is literal) — an empty glob, or one containing a regex/brace-expansion metacharacter (`.*`, `.+`, `\|`, `()`, `[]`, `+`, `^`, `$`, `{}`, `\d`/`\w`/`\s`/`\b`), is **rejected at load** (a hard schema error, not a runtime warning) — it would match no real tool name and pass a `_not_`/`_absent` assert vacuously. Applies whether the glob comes from an authored scenario or a recorded cassette's frozen assert. The bundled `scenario.py lint` does NOT perform this check — only the harness enforces it, at actual load (`run`/`skill`/`record`) |
252
+ | `tool_not_called: <glob>` | NO tool the agent ran matched this glob (`mcp__*` = "no MCP tool ran"). Same glob semantics as `tool_called`, including the empty/regex-ish rejection |
253
253
  | `tool_result_contains: <str>` | a tool result includes the literal string (content / replay-checkable — substring match) |
254
254
  | `tool_result_not_contains: <str>` | no tool result includes the literal string (content / replay-checkable; fails loud when tool results are absent) |
255
- | `subagent_tool_used: <glob>` | a sub-agent used a tool matching this glob (same `*`/`?`, anchored, case-sensitive semantics as `tool_called`) |
256
- | `subagent_tool_absent: <glob>` | no sub-agent used a tool matching this glob |
255
+ | `subagent_tool_used: <glob>` | a sub-agent used a tool matching this glob (same `*`/`?`, anchored, case-sensitive semantics as `tool_called`, including the empty/regex-ish rejection) |
256
+ | `subagent_tool_absent: <glob>` | no sub-agent used a tool matching this glob (same rejection) |
257
257
  | `subagent_dispatched: <regex>` | a sub-agent whose `agentType` **or dispatch description** matches |
258
258
  | `subagent_declared_but_unused: <Tool>` | a sub-agent declared the tool but never used **that** tool (even if it used others) |
259
- | `subagent_output_contains: {match?, contains}` | a dispatched sub-agent's own output contains the substring `contains` — `match` (optional regex over `agentType`/`description`) narrows to specific dispatch(es); omitted, checks whether ANY dispatch's output contains it (existence check, not "all") |
259
+ | `subagent_output_contains: {match?, contains}` | a dispatched sub-agent's own output contains the substring `contains` — `match` (optional regex over `agentType`/`description`) narrows to specific dispatch(es); omitted, checks whether ANY dispatch's output contains it (existence check, not "all"); a miss against an output that was **truncated at the assert cap** reports evidence-unavailable instead of a proven absence — the substring could lie past the cut |
260
260
  | `dispatch_count_max: <N>` | at most N sub-agents dispatched — an author-chosen budget (Cowork imposes no in-conversation Task-dispatch cap; records only, enforces nothing — see gotcha 12) |
261
261
  | `skill_triggered: <regex>` | a skill matching the regex (invoked id, e.g. `"plugin:skill"`) was invoked via the `Skill` tool — evidence-unavailable (not a normal fail) if the agent's init tools have no `Skill` tool |
262
262
  | `no_skill_triggered: <regex>` | no invoked skill id matched — the negative-control / description-collision catcher; evidence-unavailable (never a vacuous pass) if invocation data is absent or the `Skill` tool is unobservable |
@@ -268,7 +268,7 @@ same set live from the schema.
268
268
  `subagent_tool_used`, `subagent_tool_absent` are **glob** (anchored, case-sensitive, `*`/`?`). `tool_available`,
269
269
  `skill_triggered`/`no_skill_triggered`, `skill_available`, `connector_available`, `skill_tool_used`,
270
270
  `subagent_type` are **regex** (unanchored, case-insensitive). So `tool_called: mcp__workspace__*` (glob) but
271
- `tool_available: mcp__workspace__.*` (regex) — a `.*` in a `tool_called` glob matches nothing and is warned.
271
+ `tool_available: mcp__workspace__.*` (regex) — a `.*` in a `tool_called` glob is a load-time schema error, not silently-matches-nothing.
272
272
  | `skill_tool_used: {skill, tool}` | a tool whose name matches `tool` ran inside a skill-activation window whose `skillId` matches `skill` (`RunResult.skillActivity`) — evidence-unavailable if skill-activity telemetry is absent; heuristic for inline skills (a sticky, sequential window matching the agent's `activeSkill` scope, not an exact per-tool boundary) |
273
273
  | `max_cost_usd: <N>` | the run's SDK-reported cost is ≤ N USD — evidence-unavailable if cost telemetry is absent. **Replay asserts the frozen recording's cost, not fresh spend** — a real regression needs a live `run` |
274
274
  | `max_tokens: <N>` | `usage.input_tokens + usage.output_tokens` ≤ N (cache tokens excluded) — same replay caveat as `max_cost_usd` |
@@ -281,7 +281,7 @@ same set live from the schema.
281
281
  | `compaction_occurred: true` | a context-compaction boundary occurred (a `compact_boundary` system event was recorded) — lives in the stdout stream, so meaningfully replay-checkable; evidence-unavailable if context-event telemetry is absent. **Only `true` is valid** — omit to not require it |
282
282
  | `all_tasks_completed: true` | every task in `RunResult.tasks[]` reached status `"completed"` — **requires ≥1 task** (a zero-task run fails; assert `task_count_min` for presence); evidence-unavailable if tasks telemetry is absent |
283
283
  | `task_count_min: <N>` | at least N tasks were created (`RunResult.tasks.length >= N`) — the presence companion for task assertions |
284
- | `task_status: {match, status}` | a task whose `subject` OR `id` matches the regex `match` reached `status` — evidence-unavailable if tasks telemetry is absent |
284
+ | `task_status: {match, status}` | a task whose `subject` OR `id` matches the regex `match` reached `status` — evidence-unavailable if tasks telemetry is absent; also fails **malformed** when a TaskCreate result was unparseable (corrupt task telemetry), mirroring the guard `all_tasks_completed`/`task_count_min` already had |
285
285
  | `no_scratchpad_leak: true` | every file presented via `present_files` that was in the scratchpad was successfully promoted to `mnt/outputs` (none left behind) — vacuously passes if nothing was presented (pair with a presence check to require a delivery); content-class: both the `present_files` tool_use and its own tool_result live in the ordinary events stream, so `RunResult.presentedFiles` re-derives identically on replay (meaningfully replay-checkable, same as `skill_triggered`); evidence-unavailable if `presentedFiles` telemetry is absent (an older run predating the feature). **`fidelity: container` only** — `present_files` is not served on hostloop/microvm, so a scratchpad-delivered file is neither promoted to `mnt/outputs` nor detected there (a skill that delivers via write-to-cwd→`present_files` will false-red `user_visible_artifact` on those tiers; use `container`, or write directly to `outputs/`). **Only `true` is valid** |
286
286
  | `present_files_called: true` | at least one file was actually delivered via the `present_files` tool (`RunResult.presentedFiles` is non-empty) — the presence companion to `no_scratchpad_leak` (which passes vacuously when nothing was presented). Pair them to require a delivery **and** require it not to leak. Content-class (re-derives identically on replay). **`fidelity: container` only** — `present_files` is not served on hostloop/microvm. **Only `true` is valid** |
287
287
  | `question_asked: <regex>` | the agent asked an AskUserQuestion whose text matches |
@@ -300,7 +300,7 @@ same set live from the schema.
300
300
  | `egress_allowed: <host>` | the host was allowed through |
301
301
  | `no_mcp_error: true` | no MCP round-trip failed (`RunResult.mcpErrors` is empty — no unhandled server, no handler throw) — live-only: MCP round-trips are harness-computed, not in the SDK stdout stream, so evidence-unavailable on replay (never a vacuous pass). **Only `true` is valid** |
302
302
  | `max_peak_rss_bytes: <N>` | peak sampled RSS of the agent sandbox ≤ N bytes (`RunResult.resources.peakRssBytes`) — live-only: replay never spawns a sandbox to sample, so evidence-unavailable on replay/protocol (never a vacuous pass); also evidence-unavailable when sampling captured no RSS value |
303
- | `semantic_matches: {rubric: [...], min_pass?, judge_model?}` | a pinned LLM judge grades each fixed `rubric` claim against the run's answer — the **union of the agent's final result text (`RunResult.finalMessage`), the transcript, and the final on-disk content of any files the agent authored during the run** — so a claim about content the skill led the agent to *write to a file* grades as reliably as one about inlined prose (authored-file evidence is unavailable on the **microvm** tier — no pre-run manifest to diff against, so no files are captured; `container`/`hostloop` do capture them). The assert passes iff ≥ `min_pass` claims pass (default: all — avoid for a gating scenario). Results align by claim index and are recorded per-claim in `RunResult.assertions[].semanticClaims` (`[{index, claim, pass}]`, so a consumer can diff the per-claim profile across runs); a rep whose grade can't be parsed (after one retry) is marked `RunResult.assertions[].judgeInvalid` and **never silently dropped** — it is excluded from the pass denominator, and the guard against a misleading score from that exclusion is the gate's minimum-valid-rep floor (`MIN_VALID` ≥ 4) plus this visibility, not a claim that denominator-shrinking inflation is impossible. Within a rep, a grade that's still unparseable after the retry **fails that assert outright** (evidence-unavailable, not a vacuous pass) — a persistently-flaky judge reds the run rather than silently passing. `judge_model` pins the grader (a dated id keeps a before/after comparison reproducible). Live-only: the judge is a live model call, so evidence-unavailable / skipped-loud on replay (never a vacuous pass) |
303
+ | `semantic_matches: {rubric: [...], min_pass?, judge_model?}` | a pinned LLM judge grades each fixed `rubric` claim against the run's answer — the **union of the agent's final result text (`RunResult.finalMessage`), the transcript, and the final on-disk content of any files the agent authored during the run** — so a claim about content the skill led the agent to *write to a file* grades as reliably as one about inlined prose (authored-file evidence is unavailable on the **microvm** tier — no pre-run manifest to diff against, so no files are captured; `container`/`hostloop` do capture them). Beyond the microvm case, when the authored-file evidence backing the judged document is **incomplete** — a file dropped at the capture-size cap, unreadable at read-back, or (on `--resume`) the scratchpad walk skipped — the assert fails evidence-unavailable rather than trusting a judge grade over a partial document; this is separate from the malformed-grade `judgeInvalid` path below. The assert passes iff ≥ `min_pass` claims pass (default: all — avoid for a gating scenario). Results align by claim index and are recorded per-claim in `RunResult.assertions[].semanticClaims` (`[{index, claim, pass}]`, so a consumer can diff the per-claim profile across runs); a rep whose grade can't be parsed (after one retry) is marked `RunResult.assertions[].judgeInvalid` and **never silently dropped** — it is excluded from the pass denominator, and the guard against a misleading score from that exclusion is the gate's minimum-valid-rep floor (`MIN_VALID` ≥ 4) plus this visibility, not a claim that denominator-shrinking inflation is impossible. Within a rep, a grade that's still unparseable after the retry **fails that assert outright** (evidence-unavailable, not a vacuous pass) — a persistently-flaky judge reds the run rather than silently passing. `judge_model` pins the grader (a dated id keeps a before/after comparison reproducible). Live-only: the judge is a live model call, so evidence-unavailable / skipped-loud on replay (never a vacuous pass) |
304
304
  | `artifact_json: {artifact, path, …}` | assert a JSON artifact's contents — `equals`/`gt`/`in`/`exists`/`absent`/`is_null` over a dotted `path` (`in` = membership in a list, for a stochastic/LLM value; `absent` ≠ `is_null`; an unresolved intermediate fails loud) |
305
305
  | `computer_links_resolve: true` | every `computer://` link in the model-visible transcript resolves to an artifact that exists in the run's collected outputs/mounts — a dangling link fails, naming which target was checked (a live host path, the collected work tree, or the replay manifest). **Requires ≥1 link** (zero links fails — use `computer_links_resolve_if_present` for the presence-free variant). **Only `true` is valid** (`false` is rejected by the schema) |
306
306
  | `computer_links_resolve_if_present: true` | like `computer_links_resolve` but passes vacuously when the transcript has zero `computer://` links — the presence-free variant. **Only `true` is valid** |
@@ -326,6 +326,7 @@ codes (`VerdictSignal["code"]` in `src/run/verdict.ts`):
326
326
  |---|---|---|
327
327
  | `assertion` | fail | An authored `assert:` item failed |
328
328
  | `result_error` | fail | The run's SDK result was `"error"` |
329
+ | `usage_limit` | fail | Usage/quota limit hit (not a skill failure) — retry after the limit resets. Emitted when `RunResult.resultErrorKind === "usage_limit"` |
329
330
  | `transport_error` | fail | The connection dropped mid/after-run |
330
331
  | `permissive_auto_allow` | fail | A cowork-parity auto-allow real Cowork would block (opt out: `allow_permissive_auto_allow`) |
331
332
  | `outputs_delete` | fail | An unauthorized delete touched `mnt/outputs` (opt out: author `no_delete_in_outputs`) |
@@ -344,7 +345,7 @@ overall run verdict and exit code — `assert result: success` alone won't catch
344
345
  ## Replay class
345
346
 
346
347
  A cassette (`record`/`replay`) has **no filesystem and no network**. `replay` re-evaluates only the
347
- **content** assertions. The authoritative list is `contentKeys` in `src/run/cassette.ts`.
348
+ **content** assertions. The authoritative list is `ALWAYS_CONTENT_KEYS`/`QUESTION_GATE_KEYS`/`MANIFEST_KEYS` (composed) in `src/run/cassette.ts`.
348
349
 
349
350
  **Assertion source — frozen by default, on-disk by opt-in.** A plain `replay` evaluates the `assert:` block
350
351
  **frozen in the cassette** (byte-deterministic, ignores the working tree); editing `scenarios/<name>.yaml` does
@@ -458,7 +459,7 @@ reference omits). Neither list is a strict superset of the other — reach for t
458
459
  are replay-checkable **when the cassette carries an `artifacts` manifest**; without one they are
459
460
  skipped too.) *Partial skip:* a mixed `{result, egress_denied}` greens on `result` while
460
461
  `egress_denied` is dropped. Both now warn loudly. → put egress/live-only checks on a live gate; one
461
- concern per item; run the linter. (`contentKeys` in `src/run/cassette.ts`.)
462
+ concern per item; run the linter. (`LIVE_ONLY_KEYS`/`MANIFEST_KEYS` in `src/run/cassette.ts`.)
462
463
 
463
464
  2. **Gate keys need a `controlOut` cassette.** `question_asked`, `questions_count_max`,
464
465
  `gate_answers_delivered`, `gate_answer_count_min`, `hook_blocked`, `no_hook_blocked` only evaluate on
@@ -468,7 +469,7 @@ reference omits). Neither list is a strict superset of the other — reach for t
468
469
  vacuously when zero gates fired**; use `gate_answer_count_min: 1` to also require a gate to have
469
470
  fired. A **header-only gate** (empty `question`, only `header`) can never be keyed and is rejected
470
471
  loudly — every gate needs a non-empty `question`.
471
- (`contentKeys` in `src/run/cassette.ts`; `src/assert.ts`.)
472
+ (`QUESTION_GATE_KEYS` in `src/run/cassette.ts`; `src/assert.ts`.)
472
473
 
473
474
  3. **The LLM-decider's two spellings.** Scripted answers + `on_unanswered: fail` is deterministic;
474
475
  the stochastic path flags the run `nonDeterministic`. The LLM decider is one mechanism, two
@@ -2,7 +2,7 @@
2
2
 
3
3
  Each recipe composes facts that live scattered across SKILL.md and the other references into one
4
4
  decision path. Every one answers a question a real fleet owner had to work out the hard way. Facts track the harness version in SKILL.md's
5
- front-matter (currently 0.28.0). Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
5
+ front-matter (currently 0.29.0). Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
6
6
  Recipe 3's `init-redact` shipped in 0.24.0 and are part of the current feature set — no version gate
7
7
  needed if your CLI meets SKILL.md's version floor.
8
8
 
@@ -45,7 +45,7 @@ import sys
45
45
  from pathlib import Path
46
46
 
47
47
  # --- the replay-class taxonomy ---
48
- # NB: this is NOT a 1:1 mirror of `contentKeys` in src/run/cassette.ts. cassette.ts keeps the verdict
48
+ # NB: this is NOT a 1:1 mirror of the ALWAYS_CONTENT_KEYS/QUESTION_GATE_KEYS/MANIFEST_KEYS buckets in src/run/cassette.ts. cassette.ts keeps the verdict
49
49
  # modifiers (VERDICT_MODIFIER_KEYS) in its content set so they replay as no-op passes; the linter
50
50
  # deliberately keeps them OUT of CONTENT_KEYS so a modifier-only scenario still trips the `replay-noop`
51
51
  # warning below (a no-op pass verifies nothing real — exactly what that warning is for).
@@ -688,6 +688,12 @@ def cmd_lint(args):
688
688
  # v1 declines to do.
689
689
 
690
690
  _PLUGIN_ROOT_TOKEN = re.compile(r"\$\{?CLAUDE_PLUGIN_ROOT\}?")
691
+ # A runtime SELF-HEAL for a dead ${CLAUDE_PLUGIN_ROOT}: discovering the real mount under /sessions at run
692
+ # time (the prescribed pattern — e.g. `[ -d "$X" ] || X=$(find /sessions ... -name ...)`, or an inline
693
+ # `|| python3 "$(find /sessions ...)"`). When a bash block that uses the token ALSO contains a `find` over
694
+ # /sessions, the token is dead but the block rescues it → downgrade the WARN to INFO (Item 4). Conservative:
695
+ # we do NOT verify the find pattern actually matches the plugin's layout (hence the INFO's "not validated").
696
+ _SELF_HEAL = re.compile(r"\bfind\b[^\n]*/sessions")
691
697
  # Opening/closing fence: ``` or ~~~ (>=3), optional info string (language).
692
698
  _FENCE = re.compile(r"^\s*(`{3,}|~{3,})\s*([A-Za-z0-9_+-]*)\s*$")
693
699
  # A hooks-config command string: `"command": "<value>"` (value may contain escaped quotes).
@@ -717,6 +723,19 @@ def _finding_plugin_root(path, line, ctx_label):
717
723
  )
718
724
 
719
725
 
726
+ def _finding_plugin_root_guarded(path, line, ctx_label):
727
+ return Finding(
728
+ "INFO",
729
+ "plugin-root-guarded",
730
+ f"`${{CLAUDE_PLUGIN_ROOT}}` used in an in-VM bash context ({ctx_label}), but the same block "
731
+ "self-heals it (a runtime `find` under /sessions) — the dead token is harmless here.",
732
+ "Guard not validated: the linter does not check the `find` pattern actually matches the plugin's "
733
+ "layout. Prefer resolving the mount from the script's own location over a find-fallback.",
734
+ path,
735
+ line,
736
+ )
737
+
738
+
720
739
  def _finding_hook_host_write(path, line, what):
721
740
  return Finding(
722
741
  "WARN",
@@ -748,6 +767,24 @@ def _lint_skill_text(path, raw_lines, force_json=False):
748
767
  fence_char = ""
749
768
  fence_len = 0
750
769
  fence_lang = ""
770
+ # Per-bash-fence buffer (Item 4): plugin-root token hit line numbers + the whole block's text, so a
771
+ # ${CLAUDE_PLUGIN_ROOT} use that is self-healed elsewhere IN THE SAME BLOCK downgrades WARN -> INFO.
772
+ # Emission is deferred to fence close (or EOF) but original 1-based line numbers are preserved.
773
+ bash_token_lines = []
774
+ bash_block_text = []
775
+
776
+ def flush_bash():
777
+ if bash_token_lines:
778
+ healed = _SELF_HEAL.search("\n".join(bash_block_text)) is not None
779
+ for ln in bash_token_lines:
780
+ findings.append(
781
+ _finding_plugin_root_guarded(path, ln, "```bash block")
782
+ if healed
783
+ else _finding_plugin_root(path, ln, "```bash block")
784
+ )
785
+ bash_token_lines.clear()
786
+ bash_block_text.clear()
787
+
751
788
  for i, line in enumerate(raw_lines, start=1):
752
789
  m = _FENCE.match(line)
753
790
  if m:
@@ -757,6 +794,8 @@ def _lint_skill_text(path, raw_lines, force_json=False):
757
794
  continue
758
795
  # A closing fence uses the same char, is at least as long, and carries no language.
759
796
  if marker[0] == fence_char and len(marker) >= fence_len and not lang:
797
+ if fence_lang in _BASH_FENCE_LANGS:
798
+ flush_bash() # decide WARN vs INFO now that the whole block is known
760
799
  in_fence = fence_char = ""
761
800
  fence_len = 0
762
801
  fence_lang = ""
@@ -775,8 +814,9 @@ def _lint_skill_text(path, raw_lines, force_json=False):
775
814
  ctx = "prose"
776
815
 
777
816
  if ctx == "bash":
817
+ bash_block_text.append(line) # buffer the block; token hits emit on flush (self-heal aware)
778
818
  if _PLUGIN_ROOT_TOKEN.search(line):
779
- findings.append(_finding_plugin_root(path, i, "```bash block"))
819
+ bash_token_lines.append(i)
780
820
  elif ctx == "json":
781
821
  for cm in _HOOK_CMD.finditer(line):
782
822
  _check_hook_command(path, i, cm.group(1), findings)
@@ -786,6 +826,10 @@ def _lint_skill_text(path, raw_lines, force_json=False):
786
826
  for bm in _BASH_DIRECTIVE.finditer(line):
787
827
  if _PLUGIN_ROOT_TOKEN.search(bm.group(1)):
788
828
  findings.append(_finding_plugin_root(path, i, "Bash() directive"))
829
+ # A bash fence left unclosed at EOF still has buffered token hits — flush them (else a real WARN/INFO
830
+ # would be silently dropped).
831
+ if in_fence and fence_lang in _BASH_FENCE_LANGS:
832
+ flush_bash()
789
833
  return findings
790
834
 
791
835
 
package/AGENTS.md CHANGED
@@ -42,10 +42,11 @@ protocol layer or run-loop bookkeeping in the CLI.
42
42
  scenario key is retired vocabulary — the alias is gone and it is now rejected as an unknown key — do
43
43
  not reintroduce the term.
44
44
  - **A new assertion must pick its replay class.** *Content* assertions (read only `ctx.transcript` / the
45
- record) go in `src/run/cassette.ts` `contentKeys` so they run on the token-free `replay` PR gate;
46
- *filesystem / egress* assertions do NOT — they are excluded **by omission** from `contentKeys`, so they
47
- only run on live (non-replay) gates. `contentKeys` is the single source of truth; the README's "what
48
- replay checks" prose just describes it. The wrong choice is a **silent no-op in CI**.
45
+ record) go in one of `src/run/cassette.ts`'s exported buckets — `ALWAYS_CONTENT_KEYS`, `QUESTION_GATE_KEYS`,
46
+ or `MANIFEST_KEYS` — so they run on the token-free `replay` PR gate; *filesystem / egress* assertions go
47
+ in `LIVE_ONLY_KEYS` instead, so they only run on live (non-replay) gates. These four exported constants
48
+ are the single source of truth; the README's "what replay checks" prose just describes them. The wrong
49
+ bucket is a **silent no-op in CI**.
49
50
  - **`replay` consumes `controlOut` and re-serializes via `serializeDecision` to guard the AskUserQuestion
50
51
  answer shape (O7) on the token-free lane. A new decision *kind* must extend BOTH `serializeDecision`
51
52
  AND `deserializeDecision` (declared inverses in `src/agent/session.ts`) — they must not drift.**
package/CHANGELOG.md CHANGED
@@ -6,6 +6,144 @@ All notable changes to this project are documented here. The format is based on
6
6
 
7
7
  ## [Unreleased]
8
8
 
9
+ ## [0.29.0] — 2026-07-11
10
+
11
+ ### Added
12
+
13
+ - **Parity baseline for Claude Desktop 1.20186.0.** `baseline: latest` now resolves to `desktop-1.20186.0`
14
+ (VM agent ELF 2.1.202 unchanged; native host app 2.1.205). The 1.20186.0 asar was a minifier re-anchor —
15
+ the value-resolved spawn contract is byte/behaviourally identical; the sync extractor's structural anchors
16
+ were re-anchored to the new bundle shapes (hoisted helpers became namespace-method calls; const spreads
17
+ became export aliases). Verified live across the `protocol`, `container`, and `hostloop` tiers.
18
+ - **`sync` now guards Cowork system-prompt drift.** Each sync fingerprints the prompt (a minifier-independent
19
+ content hash plus a `{{placeholder}}` / `<section>` inventory) and **refuses to write** when the content
20
+ drifts from the committed fingerprint until a new fingerprint entry is added, and **hard-fails** on a
21
+ `{{placeholder}}` the renderer neither substitutes nor explicitly allowlists. This closes a gap where a
22
+ prompt change (e.g. the deployment-gated `{{modelIdentity}}` placeholder added in 1.20186.0, which is
23
+ stripped on first-party so the rendered prompt is unchanged) could slip past the coarse `asarFingerprint`.
24
+ - **Usage/quota-limit runs are now classified as `resultErrorKind: "usage_limit"`.** A session/weekly/model/
25
+ spend/org quota-exhaustion arrives as an `is_error` result with the SDK's misleading `subtype: "success"`
26
+ and HTTP 429 — previously an undifferentiated error, indistinguishable from a real skill regression.
27
+ Detection is conjunctive (`api_error_status === 429` **and** a terminal usage-limit message; a bare 429 is
28
+ a transient overload and is not reclassified). `resultSubtype` is kept verbatim (faithful SDK passthrough).
29
+ Surfaced on `RunResult`, `status.json`, and a distinct verdict signal ("retry after reset") so a batch can
30
+ halt-fast; the `claude -p` decider transport fails fast (non-retryable) on a usage limit instead of
31
+ retrying into a spent quota.
32
+ - **`input_unmodified` can now guard uploaded files (`uploads/**`).** Uploads are captured as a read-only
33
+ input root (hash-only in cassettes — a private upload is never inlined), so a scenario can assert the agent
34
+ didn't mutate an uploaded file; a mutation is attributed to the agent (not excused as an external edit).
35
+ Previously only connected-folder inputs were guardable.
36
+ - **`RunResult.command`** records the exact command that produced a result (`run`/`skill`/`record`/`chat`/
37
+ `replay`) — finer than `mode` (which only distinguishes run vs. chat, so a `skill`/`record` run was
38
+ indistinguishable from a plain `run`). `reindex` now prefers this field, so a rebuilt run index no longer
39
+ relabels a `skill`/`record` run as `run`; it falls back to a prior index row, then to `mode`, on older
40
+ results that predate the field.
41
+ - **`RunResult.evidenceErrors.egressParse`** counts proxy-log lines the egress sidecar dropped as malformed
42
+ (bad JSON, missing host, unknown decision) — previously filtered out silently. Observability only (egress
43
+ assertions are positive, so a dropped line already fails loud), but now visible.
44
+ - **`subagents[].outputTruncated` and `toolResults[].assertTextTruncated`** flag when a captured value was cut
45
+ at the 10 KB assertion cap, so `subagent_output_contains` (and the equivalent substring checks) report
46
+ "evidence unavailable" instead of a false "not found" when the searched text could lie past the cut.
47
+ - **`fingerprint.frozen`** marks, on the replay lane, that the staleness fingerprint is the cassette's
48
+ record-time snapshot rather than a fresh recompute, so a consumer can't mistake one for the other.
49
+ - **Replay results now populate `prompt` and `toolResults`** (the scenario prompt that drove the re-drive and
50
+ the tool-result records already built for evaluation) — both were previously always `undefined` on replay.
51
+ - **`ResourceSummary.probeFailures`** counts resource-sampling probes that actually failed (nonzero exit,
52
+ timeout, parse error), distinct from a tier that was never sampleable (protocol/replay) — "sampling broke"
53
+ is now told apart from "sampling wasn't attempted."
54
+ - **`chat` now samples real resources on the `container`/`hostloop` tiers.** Previously no `ResourceSampler`
55
+ was ever started for a chat session on either tier, so `resources.jsonl` was always empty despite
56
+ `chat-result.ts`'s doc-comment claiming resource parity with `run`; a protocol-tier chat legitimately has
57
+ none (no container/process id to probe against the host `claude` binary).
58
+
59
+ ### Changed
60
+
61
+ - **`input_unmodified` accepts a single glob string** as well as an array (`input_unmodified: 'uploads/**'`
62
+ no longer errors "expected array").
63
+ - **`sessionFingerprintDrift` keeps an unresolvable session fingerprint non-failing (informational)** and
64
+ downgrades a mismatch to a non-failing note when the cassette's session was only resolved via a
65
+ name-lookup fallback (the SAME low-confidence signal `scenarioContentDrift` already downgrades on) —
66
+ a relocated cassette whose relative-offset resolution lands on an unrelated same-named session file no
67
+ longer reads as a false "session shape drifted."
68
+ - **`artifact_json`'s replay remedy names an uploaded input distinctly** — a body-less target that was
69
+ captured hash-only because it's an *upload* now says so directly, instead of the prior combined
70
+ readonly-or-over-cap wording that didn't cover the uploads case.
71
+ - **`scenario.py lint-skill`** downgrades a `${CLAUDE_PLUGIN_ROOT}`-in-VM-bash use to a `plugin-root-guarded`
72
+ **INFO** (from WARN) when the same bash block self-heals it at runtime (`[ -d ] || find /sessions …`), so a
73
+ correctly-guarded block no longer shares the alarming WARN class with a genuinely-unguarded use.
74
+ - **`no_unexpected_files` now requires a complete post-run filesystem walk.** An unreadable subtree
75
+ (permission error, race) previously collapsed to "no files found there" and could vacuously pass; the check
76
+ now fails "evidence unavailable" when any part of the tree couldn't be observed.
77
+ - **`semantic_matches` refuses to grade over incomplete authored-file evidence.** When a file the run authored
78
+ was dropped at the capture-size budget or was unreadable at read-back (or, on `--resume`, the scratchpad was
79
+ skipped to avoid misattributing a prior turn's files), the assertion fails "evidence unavailable" rather than
80
+ trusting a judge grade made without that content. The judged document is also size-capped now (per-section
81
+ and total, with an explicit truncation marker) so a long run can't overflow the judge's context; secret
82
+ scrubbing runs before the cap, so a secret can never be exposed by falling on a truncation boundary.
83
+ - **`task_status` now honors corrupt task telemetry** — when a `TaskCreate` result was unparseable it fails
84
+ "malformed" instead of matching against the surviving subset, consistent with `all_tasks_completed` /
85
+ `task_count_min`.
86
+ - **Empty or regex/brace-expansion tool globs are rejected at load.** `tool_called` / `tool_not_called` /
87
+ `subagent_tool_used` / `subagent_tool_absent` values that are empty or contain regex/brace metacharacters
88
+ (`.*`, `|`, `[]`, `{}`, …) match no real tool name and used to pass a `_not_`/`_absent` assertion vacuously;
89
+ they are now a hard schema error for authored scenarios and recorded cassettes alike.
90
+ - **`readRunStatus` validates the shape of `status.json`**, not just that it parses — a truncated or otherwise
91
+ structurally-invalid write (e.g. `{}`) is reported malformed instead of being trusted as a valid status; a
92
+ `--follow` loop resolves only after observing a genuinely valid status.
93
+ - **Answered gates reconcile against actual delivery.** A control-response that was optimistically reported
94
+ delivered (queued) but whose write later failed before reaching the agent is corrected once the run
95
+ completes, so delivery telemetry and `trace --view questions` read the true outcome.
96
+ - **A `TaskCreate` / `WebSearch` / `present_files` call whose result never arrived before the stream ended now
97
+ counts as incomplete evidence** (`evidenceErrors.*`) instead of silently vanishing from the resolved set, so
98
+ the dependent assertion fails "cannot verify" rather than grading a truncated subset as complete.
99
+ - **Bounded hashing/reads.** `classifyWorkspaceFiles` caps per-file hashing (50 MiB, matching the pre-run
100
+ manifest) and reports `hashError: "over-cap"` instead of reading a huge file whole; authored-file capture
101
+ reads only the bytes it retains.
102
+
103
+ ### Fixed
104
+
105
+ - **`sync` no longer derives a phantom `nativeStagedPath`.** The native macOS agent app (`claude-code/`) and
106
+ the container/microvm Linux ELF (`claude-code-vm/`) version on independent cadences; `sync` derived the
107
+ native (hostloop) path from the VM `.sdk-version`, producing a path that did not exist whenever the two had
108
+ drifted — and freezing that phantom into the written baseline. It now resolves the native path from the
109
+ newest app actually staged under `claude-code/`, falling back to the version-derived convention only when no
110
+ native app is staged at all.
111
+ - **Two anonymous sub-agent dispatches in different assistant turns could collide on the same synthesized id**
112
+ (the fallback id was derived only from the block's position within its own message). It now derives from the
113
+ message id plus block index — unique across the run and stable across record→replay — fixing dispatch nesting
114
+ in `trace --view dispatches` for skills that dispatch multiple unnamed sub-agents.
115
+ - **The semantic judge's recorded model could name the wrong one.** `assertions[].judgeModel` was stamped from
116
+ the requested alias (e.g. `opus`) before the call; the transport resolves an alias to a concrete model per
117
+ call, so the resolved model is now recorded after the call completes.
118
+ - **`matrix --repeat`'s per-cell pass/fail glyph didn't honor `--allow-budget-stop`**, though the aggregate
119
+ "N/M cells passed" count did — a cell that passed under a permitted budget stop could print as failed next to
120
+ its own row while counting as a pass in the summary.
121
+ - **The resource sampler could miss a short run's only sample.** `stop()` now waits (bounded) for an in-flight
122
+ probe before returning, so a run shorter than one sampling interval reliably records its first sample instead
123
+ of racing teardown.
124
+ - **A corrupt `timeline.jsonl` header was indistinguishable from an absent timeline** — both read as "no
125
+ timeline," so a corrupt read could be baked into a recorded cassette as a clean empty timeline. A corrupt
126
+ header now reads as a distinct "present but corrupt" state, kept out of both live results and cassettes.
127
+ - **A malformed `system/init` frame** (a non-array `tools`/`mcp_servers`/`skills`, or a non-object content
128
+ block) could crash with an uncaught `TypeError`; it is now rejected as a typed protocol error at parse time,
129
+ and the `trace` reconstruction lane skips it instead of aborting.
130
+ - **`verify-cassettes` no longer aborts the whole batch on a nameless cassette.** A lenient cassette with
131
+ no `scenario.name` (and no `scenarioSource`) threw inside the per-file verification, which aborted the
132
+ entire run (`results: []` — a false-green by abort) instead of reporting that one file's error and
133
+ continuing. Each file's verification is now wrapped so a per-file crash becomes that file's error row
134
+ (mirrors `replay`'s existing per-file catch).
135
+
136
+ ### Internal
137
+
138
+ - **Eval-gate hardening** (`scripts/eval-gate.ts`, maintainer instrument — not shipped with the skill): a
139
+ missing/edited scenario or discriminating claim now fails the gate (opt-out `--allow-unmatched`); a mixed-model
140
+ capture, a null candidate model against a concrete baseline, and a malformed profile/fraction are refused
141
+ instead of silently collapsing; child runs are bounded by a timeout, output cap, and process-group kill.
142
+ - **Skill-critique hardening** (`scripts/skill-critique.ts`, `src/critique/*`): the reflection turn's exit,
143
+ envelope, and session/output continuity are validated before evaluation; a corrupt or missing turn archive is
144
+ reported rather than substituting turn-2 data; the untrusted self-report is fenced as inert data before the
145
+ evaluator prompt; child turns are bounded and orphaned process groups are killed on interrupt.
146
+
9
147
  ## [0.28.0] — 2026-07-10
10
148
 
11
149
  ### Added
package/DESIGN.md CHANGED
@@ -165,7 +165,7 @@ The policy that produces those `allow`/`deny` responses is the **Decider** seam
165
165
 
166
166
  > **Machine-readable form:** the five shapes below are schema'd as `schema/protocol.v1.json`, with a golden vector pack at `fixtures/protocol/v1/` — see [docs/protocol.md](./docs/protocol.md) for scope, versioning, and how to conformance-test against them.
167
167
 
168
- ### Control protocol — VERIFIED end-to-end against the live host CLI (macOS build 2.1.177+; the staged in-VM agent that L1/L2 run is **2.1.202**, baseline **`desktop-1.19367.0`** — a fresh live end-to-end pass across the `protocol`, `container`, and `hostloop` tiers was run against this baseline on 2026-07-10, superseding the prior `1.17377.1` pin; the intervening `1.17377.1`→`1.19367.0` bumps were byte/behaviourally-identical asar analysis, now confirmed live)
168
+ ### Control protocol — VERIFIED end-to-end against the live host CLI (macOS build 2.1.177+; the staged in-VM agent that L1/L2 run is **2.1.202**, the native host app that `hostloop` runs is **2.1.205**, baseline **`desktop-1.20186.0`** — a fresh live end-to-end pass across the `protocol`, `container`, and `hostloop` tiers was run against this baseline on 2026-07-11, superseding the prior `1.19367.0` pin; the intervening `1.19367.0`→`1.20186.0` bump was a minifier re-anchor with a byte/behaviourally-identical value-resolved spawn contract, now confirmed live)
169
169
 
170
170
  > The staged agent ELF is unchanged (2.1.181) across the 1.14271.0→1.15200.0 asar bump, and 2.1.187 across the 1.15200.0→1.15962.0 bump. The live scenario suite (`protocol` + `container` tiers) was re-run against the 1.15200.0 baseline; the 1.15962.0 bump was verified via asar analysis (content byte-identical: host-loop generator, system prompt, identity string, gates, and egress domains all unchanged) plus a full local test suite pass. The 1.15962.1→1.17377.1 bump moved the staged agent to **2.1.197** and added `api.claude.ai` to the egress allowlist; re-verified via `sync` (no unknown deltas) plus a manual asar spot-check of the reconstructed prompt content (substantively unchanged — see the Parity entry in CHANGELOG.md) and a full live scenario-suite pass (`protocol` + `container` tiers).
171
171
 
@@ -179,7 +179,7 @@ The handshake and shapes below were confirmed empirically with an end-to-end run
179
179
 
180
180
  > **Cowork mode is enabled by env, not a flag.** In the staged agent (2.1.197) `--cowork` is a *plugin-scope* flag ("can only be used with user scope") and is rejected by the agent invocation; cowork mode is entered via **`CLAUDE_CODE_IS_COWORK=1`**. (Do **not** also set `CLAUDE_CODE_USE_COWORK_PLUGINS` — Desktop doesn't, and it flips the agent's userSettings filename to `cowork_settings.json` and plugin cache to `cowork_plugins/` via `TSO()` — the minified Desktop helper that derives the cowork settings/cache paths; plugins are delivered via `--plugin-dir`.) The host CLI is a different (macOS) build, so L0 runs plain (control-loop validation only); L1/L2 run the staged **Linux/arm64** binary — bind-mounted from the user's own install.
181
181
 
182
- ### Spawn contract + host-loop vs VM-loop (binary-verified through asar 1.17377.1; asar analysis since carried through 1.19367.0 — the full spawn contract is byte-identical across the 1.18286.0→1.18286.2 bump, and behaviorally identical across the 1.18286.2→1.19367.0 Vite code-split/re-minify: the value-resolved contract is unchanged and only minified symbol names + bundle layout moved, re-verified via the structural anchors and the live-asar spawn-contract tests — the live end-to-end pass remains pinned to 1.17377.1)
182
+ ### Spawn contract + host-loop vs VM-loop (binary-verified through asar 1.17377.1; asar analysis since carried through 1.20186.0 — the full spawn contract is byte-identical across the 1.18286.0→1.18286.2 bump, and behaviorally identical across the 1.18286.2→1.19367.0→1.20186.0 re-minifications: the value-resolved contract is unchanged and only minified symbol names + bundle layout moved (in 1.20186.0, hoisted helpers became namespace-method calls and const spreads became export aliases), re-verified via the structural anchors and the live-asar spawn-contract tests plus a 1.20186.0-shaped drift-guard fixture — the live end-to-end pass now covers `1.20186.0` too (see the "Control protocol" note above), superseding the prior `1.19367.0` pin)
183
183
 
184
184
  The full Desktop→agent spawn contract (cwd `/sessions/<id>`, `CLAUDE_CONFIG_DIR=mnt/.claude`, the env object, `--tools`/`--allowedTools`/`--plugin-dir`/`--effort`/`--setting-sources`, permission layers, prompt templates) is documented in [docs/cowork-spawn-contract-1.12603.1.md](./docs/cowork-spawn-contract-1.12603.1.md) and encoded in `baseline.spawn`.
185
185